mkfifo fifo
seq 1000 > fifo
Can kdb read from it? Do read0 and read1 work? Thx.
More generally, how can I stream text or binary data into kdb at 100+ MB/sec w/o tokenizing them?
I think you can make q read data from stdin. I did it for one of my fh implementations (where the data source was a blackbox executable that can dump messages as text into stdout, I just piped that output into my q process and parse it for tp consumption).
When a FIFO queue, u should be able to use the normal file I/O capability of q. This helps, too: http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles
reading from a.txt worked while fifo failed.
$ yes “seq 3 > /tmp/fifo” | sh -v
.Q.fsn[0N!; `$“:/tmp/a.txt”; 1024]
.Q.fsn[0N!; `$“:/tmp/fifo”; 1024]
show read0 `$“:/tmp/a.txt”
show read0 `$“:/tmp/fifo”
show read1 `$“:/tmp/a.txt”
show read1 $":/tmp/fifo" / <font color='"#ff0000"'>zero length byte array byte$()
read0/read1 will not work on FIFOs. They read first 8 bytes of a file (I can imagine this is needed to check if a file is compressed or something like this) and then attempt to seek to offset 0 again to start reading a file’s contents. This is where they fail as seek is not supported on pipes.
But you can experiment with reading from STDIN as Flying suggested. This worked for me:
$ dd if=/tmp/tmpfifo | q test.q
where test.q is “show read0 0”
Then if you write something to /tmp/tmpfifo q will echo that.
read0 0 worked thx
i keep getting 10239 bytes per read. it is probably 10KB - 1 byte null. how can i change this buffer size? thx
I don’t think this is related to a buffer size. I tried running “seq 1500000 > /tmp/tmpfifo” and a q process echoed all the numbers. Looks like you have 10240 bytes in your FIFO, read0 reads them all and exits. If you expect more data coming to the pipe you can resume reading.
Maybe the buffer size is controlled by the pipe to the left, or the producer to the left, and not the kdb consumer to the right
You might find this man page useful: http://man7.org/linux/man-pages/man7/pipe.7.html
It could be that you’ve got 10K of data because there’s nothing left to read from the pipe:
“If all file descriptors referring to the write end of a pipe have been closed, then an attempt to read(2) from the pipe will see end-of-file (read(2) will return 0)”
It is difficult to tell for sure without knowing how the producer operates.
The 3 argument form didn’t work for me. How can I limit the amount of data consumed from the stream? Will it go wsfull if too much data is given?
http://code.kx.com/wiki/Reference/read0
/ test.q in 3 argument
1 (string count read0 (0; 0; 1024));
1 “\n”;
I believe kdb doesn’t support reading from continuous streams. read0 0 will read STDIN till EOF potentially causing wsful. Looks like you have to write a custom C extension if you need to read from (potentially infinite) streams.
.Q.fs works with STDIN, I think, no C extension needed. Although I don’t have my old code with me (it was work done with my ex-employer), I recall using .Q.fs in combination with STDIN to construct a q parser for a continuous message stream.
.Q.fs[0N!] `:/dev/stdin; / NOT WORK
.Q.fs[0N!] `:/proc/self/fd/0; / NOT WORK
.Q.fs[0N!] 0; / NOT WORK
.Q.fs[0N!] `:/tmp/a.txt; / works
Very interesting. Looking at .Q.fs definition I can’t see how it could work, this bit in particular: 1:(s;x;n). This will require seek and I don’t see how it is possible to do seek on streams.
My bad. I looked through the reference again, and figured that what was in use in my previous project was .z.pi instead of filehandle 0. With the assumption that whatever piped in can be parsed line by line, the data handle can actually be implemented in .z.pi.
I tried .z.pi too. It was covered in 2008 here