Reading server-sent events from HTTP request

Hello,

A broker I want to integrate has their tick system sending prices using an event stream (see: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent\_events/Using\_server-sent\_events#Sending\_events\_from\_the\_server ). It is a long running HTTP request that output every tick is in a separate line delimited by two line breaks.

echo 'data: {"pair": "EUR/USD", "buy": "1.123", "sell": "1.122", "time": 1440365747319 }';echo "\n\n";

I was wondering if there is away (and should be done) in q or should I put the request and parsing on the Python side?

Thank you

Clu,

If it does come in as such, assume you can use read0 on it then maybe something like this might work (I don’t know what format the time is in…):

q)quote:([] sym:$();buy:float$(); sell:float$(); time:long$());q)quote insert "SFFJ"$ssr[;"[\"\" }';]";""] each last each ":" vs/: "," vs first read0 httpRequest,0q)quotesym buy sell time---------------------------------EUR/USD 1.123 1.122 1440365747319

Thanks Zak,

The problem is that it’s one long streaming HTTP request with all the ticks.

I tried:

txt: :http://example.com “GET /stream HTTP/1.1\r\nHost:example.com\r\nAuthorization: Basic reqsAuthRandomHash\r\nCache-Control: no-cache\r\nAccept: text/event-stream\r\n\r\n”;txt`

Since the request doesn’t finish / connection doesn’t close, kdb seems to stay on this line and just keeps going forever.

q).j.k “{"pair": "EUR/USD", "buy": "1.123", "sell": "1.122", "time": 1440365747319 }”

pair| “EUR/USD”

buy | “1.123”

sell| “1.122”

time| 1.440366e+012

Clu,

I believe keeping the connection open in .z.ph would hold up the rest of the process since kdb+ is single-threaded by default so it might be simplest to handle the long-lived connection from a separate process.

You can test this by starting up a Q process with an open port, running \sleep 100 (or some sufficiently long interval) and attempting to access it through HTTP or sending a message to it from another process. Either method should hang while Q is executing the sleep command.

Clu,

I believe keeping the connection open in .z.ph would hold up the rest of the process since kdb+ is single-threaded by default so it might be simplest to handle the long-lived connection from a separate process.

You can test this by starting up a Q process with an open port, running\sleep 100 (or some sufficiently long interval) and attempting to access it through HTTP or sending a message to it from another process. Either method should hang while Q is executing the sleep command.


(Apologies in advance if this shows up as a double-post.)

write something that consumes the stream and prints the payload to stdout. “wget -O -” may work. i havent your http streaming server to test it here. below please find the code that streams payload to stdout.

$ seq 3 | perl -ne ‘print “data: {"pair": "EUR/USD", "buy": "$.", "sell": "1.122", "time": 1440365747319 }\n”’ | ncat -klp 6000

$ ncat localhost 6000 | perl -ne ‘BEGIN {print “.z.pi:{show .j.k 6_x};\n”}; print’ | q

Welcome to kdb+ 32bit edition

For support please see http://groups.google.com/d/forum/personal-kdbplus

Tutorials can be found at http://code.kx.com/wiki/Tutorials

To exit, type \

To remove this startup msg, edit q.q

close: No error

pair| “EUR/USD”

buy | ,“1”

sell| “1.122”

time| 1.440366e+012

pair| “EUR/USD”

buy | ,“2”

sell| “1.122”

time| 1.440366e+012

pair| “EUR/USD”

buy | ,“3”

sell| “1.122”

time| 1.440366e+012

Consuming SSE would indeed be a useful feature. if anyone wants a live feed to test against, there is one here:

curl https://bitcoinity.org/ev/markets/markets_bitstamp_USD

Cheers,

Donovan.