websocket client in v3.2

Community Support
1

For those of you interested in websockets, kdb+ 3.2 can now act as a client, in addition to as a server. Documented here

http://code.kx.com/wiki/Cookbook/Websocket

This is an interesting way to connect to websocket services, e.g. pusher.com or bitcoin exchanges etc.

Added websocket client functionality. .z.ws must be defined before opening a websocket, e.g.
 q).z.ws:{0N!x;} / print incoming msgs to console, no echo.
 Then open a websocket, e.g.
 q)r:(`quot;:ws://host:port")"“GET / HTTP/1.1\r\nHost: host:port\r\n\r\n”
 If successful it will return a 2 element list of (handle;http response), e.g.
 (3i;“HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=\r\nSec-WebSocket-Extensions: permessage-deflate\r\n\r\n”)
 and from that point on will callback via .z.ws when msgs arrive. To send msgs, use neg[handle]“text” or neg[handle]byteVector.
 If the protocol upgrade from http to websocket failed, it returns the 2 element list, with the handle as 0Ni, e.g.
 (0Ni;“HTTP/1.1 400 Bad Request\r\nContent-Type: text/html; charset=UTF-8…”)
 The response text is returned for debug purposes only; ideally, you need only be concerned whether the handle is valid.
 Any other error is thrown as usual, e.g.
 'www.nonexist.badcom: No route to host
 Should you need to use websockets over ssl, e.g. wss://host:port, consider stunnel, and open from kdb+ to that stunnel with ws://.
 Basic Authentication can be passed in the char vector on opening, along with any other necessary fields such as cookies etc.

An example session -

q).z.ws:{0N!x;}

q)`:ws://ws.blockchain.info:80 “GET /inv HTTP/1.1\r\nHost: ws.blockchain.info\r\n\r\n”
3i
“HTTP/1.1 101 Switching Protocols\r\nServer: nginx\r\nDate: Sat, 26 Jul 2014 09:53:02 GMT\r\nConnection: upgrade\r\nUpgrade: websocket\r\nSec-WebSocket-Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=\r\n\r\n”

q)neg[3]“{"op":"unconfirmed_sub"}”

then, after a few secs, callback prints
"{"op":"utx","x":{"hash":"0b3dc1108477539c3b635b30e4e67630b7e038c65a013efb62653276216b8267","vin_sz":1,"vout_sz":2,"lock_time":"Unavailable","size":226,"relayed_by":"188.226.161.239","tx_index":61038847,"time":1406368531,"inputs":[{"prev_out":{"value":5192700,"addr":"166wZYCWrBwj1eppfbV8K4aJV2pRtsaY4d","type":0}}],"out":[{"value":5082700,"addr":"1PXZ8fpGKnWH9tUucbXGciiJ7kMW4yiACq","type":0},{"value":100000,"addr":"1LuckyR1fFHEsXYyx5QK4UFzv3PEAepPMK","addr_tag":"LuckyBit red","addr_tag_link":"[http://luckyb.it/\](“http://luckyb.it/\“)”,\“type\”:0}]}}”

2

Hello, Can this new feature be used to access a streaming API like this?

Using curl:

C:&gt;curl -k -H “Authorization: Bearer 137529d301a65re3a32b00ee2c94b8f3-72ffdf45bb72dacy72w40fc67b6b9fbfa” “https://stream-fxpractice.oanda.com/v1/prices?accountId=1234567&amp;instruments=AUD_CAD%2CAUD_CHF"<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:44:45.688156Z”,“bid”:0.98166,“ask”:0.98188}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T02:44:42.806837Z”,“bid”:0.83898,“ask”:0.83932}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:44:47.293618Z”,“bid”:0.98164,“ask”:0.98188}}<br>{“heartbeat”:{“time”:“2014-10-03T02:44:47.744523Z”}}<br>{“heartbeat”:{“time”:“2014-10-03T02:44:50.112267Z”}}<br>{“heartbeat”:{“time”:“2014-10-03T02:44:52.744526Z”}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:44:53.304949Z”,“bid”:0.98164,“ask”:0.98189}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:44:53.441623Z”,“bid”:0.98164,“ask”:0.98188}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:44:54.442378Z”,“bid”:0.98166,“ask”:0.98188}}<br>{“heartbeat”:{“time”:“2014-10-03T02:44:55.091689Z”}}<br>{“heartbeat”:{“time”:“2014-10-03T02:44:57.744556Z”}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:44:59.854973Z”,“bid”:0.98165,“ask”:0.98189}}<br>{“heartbeat”:{“time”:“2014-10-03T02:45:00.122216Z”}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:45:01.241442Z”,“bid”:0.98168,“ask”:0.98189}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:45:01.345852Z”,“bid”:0.98168,“ask”:0.98191}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T02:45:02.240562Z”,“bid”:0.9817,"ask”:0.98189}}
^C

I don’t know how to setup q to access the streaming API.

Any help please?

Cheers

Francisco

3

That particular interface doesn’t look like it supports websockets.

They list a websocket api, at least experimental, here

http://developer.oanda.com/docs/v1/getting-started/#experimental-websocket-streaming-api

but their streaming demo

https://github.com/oanda/apidocs/blob/master/sections/streaming.md

doesn’t appear to work at the moment; maybe their test server is down. There’s a couple of questions on their forums about websockets, but I didn’t find any news since Feb2014 when they stated they are working on a http streaming api.

Which OS are you using?

4

Thanks for your answer.

I’m using MSwindows 8.1 

Oanda has 3 different environments: 1 for real trading and 2 for testing purposes. The fx-practice environment that I used in my prior post is for testing purposes but simulates the real trading machine so it requires authentication.  The sandbox environment is also for testing purposes, doesn’t require autentication and it’s working ok:

C:&gt;curl -k “http://stream-sandbox.oanda.com/v1/prices?accountId=1125870&amp;instruments=AUD_CAD%2CAUD_CHF"<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T09:11:08.569581Z”,“bid”:0.93847,“ask”:0.93898}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:08.573002Z”,“bid”:0.97679,“ask”:0.97716}}<br>{“heartbeat”:{“time”:“2014-10-03T09:11:09.783949Z”}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:09.939337Z”,“bid”:0.97691,“ask”:0.97728}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T09:11:10.202582Z”,“bid”:0.9387,“ask”:0.93921}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:10.205480Z”,“bid”:0.97698,“ask”:0.97735}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:10.311909Z”,“bid”:0.97702,“ask”:0.97739}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T09:11:10.397303Z”,“bid”:0.93876,“ask”:0.93927}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T09:11:10.738665Z”,“bid”:0.9388,“ask”:0.93931}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:10.749701Z”,“bid”:0.97706,“ask”:0.97743}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:11.317733Z”,“bid”:0.9771,“ask”:0.97747}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T09:11:11.961576Z”,“bid”:0.93886,“ask”:0.93937}}<br>{“heartbeat”:{“time”:“2014-10-03T09:11:12.308776Z”}}<br>{“tick”:{“instrument”:“AUD_CHF”,“time”:“2014-10-03T09:11:13.276997Z”,“bid”:0.97706,“ask”:0.97743}}<br>{“tick”:{“instrument”:“AUD_CAD”,“time”:“2014-10-03T09:11:14.229094Z”,“bid”:0.93883,“ask”:0.93934}}<br>{“heartbeat”:{“time”:"2014-10-03T09:11:14.783964Z”}}
^C

Cheers

Francisco 

5

kdb+ currently doesn’t support http streaming. Aside from using curl to save to a file and then import it, you can hook up a live feed via java very easily, using

http://kx.com/q/c/kx/c.java

and the following java

import java.net.*;

import java.io.*;

import kx.c;

public class URLConnectionReader {

  public static void main(String args) throws Exception {

    URL url=new URL(“http://stream-sandbox.oanda.com/v1/prices?accountId=1125870&instruments=AUD_CAD%2CAUD_CHF”);

    c c= new c(“127.0.0.1”,5000,“username:password”);

    URLConnection yc=url.openConnection();

    BufferedReader in=new BufferedReader(new InputStreamReader(yc.getInputStream()));

    String inputLine;

    while((inputLine=in.readLine())!=null)

      c.ks(“upd”,inputLine.toCharArray());

    in.close();

    c.close();

  }

}

with kdb+ listening on locahost, port 5000

q)upd:{0N!.j.k x}

then prints each line as it is received

(,tick)!+instrumenttimebid`ask!(,“AUD_CAD”;,“2014-10-03T18:37:56.449661Z”;,0.93852;,0.93903)

(,tick)!+instrumenttimebid`ask!(,“AUD_CHF”;,“2014-10-03T18:37:52.225257Z”;,0.97927;,0.97964)

(,heartbeat)!+(,time)!,“2014-10-03T20:43:47.355207Z”

and if you define

upd:{x:.j.k x;(first key x) upsert value x}

then it will insert each row into the respective table

q)upd:{x:.j.k x;(first key x) upsert value x}

q)tick

instrument time                          bid     ask    

“AUD_CAD”  “2014-10-03T18:37:56.449661Z” 0.93852 0.93903

“AUD_CHF”  “2014-10-03T18:37:52.225257Z” 0.97927 0.97964

q)heartbeat

time                         

“2014-10-03T20:51:22.431392Z”

“2014-10-03T20:51:27.431396Z”

you can cast the incoming data as necessary

q)upd:{k upsert update “P”$-1_'time from $[tick=k:first key x:.j.k x;x:update$instrument from value x;value x];}

q)tick

instrument time                          bid     ask    

AUD_CAD    2014.10.03D18:37:56.449661000 0.93852 0.93903

AUD_CHF    2014.10.03D18:37:52.225257000 0.97927 0.97964

q)heartbeat

time                         

2014.10.03D21:01:12.495310000

2014.10.03D21:01:17.495362000

2014.10.03D21:01:22.505329000

this is fine for an example, but for real world use, aside from adding error handling, for efficiency you should bulk up data to multiple rows per event/msg. Don’t forget to look at kdb+tick if you want to flesh this out

http://code.kx.com/wiki/Startingkdbplus/tick

Maybe oanda will finish their websocket api - post into their forums to let them know if you’re interested.

6

Thanks a lot for your help.

Francisco