KDB+ Websocket Invalid Handle

Community Support
1

Hi,

I can connect to the following websocket without a problem. (I have SSL_VERIFY_CLIENT set to NO as an Environment Variable.)

//wss://api.bitfinex.com/ws/2//{"event":"subscribe","pair":"BTCUSD","channel":"ticker"}.z.ws:{0N!x;};r:($“:wss://api.bitfinex.com”)“GET /ws/2 HTTP/1.1\r\nHost: api.bitfinex.com\r\n\r\n”;rneg[r 0]“{"event":"subscribe","pair":"BTCUSD","channel":"ticker"}”`

Why can’t I connect to the following websocket? The handle just comes back as invalid. 

//wss://otcsf.spreadex.com/.z.ws:{0N!x;};r:($“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\n\r\n”;r`

Both websockets work if I just test them at https://www.websocket.org/echo.html. 

2

I think you just need some additional headers on your HTTP request for opening the websocket:

q)r:(`$“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nOrigin: otcsf.spreadex.com\r\nSec-WebSocket-Version: 13\r\n\r\n”

q)r 

3i 

"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: HSmrc0sMlY.. 

q)“{type:"poll"}”

“{type:"poll"}”

“{type:"poll"}” 

Based off theprotocol handshake description at https://en.wikipedia.org/wiki/WebSocket#Protocol_handshake

As a side note, I would recommend against setting SSL_VERIFY_CLIENT to NO, as you lose the security of SSL. The “correct” way to access wss:// and https:// URLs is to set up a certificate file using the SSL_CA_CERT_FILE env var. I suggest downloading certificate bundle from here: https://curl.haxx.se/ca/cacert.pem & then point the SSL_CA_CERT_FILE variable to this file

With this set up, you’ll verify the certificates on SSL connections and ensure that you’re connecting to who you think you are. With SSL_VERIFY_CLIENT=NO you are susceptible to man-in-the-middle attacks


Hope that helps

Jonathon

From: ‘Mike’ via Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Sent: 21 March 2018 18:59
To: Kdb+ Personal Developers
Subject: [personal kdb+] KDB+ Websocket Invalid Handle
 

Hi,

I can connect to the following websocket without a problem. (I have SSL_VERIFY_CLIENT set to NO as an Environment Variable.)

//wss://api.bitfinex.com/ws/2//{"event":"subscribe","pair":"BTCUSD","channel":"ticker"}.z.ws:{0N!x;};r:($“:wss://api.bitfinex.com”)“GET /ws/2 HTTP/1.1\r\nHost: api.bitfinex.com\r\n\r\n”;rneg[r 0]“{"event":"subscribe","pair":"BTCUSD","channel":"ticker"}”`

Why can’t I connect to the following websocket? The handle just comes back as invalid. 

//wss://otcsf.spreadex.com/.z.ws:{0N!x;};r:($“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\n\r\n”;r`

Both websockets work if I just test them at https://www.websocket.org/echo.html.&nbsp;

–
Submitted via Google Groups

3

Great. Thanks again. 

4

As a follow up on this, I have written a very simple library for basic WebSocket use in Q, which allows you to connect to multiple WebSockets in one session, each with it’s own callback function. You can find this here: https://github.com/jonathonmcmurray/ws.q

As an example using your two hosts:

$ q ws.q KDB+ 3.5 2017.10.11 Copyright (C) 1993-2017 Kx Systems l32/ 2()core 1945MB jonny grizzly 127.0.1.1 NONEXPIRE q).bfx.upd:{.bfx.x,:enlist x} //define upd func for bitfinex q).spx.upd:{.spx.x,:enlist x} //define upd func for spreadex q).bfx.h:.ws.open["wss://api.bitfinex.com/ws/2";`.bfx.upd] //open bitfinex socket q).spx.h:.ws.open["wss://otcsf.spreadex.com/";`.spx.upd] //open spreadex socket q).bfx.h .j.j `event`pair`channel!`subscribe`BTCUSD`ticker //send subscription message over bfx socket q).bfx.x //check raw messages stored "{\"event\":\"info\",\"version\":2,\"platform\":{\"status\":1}}" "{\"event\":\"subscribed\",\"channel\":\"ticker\",\"chanId\":3,\"symbol\":\"tBTCUSD\",\"pair\":\"BTCUSD\"}" "[3,[8903.2,67.80649424,8904.2,49.22740929,27.3,0.0031,8904.2,43651.93267067,9177.5,8752]]" q).spx.x //check raw messages stored "{type:\"poll\"}" "{type:\"poll\"}" "{type:\"poll\"}" "{type:\"poll\"}"

In reality, I’m not sure how practical it would be to connect to multiple sockets within one session, it may be more useful to have one process per host you wish to connect to. But if you do want to do something like this, it can be done ?

Another side note: I used .j.j to generate the JSON subscribe message for bitfinex from a KDB dictionary, a little nicer than writing the JSON string directly IMO

Regards

Jonathon

From: personal-kdbplus@googlegroups.com <personal-kdbplus@googlegroups.com> on behalf of Student <niallsbonner@gmail.com>

Sent: 21 March 2018 20:14:35

To: Kdb+ Personal Developers

Subject: Re: [personal kdb+] KDB+ Websocket Invalid Handle
 

Great. Thanks again. 

On Wednesday, March 21, 2018 at 7:41:22 PM UTC, Jonathon McMurray wrote:

I think you just need some additional headers on your HTTP request for opening the websocket:

q)r:(`$“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nOrigin: otcsf.spreadex.com\r\nSec-WebSocket-Version: 13\r\n\r\n”

q)r 

3i 

"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: HSmrc0sMlY..

q)“{type:"poll"}”

“{type:"poll"}”

“{type:"poll"}” 

Based off the protocol handshake description at https://en.wikipedia.org/wiki/WebSocket#Protocol_handshake

As a side note, I would recommend against setting SSL_VERIFY_CLIENT to NO, as you lose the security of SSL. The “correct” way to access wss:// and https:// URLs is to set up a certificate file using the SSL_CA_CERT_FILE env var. I suggest downloading certificate bundle from here: https://curl.haxx.se/ca/cacert.pem & then point the SSL_CA_CERT_FILE variable to this file

With this set up, you’ll verify the certificates on SSL connections and ensure that you’re connecting to who you think you are. With SSL_VERIFY_CLIENT=NO you are susceptible to man-in-the-middle attacks



Hope that helps

Jonathon

From: ‘Mike’ via Kdb+ Personal Developers <personal…@googlegroups.com>

Sent: 21 March 2018 18:59

To: Kdb+ Personal Developers

Subject: [personal kdb+] KDB+ Websocket Invalid Handle
 

Hi,

I can connect to the following websocket without a problem. (I have SSL_VERIFY_CLIENT set to NO as an Environment Variable.)

//wss://api.bitfinex.com/ws/2 //{"event":"subscribe","pair":"BTCUSD","channel":"ticker"} .z.ws:{0N!x;}; r:($“:wss://api.bitfinex.com”)“GET /ws/2 HTTP/1.1\r\nHost:
api.bitfinex.com\r\n\r\n”;r
neg[r 0]“{"event":"subscribe","pair":"BTCUSD","channel":"ticker"}”
`

Why can’t I connect to the following websocket? The handle just comes back as invalid. 

//wss://otcsf.spreadex.com/ .z.ws:{0N!x;}; r:($“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost:
otcsf.spreadex.com\r\n\r\n”;r
`

Both websockets work if I just test them at https://www.websocket.org/echo.html. 

–
Submitted via Google Groups

5

Very useful. Thanks again. 

I’m having trouble with the second websocket. The JSON that’s coming out of it doesn’t have quotes around the variable names, so it’s not really ‘valid’ JSON. I’m having difficulty parsing the incoming messages and saving them to a table. The code to subscribe to that websocket is as follows:

.z.ws:{0N!x;}; / print incoming msgs to console, no echo.

r:(`$“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nOrigin: otcsf.spreadex.com\r\nSec-WebSocket-Version: 13\r\n\r\n”;

neg[r 0]“{"cmd":"authenticate","token":"guest"}”

neg[r 0]“{"cmd":"subscribe","dref":"TraFinSprXGrp|105","depth":2}”

Should I try to parse this just with delimiters, or is there an easier way to do this?

6

Yeah that’s going to be annoying to handle. You might be best to take a look at the old JSON parser published by KX here: https://github.com/KxSystems/kdb/blob/master/e/json.k<o:p></o:p>

<o:p> </o:p>

(I believe this was published before .j.k was added to KDB)<o:p></o:p>

<o:p> </o:p>

It might be best to modify this parser to handle the ‘bad’ JSON you are receiving. Alternatively as a rough workaround you could do some pre-processing on the strings to try and convert them into valid JSON<o:p></o:p>

<o:p> </o:p>

For example (using ws.q I posted previously)<o:p></o:p>

<o:p> </o:p>

$ q ws.q<o:p></o:p>

KDB+ 3.5 2017.11.30 Copyright (C) 1993-2017 Kx Systems<o:p></o:p>

l64/ 8()core 16048MB jmcmurray homer.aquaq.co.uk 127.0.1.1 EXPIRE 2018.06.30 AquaQ #50170<o:p></o:p>

<o:p> </o:p>

q).spx.k:{.j.k “"“sv(0,asc raze 1 1 0+'x ss/:”{,:”)cut x}<o:p></o:p>

q).spx.upd:{show .spx.k x}<o:p></o:p>

q).spx.h:.ws.open[“wss://otcsf.spreadex.com”;`.spx.upd]<o:p></o:p>

q).spx.h .j.j cmdtoken!authenticateguest<o:p></o:p>

q)type      | “authenticate”<o:p></o:p>

successful| “true”<o:p></o:p>

keepAlive | “6000”<o:p></o:p>

nodeId    | ,“6”<o:p></o:p>

serverTime| “1521813347131”<o:p></o:p>

type| “poll”<o:p></o:p>

.spx.h .j.j cmddrefdepth!(subscribe;“TraFinSprXGrp|105”;2)<o:p></o:p>

q)'illegal char * at 101<o:p></o:p>

'illegal char D at 82<o:p></o:p>

'illegal char D at 87<o:p></o:p>

'illegal char D at 86<o:p></o:p>

'illegal char G at 80<o:p></o:p>

'illegal char E at 80<o:p></o:p>

'illegal char D at 90<o:p></o:p>

dref     | “TraFinSprXMkt|87791”<o:p></o:p>

sell     | 11961f<o:p></o:p>

hash     | “A8-43-EB-66-E1”<o:p></o:p>

buy      | 11962f<o:p></o:p>

timestamp| “5248260119971866533”<o:p></o:p>

chg      | -122.5<o:p></o:p>

dref     | “TraFinSprXMkt|83430”<o:p></o:p>

sell     | 1.23343<o:p></o:p>

hash     | “27-4D-0D-7E-B8”<o:p></o:p>

buy      | 1.23349<o:p></o:p>

timestamp| “5248260119972511089”<o:p></o:p>

chg      | 0.00154<o:p></o:p>

<o:p> </o:p>

Here the .spx.k function attempts to reconstruct the JSON with the quotation marks inserted. This is done in a fairly naïve way, and clearly doesn’t work for all messages (hence the ‘illegal char errors – I think the problem in most of these is the presence of commas within strings inside the JSON-like object), but it may be at least a starting point for you to build upon.<o:p></o:p>

<o:p> </o:p>

Hope this helps<o:p></o:p>

Jonathon<o:p></o:p>

<o:p> </o:p>

From:‘Student’ via Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Sent: 22 March 2018 20:28
To: Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Subject: [personal kdb+] Re: KDB+ Websocket Invalid Handle<o:p></o:p>

<o:p> </o:p>

Very useful. Thanks again. <o:p></o:p>

<o:p> </o:p>

I’m having trouble with the second websocket. The JSON that’s coming out of it doesn’t have quotes around the variable names, so it’s not really ‘valid’ JSON. I’m having difficulty parsing the incoming messages and saving them to a table. The code to subscribe to that websocket is as follows:<o:p></o:p>

<o:p> </o:p>

.z.ws:{0N!x;}; / print incoming msgs to console, no echo.<o:p></o:p>

r:(`$“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nOrigin: otcsf.spreadex.com\r\nSec-WebSocket-Version: 13\r\n\r\n”;<o:p></o:p>

neg[r 0]“{"cmd":"authenticate","token":"guest"}”<o:p></o:p>

neg[r 0]“{"cmd":"subscribe","dref":"TraFinSprXGrp|105","depth":2}”<o:p></o:p>

<o:p> </o:p>

Should I try to parse this just with delimiters, or is there an easier way to do this?<o:p></o:p>

<o:p> </o:p>

<o:p> </o:p>

<o:p> </o:p>

On Wednesday, March 21, 2018 at 7:00:17 PM UTC, Student wrote:<o:p></o:p>

Hi,<o:p></o:p>

<o:p> </o:p>

I can connect to the following websocket without a problem. (I have SSL_VERIFY_CLIENT set to NO as an Environment Variable.)<o:p></o:p>

<o:p> </o:p>

//wss://api.bitfinex.com/ws/2
//{“event”:“subscribe”,“pair”:“BTCUSD”,“channel”:“ticker”}
.z.ws:{0N!x;};
r:(`$“:wss://api.bitfinex.com”)“GET /ws/2 HTTP/1.1\r\nHost: api.bitfinex.com\r\n\r\n”;r
neg[r 0]“{"event":"subscribe","pair":"BTCUSD","channel":"ticker"}”<o:p></o:p>

<o:p> </o:p>

Why can’t I connect to the following websocket? The handle just comes back as invalid. <o:p></o:p>

<o:p> </o:p>

//wss://otcsf.spreadex.com/
.z.ws:{0N!x;};
r:(`$“:wss://otcsf.spreadex.com”)“GET / HTTP/1.1\r\nHost: otcsf.spreadex.com\r\n\r\n”;r<o:p></o:p>

<o:p> </o:p>

Both websockets work if I just test them at https://www.websocket.org/echo.html. <o:p></o:p>

<o:p> </o:p>

<o:p> </o:p>

<o:p> </o:p>

<o:p> </o:p>

–
Submitted via Google Groups

7

Annoying indeed - but the incoming messages from spreadex in that example are js objects - not json.

Perhaps spreadex has a messageResponseType key in the subscription call you can change?

It’s likely not a good idea to put this in to fast flowing websocket subscriptions, but to try and help the problem with regards to getting from js->json, json5 is a sample tool I used before for text files that can do that for you. It should handle all messages, tried and tested already, so is beneficial there over creating a pre-processor yourself - https://github.com/json5/json5. You can pre-process the messages with this, it just wont be fast, and spreadex are bound to send json! Maybe they have their reasons for only sending that type.

// example reading line by line

while IFS= read line ; do echo $line ; echo “$line” | json5 ; echo -e “\n” ; done < js.txt

{type:“authenticate”,successful:“true”,keepAlive:“6000”,nodeId:“8”,serverTime:“1521817382323”}

{“type”:“authenticate”,“successful”:“true”,“keepAlive”:“6000”,“nodeId”:“8”,“serverTime”:“1521817382323”}

{dref:“TraFinSprXGrp|105”,initial:“true”,name:“Public Prices”,children:“TraFinSprXMkt|87623,*|87790,*|87791,*|83169,*|83430,*|293796”}

{“dref”:“TraFinSprXGrp|105”,“initial”:“true”,“name”:“Public Prices”,“children”:“TraFinSprXMkt|87623,*|87790,*|87791,*|83169,*|83430,*|293796”}

..then .j.k will work…

I’d be interested to know if there are better tools out there for this conversion tbh