Would appreciate any help with this.
How come I can see a handshake response when I send this off:
r:(`$“:ws://echo.websocket.org”)“GET / HTTP/1.1\r\nHost: echo.websocket.org\r\n\r\n”;
r
But not when I run this?
r:(`$“:wss://echo.websocket.org”)“GET / HTTP/1.1\r\nHost: echo.websocket.org\r\n\r\n”;
r
Both should exists according to https://www.websocket.org/echo.html.
it appears to work ok for me. What are you doing different to the below?
$ q
KDB+ 3.6 2018.12.06 Copyright (C) 1993-2018 Kx Systems
m64/ 4(16)core …
q)r:(`$“:ws://echo.websocket.org”)“GET / HTTP/1.1\r\nHost: echo.websocket.org\r\n\r\n”;
'.z.ws undefined
[0] r:(`$“:ws://echo.websocket.org”)“GET / HTTP/1.1\r\nHost: echo.websocket.org\r\n\r\n”;
^
q).z.ws:0N!
q)r:(`$“:ws://echo.websocket.org”)“GET / HTTP/1.1\r\nHost: echo.websocket.org\r\n\r\n”;
q)r
6i
"HTTP/1.1 101 Web Socket Protocol Handshake\r\nConnection: Upgrade\r\nDate: S..
q)r:(`$“:wss://echo.websocket.org”)“GET / HTTP/1.1\r\nHost: echo.websocket.org\r\n\r\n”;
q)r
7i
"HTTP/1.1 101 Web Socket Protocol Handshake\r\nConnection: Upgrade\r\nDate: S..
q)-26!(::)
SSLEAY_VERSION | LibreSSL 2.2.7
SSL_CERT_FILE | /Users/charlie/certs/server-crt.pem
SSL_CA_CERT_FILE | /private/etc/ssl/cacert.pem
SSL_CA_CERT_PATH | /private/etc/ssl
SSL_KEY_FILE | /Users/charlie/certs/server-key.pem
SSL_CIPHER_LIST | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| NO
q)-7 “hello world” / send a message over the tls handle 7, .z.ws will print the response below
“hello world”
Nothing as far as I can see. I get this response from the secure websocket:
ERROR: 'conn. OS reports: An established connection was aborted by the software in your host machine.
There are entries as following for the cert and key files, but I never took the time to configure SSL: I just set both of the SSL_VERIFY environment variables to NO and moved on. It must be SSL related so I’ll tackle that now and post again when I make some progress.
q)-26!(::)
SSLEAY_VERSION | OpenSSL 1.0.0k 5 Feb 2013
SSL_CERT_FILE | /usr/local/ssl/server-crt.pem
SSL_CA_CERT_FILE | /usr/local/ssl/cacert.pem
SSL_CA_CERT_PATH | /usr/local/ssl
SSL_KEY_FILE | /usr/local/ssl/server-key.pem
SSL_CIPHER_LIST | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| NO
q)
I managed to get the handshake for Websocket.org to work by upgrading OpenSSL to the following:
SSLEAY_VERSION | OpenSSL 1.0.2q 20 Nov 2018
SSL_CERT_FILE | /usr/local/ssl/client-crt.pem
SSL_CA_CERT_FILE | /usr/local/ssl/cacert.pem
SSL_CA_CERT_PATH | /usr/local/ssl
SSL_KEY_FILE | /usr/local/ssl/client-key.pem
SSL_CIPHER_LIST | ALL
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| NO
I couldn’t get the newest version of OpenSSL to work and I’m not sure the two files referred to in the Cookbook (ssleay32.dll,libeay32.dll) are included by default anymore in OpenSSL. The following code works and I can see a handshake:
server:“echo.websocket.org”;
location:“/”;
(`$“:wss://”,server)“GET “,location,” HTTP/1.1\r\nHost: “,server,”\r\nSec-WebSocket-Extensions: permessage-deflate\r\nSec-WebSocket-Protocol: zap-protocol-v1\r\n\r\n”
But there’s still something amiss, as I can’t get a valid handshake from the following site:
server:“premws-pt1.365lpodds.com”;
location:“/zap/”;
(`$“:wss://”,server)“GET “,location,” HTTP/1.1\r\nHost: “,server,”\r\nSec-WebSocket-Extensions: permessage-deflate\r\nSec-WebSocket-Protocol: zap-protocol-v1\r\n\r\n”
This page outlines that a permessage-deflate extension is needed and I think v3.2t 2014.05.08 has this: http://bet365techblog.com/bi-directional-communications-for-the-web. Is there anything else obvious going on here since I can’t seem to get a valid handle in q, but can in other settings with the same headers? Should I move on from OpenSSL altogether if that’s the issue? I note that Charles is using LibreSSL 2.2.7 in his earlier post.
Would appreciate any pointers at all.
ahah. I see we have a bug - kdb+ defers loading zlib/deflate until needed, and websockets doesn’t trigger this.
You can force it to be loaded by compressing a file. We’ll fix this.
Then the following works for me
$more bad.q
\c 1000 1000
.z.zd:19 2 1;:dummy set 1000#0;hdel:dummy;system"x .z.zd" / loads zlib as side-effect
.z.ws:0N!;
server:“premws-pt1.365lpodds.com”;
location:“/zap/”;
(`$“:wss://”,server)“GET “,location,” HTTP/1.1\r\nHost: “,server,”\r\nSec-WebSocket-Protocol: zap-protocol-v1\r\n\r\n”
$ q bad.q
KDB+ 3.6 2018.12.06 Copyright (C) 1993-2018 Kx Systems
m64/ 4(16)core…
(6i;“HTTP/1.1 101 Switching Protocols\r\nconnection: Upgrade\r\nsec-websocket-extensions: permessage-deflate; server_no_context_takeover\r\nsec-websocket-protocol: zap-protocol-v1\r\nupgrade: websocket\r\nsec-websocket-accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=\r\n\r\n”)
q)\
Thanks very much for the help with this, Charles.
I haven’t had any success with your code, unfortunately. I was getting this error when I run the ‘set’ command:
The specified module could not be found.zlib
I downloaded zlib.dll, and put it into Windows/System32 and Windows/SysWOW64 so it would be picked up. I refered to https://code.kx.com/wiki/Cookbook/FileCompression and ran the first example. It created a 78MB file and then a 135KB file, so the compression module might be loaded.
Welcome to kdb+ 32bit edition
For support please see http://groups.google.com/d/forum/personal-kdbplus
Tutorials can be found at http://code.kx.com/wiki/Tutorials
To exit, type \
To remove this startup msg, edit q.q
q)`:test set asc 10000000?100;
q)a:-19!(:test;:ztest;17;2;6);
q)get[:test]~get:ztest
1b
q).z.zd:19 2 1;:dummy set 1000#0;hdel:dummy;system"x .z.zd" / loads zlib as side-effect
q).z.ws:0N!;
q)server:“premws-pt1.365lpodds.com”;
q)location:“/zap/”;
q)(`$“:wss://”,server)“GET “,location,” HTTP/1.1\r\nHost: “,server,”\r\nSec-WebSocket-Protocol: zap-protocol-v1\r\n\r\n”
0Ni
"HTTP/1.1 101 Switching Protocols\r\nconnection: Upgrade\r\nsec-websocket-ext..
q)(-26!)
SSLEAY_VERSION | OpenSSL 1.0.2q 20 Nov 2018
SSL_CERT_FILE | /usr/local/ssl/server-crt.pem
SSL_CA_CERT_FILE | /usr/local/ssl/cacert.pem
SSL_CA_CERT_PATH | /usr/local/ssl
SSL_KEY_FILE | /usr/local/ssl/server-key.pem
SSL_CIPHER_LIST | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| NO
q)
I still can’t see a valid handle, though. I’m using OpenSSL 1.0.2q 20 Nov 2018 and have testing the client configuration at www.howsmyssl.com as outlined, but it says it’s fine. Both the ws:// and wss;// websockets at websocket.org are showing valid handles when I connect using this version of SSL. I’m just using the default list of ciphers in the example, but not sure it’s relevant. I’ve double-checked that I’ve set SSL_VERIFY_SERVER=NO in the User environment variables, so I’ve haven’t created or referenced any certs. If I force the websocket call through a proxy and examine it in Fiddler outside q it looks fine and gets a response: it just closes immediately.
r:`:ws://127.0.0.1:8888 “GET https://”, server, “/zap/ HTTP/1.1\r\nHost:”, server, “\r\nSec-WebSocket-Protocol: zap-protocol-v1\r\n\r\n”;
Is there anything I can do to see a more verbose output to see what’s happening during the handshake? Or to test whether either OpenSSL or the compression library are being loaded properly?
Would appreciate any help with this.
a 0Ni for a handle at this point would suggest the response was lacking “sec-websocket-accept”.
You can see more of the response if you set \c accordingly - can you copy it out in full and post it?
Which version/release of kdb+ are you using?
This appears to work ok for me on latest 3.6 w32. Do you see a different response text?
KDB+ 3.6 2018.12.06 Copyright (C) 1993-2018 Kx Systems
w32/ 1()core
q).z.zd:17 2 1
q):f set 1000#0;<br>q).z.ws:0N!;<br>q)server:"<a href='"http://premws-pt1.365lpodds.com"' target='"_blank"'>premws-pt1.365lpodds.com</a>";<br>q)location:"/zap/";<br>q)($“:wss://”,server)“GET “,location,” HTTP/1.1\r\nHost: “,server,”\r\nSec-WebSocket-Protocol: zap-protocol-v1\r\n\r\n”
384i
“HTTP/1.1 101 Switching Protocols\r\nconnection: Upgrade\r\nsec-websocket-extensions: permessage-deflate;
server_no_context_takeover\r\nsec-websocket-protocol:zap-protocol-v1\r\nupgrade: websocket\r\nsec-websocket-accept:HSmrc0sMlYUkAGmm5OPpG2HaGWk=\r\n\r\n”
q)-26!(::)
SSLEAY_VERSION | OpenSSL 1.0.2k 26 Jan 2017
SSL_CERT_FILE | /usr/local/ssl/server-crt.pem
SSL_CA_CERT_FILE | /usr/local/ssl/cacert.pem
SSL_CA_CERT_PATH | /usr/local/ssl
SSL_KEY_FILE | /usr/local/ssl/server-key.pem
SSL_CIPHER_LIST | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| NO
q)
there was a related fix in v3.5
2017.02.11
FIX
websocket client was sensitive to the case of the response headers sec-websocket-accept and sec-websocket-extensions.
perhaps you’re using a version/release older than this?
Thanks again for the help with this.
Making progress with version 3.6. I think I tried this at some point over the last week, but wasn’t aware of the compression issue and just reverted to 3.4 when it didn’t work the first time. . !