Interfacing with Node.js

Community Support
1

Hi everyone,

thanks to http://kx.com/q/c/c.js we have JavaScript (de)serialization to use with WebSockets.

With Node.js you can also open Sockets and talk directly to Q. To make this work you need to tweak things a bit so I released a package called node-q .

https://github.com/cinovo/node-q

Example:

var nodeq = require(“node-q”);

nodeq.connect(“localhost”, 5000, function(err, con) {

if (err) throw err;

console.log(“connected”);

con.k(“sum”, [1, 2, 3], function(err, res) {

if (err) throw err;

console.log(“result”, res);

});

});

Have fun :)

2

Hi,
great work! I once wrote a node c++ module using the c interface, but with all the node changes it doesnt compile anymore. 

So I just checked your package. I think its really great, but I think I found some issues:

  • you assume that a whole message comes in one chunk in the data event  “this.socket.on(“data”, function(buffer) {.”

  unfortunately this is not always correct. Tcp can split the data and so the data callback might be triggered mutliple times for one k ipc message. You have to buffer the data and check the length.

  • i think you have a race condition in your  message dispatcher:   assume you are sending a sync message from node. But while you do so, kdb sends you an upd.  in your code the activeRequestResponse is set to true.

  The solution is to correctly check the ipc header (see here: http://code.kx.com/wiki/Reference/ipcprotocol). I think you can go for the message type

Mabye you also want to allow multiple sync queries on parallel, this would be an important feature for a production environment i think. 

Keep on the great work!

Markus

3
  • you assume that a whole message comes in one chunk in the data event  “this.socket.on(“data”, function(buffer) {.”

  unfortunately this is not always correct. Tcp can split the data and so the data callback might be triggered mutliple times for one k ipc message. You have to buffer the data and check the length.

you can also receive multiple k-messages in one data chunk or even one and a half…

4

Hi Markus! If I remember you correctly we talked about using Node.js with Q / Kdb+ in Ireland in 2012? :)

You are right with the chunked packages. I will implement this as soon as possible. I copied your “issue” to https://github.com/cinovo/node-q/issues/1. Hope that’s okay.

Mixing sync and async requests is not a good idea. You are right. I will check if I can use the 2nd byte in the header to sort out the async messages.

But I am still not sure how you will keep track of replies if you send multiple requests. Because as far as I know there is no such thing as a correlation id in the headers? Modyfying http://code.kx.com/wiki/Reference/dotzdotpg is not allowed to solve the task :)

Michael from Stuttgart

2014-10-23 11:53 GMT+02:00 Markus Sieber <sieber@boerse-go.de>:

Hi,
great work! I once wrote a node c++ module using the c interface, but with all the node changes it doesnt compile anymore. 

So I just checked your package. I think its really great, but I think I found some issues:

  • you assume that a whole message comes in one chunk in the data event  “this.socket.on(“data”, function(buffer) {.”

  unfortunately this is not always correct. Tcp can split the data and so the data callback might be triggered mutliple times for one k ipc message. You have to buffer the data and check the length.

  • i think you have a race condition in your  message dispatcher:   assume you are sending a sync message from node. But while you do so, kdb sends you an upd.  in your code the activeRequestResponse is set to true.

  The solution is to correctly check the ipc header (see here: http://code.kx.com/wiki/Reference/ipcprotocol). I think you can go for the message type

Mabye you also want to allow multiple sync queries on parallel, this would be an important feature for a production environment i think. 

Keep on the great work!

Markus

5

I also fixed the problem with multiple messages in one chunk. Version 0.2.0 is coming. If you have time to check the source code that will be super :)

2014-10-23 15:21 GMT+02:00 Michael Wittig <michael.wittig@tullius-walden.com>:

Hi Markus! If I remember you correctly we talked about using Node.js with Q / Kdb+ in Ireland in 2012? :)

You are right with the chunked packages. I will implement this as soon as possible. I copied your “issue” to https://github.com/cinovo/node-q/issues/1. Hope that’s okay.

Mixing sync and async requests is not a good idea. You are right. I will check if I can use the 2nd byte in the header to sort out the async messages.

But I am still not sure how you will keep track of replies if you send multiple requests. Because as far as I know there is no such thing as a correlation id in the headers? Modyfying http://code.kx.com/wiki/Reference/dotzdotpg is not allowed to solve the task :)

Michael from Stuttgart

2014-10-23 11:53 GMT+02:00 Markus Sieber <sieber@boerse-go.de>:

Hi,
great work! I once wrote a node c++ module using the c interface, but with all the node changes it doesnt compile anymore. 

So I just checked your package. I think its really great, but I think I found some issues:

  • you assume that a whole message comes in one chunk in the data event  “this.socket.on(“data”, function(buffer) {.”

  unfortunately this is not always correct. Tcp can split the data and so the data callback might be triggered mutliple times for one k ipc message. You have to buffer the data and check the length.

  • i think you have a race condition in your  message dispatcher:   assume you are sending a sync message from node. But while you do so, kdb sends you an upd.  in your code the activeRequestResponse is set to true.

  The solution is to correctly check the ipc header (see here: http://code.kx.com/wiki/Reference/ipcprotocol). I think you can go for the message type

Mabye you also want to allow multiple sync queries on parallel, this would be an important feature for a production environment i think. 

Keep on the great work!

Markus

6

Hi Michael,

You are probably right, I was there 2012 :)

> But I am still not sure how you will keep track of replies if you send multiple requests

This is easy: sync requests are answered in the correct order (message type is 2 - response). That means the first message sent is the first one answered.

Async Messages are not answered. 

Markus

7

0.5.0 supports multiple requests:) I also wrote a blog post about node-q (in German) here http://blog.michaelwittig.info/q-interfacing-with-node-js/

Another open problem ist that I have to convert the Buffer to ArrayBuffer and vice versa because c.js is implemented for browsers and Buffer is a Node.js specific thing. I already optimized this by using the memcpy NPM but it will still be better to read / write directly to Buffer.

But I am not sure if performance is really such a big problem here for real use cases where Node.js is used as a tier between WAN/Web and LAN/q. Users will mostly subscribe to aggregated feeds and make queries with far less than millions of results.

I already implemented qipc in erlang and that’s the reason why I get started with c.js:) c.js is tested and used by many people. Without a proper test suite it is very hard to know that the protocol is implemented correctly.

Charly send me a Node.js implementation. I will check if I can replace c.js with that code to remove the Buffer <> ArrayBuffer overhead in the future.

Michael

2014-10-23 16:59 GMT+02:00 Markus Sieber <sieber@boerse-go.de>:

Hi Michael,

You are probably right, I was there 2012 :)

> But I am still not sure how you will keep track of replies if you send multiple requests

This is easy: sync requests are answered in the correct order (message type is 2 - response). That means the first message sent is the first one answered.

Async Messages are not answered. 



Markus

–
Submitted via Google Groups