KDB Serialization Examples and 32 bit message length field

Community Support
1

I’m trying to upgrade a “version 3” library to a “version 6” compatible library.

  • Version “3” or “6” refers to the single byte which is sent as part of the “hello” message. When a client connects to a KDB server, it sends an initial packet of data which describes which version of the protocol it supports. (See Interprocess communication | Developing | kdb+ and q documentation - kdb+ and q documentation)
  • In the case of the library I am working on, it used to send “hello\3\0” when first connecting. I have changed it to “hello\6\0” as part of my upgrade changes.

I wanted to inspect some serialization examples from a real KDB+ system. Some examples are also provided on one of the KX Systems documentation pages. (See Serialization examples | Knowledge Base | kdb+ and q documentation - kdb+ and q documentation)

Here are a couple of examples which I ran.

q)-8!1
0x0100000011000000f90100000000000000
q)-8!1i
0x010000000d000000fa01000000

In both cases, this appears to be 4 bytes of data followed by a message length which is also 4 bytes. The remaining data follows.

What surprises me is that the message length is 4 bytes (32 bit) and not 8 bytes (64 bit).

Note: These serialization examples are independent of the protocol version. The information about the protocol (version 3, 6 or other) is not included here. However, protocol version 6 supports messages larger than 2147483647 bytes (2 GB), which is the largest possible signed 32 bit number.

Can you understand my confusion? The message length field is a (signed) 32 bit value, which has a maximum of ~ 2e+9. And yet, protocol version 6 supports messages up to 1 TB, or ~ 1e+12 bytes.

So what is going on here? Is there a way to inspect the binary data produced when sending a request over IPC?

My end objective is to debug and fix some remaining issues with the library code I am working with and trying to upgrade. Currently my changes do not work and I do not get any data back from KDB suggesting that KDB doesn’t even recognize the initial message I am sending to it.

2

Please check this commit on what changes were needed for qPython to add support for protocol version 6. The size field was not increased, but one of the previously unused bytes from the header has been repurposed into an extension size byte, so the size is 5 bytes total instead of 8.