Question about storing large amounts of data per second

Community Support
1

There are 1024 real-time data per second, how to save al these 1024 floats into a partition table per second? The insert string is out of length…

Thanks ALL!

2

Not quite sure I understand the problem: are you saying that you have 1024 live/realtime messages per second and you want to write them every second to a partitioned table on disk? Or that you want to insert them into an in-memory table?

Can you give examples of what the records looks like?

3

You have quite a few options to connect to kdb.
Check out https://code.kx.com/q/interfaces/#kdb-as-server

Andras

4

not into in-mem table but a partitioned table, because of the real-time data is sensitive and have to flush them to disk  immediately.  

Thank u for response!

?2021?4?6??? UTC+8 ??11:58:28<TerryLynch> ???

Not quite sure I understand the problem: are you saying that you have 1024 live/realtime messages per second and you want to write them every second to a partitioned table on disk? Or that you want to insert them into an in-memory table?

Can you give examples of what the records looks like?

5

Thank u for response , I’m not quite sure but you mean send parameters by bytes stream ? or what…

?2021?4?7??? UTC+8 ??12:25:29<dot…@gmail.com> ???

You have quite a few options to connect to kdb.
Check out https://code.kx.com/q/interfaces/#kdb-as-server

Andras

6

I mean sending data through the q ipc protocol, yes.
I guess you used http get and you run into some limit of the url size? You can also split up the message btw, but I recommend using the ipc protocol.

András