Newbie question.. I’m streaming data into kdb+ via C and want to storeit to disk. I naively used the following:`:/path/to/mydb insert newRowwhich quickly became unwieldy as the table grew. I assumed kdb+would append to the table, but from the performance it looks like itmust be overwriting the entire file.Is there an efficient way to simply append to a file? Or do I need towrite code to do batch inserts?Thanks,Josh
you can append to a splayed table relatively efficiently
:splay/ set .Q.en[:.;(s:10?`1;a:til 10)]
:splay/ upsert .Q.en[:.;(s:10?`1;a:til 10)]
however you’ll want to bulk up data for each upsert anyway to reduce file ops.
It will rewrite columns that have an attribute (supg), so defer adding an attribute until you have finished inserting data.
if that’s too slow, you can look at kdb+tick
On 5 November 2014 16:11 UTC, Charles Skelton wrote:> you can append to a splayed table relatively efficiently>> :splay/ set .Q.en[:.;(s:10?1;a:til 10)]> :splay/ upsert .Q.en[:.;([]s:10?1;a:til 10)]>> however you’ll want to bulk up data for each upsert anyway to reduce file> ops.> It will rewrite columns that have an attribute (supg), so defer adding an> attribute until you have finished inserting data.>Thanks. Splaying the table did the trick. I also noticed that "upsert"was almost 2x faster than “insert”. I didn’t expect it to be anydifferent on an unkeyed table..