merge/copy partitioned tables efficiency

Community Support
,
1

https://learninghub.kx.com/forums/topic/merge-copy-partitioned-tables-efficiency

Hello all,

Currently, I would need to merge/copy partitioned tables from several individual folders to a common large folder:

kdbfolder_1 -  - trade - col_1,...,col_n

...
kdbfolder_n -  - trade - col_1,...,col_n

--> 

kdbcommon -  - trade - col_1,...,col_n

The columns of all individual tables are the same, and I did the merging by q codes, in which, for each individual folder, read the individual table data, and then “upsert” the data to the final common folder:

q) t:select from `:kdbfolder_n//trade;

q) `:kdbcommon//trade/ upsert t;

This works properly, but the speed seems to be slow when running “upsert”, especially, when the table size in the common folder grows very large.

I would like to ask if there is a more efficient way to perform the merging/copying of the data?

2

Hi ,

You appear to be upserting at a table level, you could try doing the upserts at a column level and peach it. something like:

{colData:get .Q.dd[`:kdbfolder_n//trade;x];upsert[.Q.dd[`:kdbcommon//trade/;x];colData]}peach except[cols trade;`date]

For this you need to have threads enabled in the command line:

-s N

Let me know if this works,

Megan

3

Hello Megan,

It’s working. Thanks so much for your help