selecting a subsection of a splayed table

Community Support
1

I have a large splayed table and would like to look at a subsection of data but I am running into memory issues when query by row. I know this is because it is a splayed table but I was wonder if anyone could give me advice on the best way to manipulate a subsection of the splayed tables

For example I create a large splayed table

// Create a large splayed table

N:60000000;

db_big: (ti:til N; p1:N?1000j; p2:N?1000j; p3:N?1000j);

`:c:/q/splay/db_big/ set db_big

// reload the table

delete from `.;

db_big: get `:c:/q/splay/db_big;

// if I try to include a large subset of rows then KDB crashed with memory issues

I: select i from db_big where p1 > 500;

count `db_big [I.x] // OK

I: select i from db_big where p1 > 10;

count `db_big [I.x]  // Crashes

For instance should I delete data column wise for each row, and then say the table to disk and then subsequently reload the subset of the splayed table.

Any advice appreciated.

Many thanks

David

2

try

db_big: get `:c:/q/splay/db_big/;

note that trailing /

which causes the cols to be mapped.

thanks

3

Fighting with 32bit addressability limits can be somewhat frustrating.

Note that you might like to use the command line option -g 1
to allow individual blocks of memory >32MB to be recycled back to the OS immediately.

And if you do happen to bloat the heap unintentionally, you can try to compact the heap using .Q.gc.

select keeps files mapped for the duration of the select. So if you run out of address space, you’ve little option but to select a single column at a time.

Or talk to sales@kx about licensing the 64bit version. ;-)

thanks