How to avoid "wsfull" error by process small part of data each time?

Community Support
1

I am using a trial version kdb+, and I have met the following problem:

E:\q>w32\q.exe loadquote.q
KDB+ 2.7 2011.04.12 Copyright (C) 1993-2011 Kx Systems
w32/ 4()core 4095MB superb superb-pc 172.31.183.5 PLAY 2011.07.11

wsfull

In loadquote.q, the related code snippet is:

loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba_mb_”;“”];h1:sv :db,($string d),quote1;h: sv :db,($string d),quote;codetime xasc h1;@[h1;code;p#];h upsert select from 0! exec ($“”,/:string columns)!(datatype!val)columns by date,code,time
from select from quote1 where date=d}

/ bas contains all file name of raw data
loadba2 each bas

Basically it is following the guide in “pivot table” (https://
code.kx.com/trac/wiki/pivot). d is the date, h1 is a temporary quote
table to be pivoted and h is the pivoted table I wanted. But the
problem here is that all quote data of all stocks (identified by code)
of a single date is too large to fit into memory.

I’ve tried some ways to avoid this problem, for example:

  1. using a loop to deal with part of stock each time inside loadba2
    function
    loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba_mb_”;“”];h1:sv :db,($string d),quote1;h: sv :db,($string d),quote;codetime xasc h1;@[h1;code;p#];cb:0;while[cb\<9999;ce:cb +1000;h upsert select from 0!exec ($“”,/:string columns)!(datatype!
    val)columns by date,code,time from select from quote1 where
    date=d,code>=cb,code
    2. manually copy the code to deal with small part of stock
    loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba_mb_”;“”];h1: sv<br>:db,($string d),quote1;h: sv :db,($string<br>d),quote;codetime xasc h1;@[h1;code;p#];h upsert select from 0!<br>exec ($“”,/:string columns)!(datatype!val)columns by date,code,time
    from select from quote1 where date=d,code>=0,code<2000;.Q.gc}

    loadba2 each bas

    loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba_mb_”;“”];h1: sv<br>:db,($string d),quote1;h: sv :db,($string<br>d),quote;codetime xasc h1;@[h1;code;p#];h upsert select from 0!<br>exec ($“”,/:string columns)!(datatype!val)columns by date,code,time
    from select from quote1 where date=d,code>=2000,code<4000;.Q.gc}

    loadba2 each bas

    …

    But none of this work.

    For my solution 1, I cannot even have data of one date written
    successfully. For my solution 2, I can get 2 days quote for stock code
    between 0 and 2000 written successfully but failed with “wsfull” after
    that.

    I really don’t know how kdb+ decide which object is “unreferenced” and
    can be released, I tried .Q.gc but not working either. Any idea of
    how to handle my problem here? Thanks!