# How to avoid "wsfull" error by process small part of data each time?

**URL:** <https://forum.kx.com/t/how-to-avoid-quot-wsfull-quot-error-by-process-small-part-of-data-each-time/7135>\
**Category:** Community Support\
**Tags:** kdb-and-q\
**Created:** [April 26, 2011, 1:39am UTC](https://forum.kx.com/t/how-to-avoid-quot-wsfull-quot-error-by-process-small-part-of-data-each-time/7135 "2011-04-26T01:39:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![bovine](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bovine](https://forum.kx.com/u/bovine)\
**Post date:** [April 26, 2011, 1:39am UTC](https://forum.kx.com/t/how-to-avoid-quot-wsfull-quot-error-by-process-small-part-of-data-each-time/7135/1 "2011-04-26T01:39:00Z")

</div>

I am using a trial version kdb+, and I have met the following problem:

E:\q\>w32\q.exe loadquote.q  
KDB+ 2.7 2011.04.12 Copyright (C) 1993-2011 Kx Systems  
w32/ 4()core 4095MB superb superb-pc 172.31.183.5 PLAY 2011.07.11

wsfull

In loadquote.q, the related code snippet is:

loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba\_mb\_”;“”];h1:`sv `:`db,(`$string d),`quote1;h:` sv `:`db,(`$string d),`quote`;`code`time xasc h1;@[h1;`code;`p#];h upsert select from 0! exec (`$“”,/:string columns)!(datatype!val)columns by date,code,time  
from select from quote1 where date=d}

/ bas contains all file name of raw data  
loadba2 each bas

Basically it is following the guide in “pivot table” (https://  
[code.kx.com/trac/wiki/pivot](http://code.kx.com/trac/wiki/pivot)). d is the date, h1 is a temporary quote  
table to be pivoted and h is the pivoted table I wanted. But the  
problem here is that all quote data of all stocks (identified by code)  
of a single date is too large to fit into memory.

I’ve tried some ways to avoid this problem, for example:

1. using a loop to deal with part of stock each time inside loadba2  
function  
loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba\_mb\_”;“”];h1:`sv `:`db,(`$string d),`quote1;h:` sv `:`db,(`$string d),`quote`;`code`time xasc h1;@[h1;`code;`p#];cb:0;while[cb\<9999;ce:cb +1000;h upsert select from 0!exec (`$“”,/:string columns)!(datatype!  
val)columns by date,code,time from select from quote1 where  
date=d,code\>=cb,code  
2. manually copy the code to deal with small part of stock  
loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba\_mb\_”;“”];h1:` sv<br>`:`db,(`$string d),`quote1;h:` sv `:`db,(`$string<br>d),`quote`;`code`time xasc h1;@[h1;`code;`p#];h upsert select from 0!<br>exec (`$“”,/:string columns)!(datatype!val)columns by date,code,time  
from select from quote1 where date=d,code\>=0,code\<2000;.Q.gc}  
  
loadba2 each bas  
  
loadba2:{[fn] d:“D”$ssr[fn;“:e:/hkexdata/bidask/ba\_mb\_”;“”];h1:` sv<br>`:`db,(`$string d),`quote1;h:` sv `:`db,(`$string<br>d),`quote`;`code`time xasc h1;@[h1;`code;`p#];h upsert select from 0!<br>exec (`$“”,/:string columns)!(datatype!val)columns by date,code,time  
from select from quote1 where date=d,code\>=2000,code\<4000;.Q.gc}  
  
loadba2 each bas  
  
…  
  
But none of this work.  
  
For my solution 1, I cannot even have data of one date written  
successfully. For my solution 2, I can get 2 days quote for stock code  
between 0 and 2000 written successfully but failed with “wsfull” after  
that.  
  
I really don’t know how kdb+ decide which object is “unreferenced” and  
can be released, I tried .Q.gc but not working either. Any idea of  
how to handle my problem here? Thanks!