I am trying to understand how the memory management works in kdb using the simple example below (and the free version of kdb). The following function (getDay) loads 1 days data and saves it down to a csv file. I would expect the memory to be released after every function call since all references are lost.
What I observe is that I get a wsfull message after running it for 30+ days (each day is is small, ~80mb when stored as csv). I’ve tried to add garbage collection a delete table inside the function but it doesn’t seem to help. Is there another reason the memory keeps accumulating?
SaveTableDataAsText: {[path; table; delimiter] hsym[path] 0: 1 _ delimiter 0: table};
getDay:{
aa:select from ticks where date=x;
SaveTableDataAsText[`$“C:/backup/”,string,“.csv”;aa;“,”];};
getMonth each date;
Did you try .Q.gc or -g 1 flags? (rather than just deleting a table)
This might help: http://www.aquaq.co.uk/q/garbage-collection-kdb/
Thanks
Jonny
.Q.gc is automatically invoked whenever wsful is encountered anyway.
What do you see if you print .Q.w after each save? Paying particular attention to the syms,symw fields..
I looked at syms and symw during each run of the loop and it does appear that it is growing steadily. I’m not entirely sure why these are increasing – what is your hunch? If it matters, ticks is a splay table.
Jonny – I did try both your suggestions prior to my post and neither worked.
some increase will be ok. What are the absolute values you see?
symbols should be used for repeating string values only.
Strings that do not repeat/are unique such as orderids should be parsed into char vectors or another type such as a guid.
I added a print statement for syms and symw in the loop and this is the result I got after running it:
q)syms 5059 symw 191712
syms 316551 symw 118006080
syms 556072 symw 209350803
syms 783043 symw 297798534
syms 1077277 symw 407012107
syms 1336126 symw 503486013
syms 1559396 symw 594480749
syms 1919556 symw 728054955
syms 2279436 symw 860739681
syms 2587780 symw 975445366
syms 2859161 symw 1093028962
wsfull
It looks like symbols aren’t the issue?
With >1GB of address space used for interned symbols, I’d say it is likely the problem - on w32 you have only 3-4GB for your complete address space, and with 1GB already used/fragmented, the remaining is quite constrained.
i.e. you are using symbol type for a column in your schema for string data which contains too many unique values.
If you restart your process and try to continue, does it progress further or does it encounter wsful almost immediately? If the schema is not the problem, then maybe you are creating transient unique symbols somewhere in your code.
If I restart the process, it continues on for a bit before it does the wsfull.
The columns in the original splay table have a sym file t hat is relatively small (<1mb), so the symbol columns shouldn’t be a problem? There is only one column that I am morphing from the original splay table:
orderID:{$[null `$x;“”;$[x like “”;“”;((!/)"I=|"0:x)[11]]]}'[FixMessage]
which does seem to be the problem (when I remove it, symw does not grow). I wasn’t aware that doing a `$x would add to the memory. Thanks for the help!