.Q.en with huge amount of symbols

Community Support
1

I made test DB with 1mil+ symbols.
Next time when I use .Q.en on empty table it consume high amount of cpu(1 core full load).

Any comments please.

eg I use \t:1000

and z.ts with func inside:

"TradesHeader: ([]Time:time$();Symbol:symbol$();Price:float$();Size:int$();Dir:byte$();OI:int$();Id:int$();Ex:symbol$());
Result below:

Blue - CPU thread load for q

Red - Total CPU load

2

symbol type is suitable for repeating strings. Other datatypes should be used for strings that are unique or repeat infrequently.

.Q.en reloads the sym file, then appends new symbols. You are probably obersving the cost of creating 1M+ symbols (first run interns them, further runs look up in the hastable).

sym mem usage can be seen via .Q.w.

3

Yes, all my symbols repeating and used often, Symbol - instrument name, Ex - exchange name.
I was surprised that CPU load was on empty table when nothing was to enumerate.

If .Q.en reload sym file it explains a lot.

Does other way to enumerate symbol? I mean without reloading them to avoid further CPU loads.

Something like checking new symbol in hashtable, next append if necessary to hashtable and to disk sym file.

It should be faster.

4

Hi Vadim,

You could create a modified .Q.en to enumerate against an in-memory variable:

q)t:( $string til 1200300) q)\t:100 :u/ upsert .Q.en[:.;t] 26118 q)k)qen:{if[#f@:&11h=@:'x f:!+x;sym??,/?:'x f;];@[x;f;sym?]} q)\t:100 :v/ upsert qen t
9818

It has more than halved the time in this case, though make sure to save the sym to disk.
While giving the same result:

q)\l .
q)(select from u)~(select from v)
1b

This does mean however that you can’t have multiple processes writing to the same folder as there is now no automatic file lock on the sym file.

If you really wanted to offload the cpu cost from the q process you could enumerate the data in the loading process.

Regards,
Ryan

–
www.timestored.com - kdb training
Tel: +44 (0) 20 3397 7535
Email: ryan@timestored.com

5

Thanks Ryan!

It solved problems.

I found strange situation with \x

when I remove .z.zd using system"x " it not completed, looks like a bug.

ps I want to do upsert(11h variable) to a sym file, but I use compression and should disable it before upserting.

Does other way exist to do this in function?

6

q).z.zd:{}

q)system"x .z.zd"

q).z.zd

'.z.zd

7

So simple -)
thanks.

2015-01-21 14:49 GMT+04:00 Charles Skelton <charlie@kx.com>:

q).z.zd:{}

q)system"x .z.zd"

q).z.zd

'.z.zd