HDB data flushed off when call u.end[.z.d] twice on ticker plant.

Community Support
1

Hi All,

  I have a simple ticker plant stack.

  

  ticker plant :  rlwrapq tick.q sym /home/kx/hdb -p 5000

  HDB : rlwrap q hdb.q /home/kx/hdb/sym -p 5002

  RDB : rlwrap q r.q localhost:5000 localhost:5002 -p5001 

And a simple feed script.


the logs and data are stored in  /home/kx/, below is the tree of /home/kx

hdb/

??? sym
? ??? 2019.07.25
? ? ??? quote
? ? ? ??? asize
? ? ? ??? ask
? ? ? ??? bid
? ? ? ??? bsize
? ? ? ??? ex
? ? ? ??? mode
? ? ? ??? sym
? ? ? ??? time
? ? ??? trade
? ? ??? cond
? ? ??? ex
? ? ??? price
? ? ??? size
? ? ??? stop
? ? ??? sym
? ? ??? time
? ??? sym
??? sym2019.07.25

When I am calling u.end[.z.d] on ticker plant. The data is being saved in HDB. Having a size of about " 595M 2019.07.25/
When I am calling u.end[.z.d] twice tThe data is being flushed off from HDB. The HDB files size is 88K 2019.07.25/

Then I need to recover the data from sym logs using -11!.

Could anyone have a clue why it is happed so? I feel I am doing something wrong but not able to identify.
How I can save the data permanently in HDB at every 30 minutes (not the end of the day).
and How I can query HDB data from ticker plant/RDB.

Thanks and Regards,
S Pratap

2

For a vanilla setup, .u.end[.z.d] is writing the data in the RDB to the hdb for the date .z.d, then flushing the RDB. By calling it again you overwriting the previously written data with the contents of the RDB, that have just been flushed and are therefore much smaller.

3

Thanks, Callum.
Understood my mistake. I am rewriting the content of RDB to HDB.

Is there any way to write the data from RDB to HDB at every 30/15 minutes without overwriting HDB.

Regards,

S Pratap

4

Well, EOD should only have a single trigger source and it should only occur once per day. Are you calling it manually for a reason?
I personally opt to have a process running on the HDB only. You could then adjust your EOD logic to check if a date exists before writing. A more simplistic, but less robust solution could be achieved by editing the rdb end function like so (note I have not tested this)

end:{$[()~key .u.lastEOD;.u.lastEOD:.z.d;if[x=.u.lasEOD;show “EOD already run”;:()]];t:tables.;t@:where g=attr each t@:sym;.Q.hdpf[$“:”,.u.x 1;:.;x;sym];@[;sym;g#] each t;}`

5

Hi Sheroo,

This whitepaper on intraday writedowns may be helpful: https://code.kx.com/v2/wp/intraday_writedown_solutions.pdf

Thanks,

Rebecca

Rebecca Kelly | Kx Technical Evangelist | ML COE US | Kx |  (646) 630-4381 | rebecca@kx.com

""

6

Thanks a lot, Callum and Rebecca.
It helped a lot.

Regards,

Pratap