Currently using a basic tickerplant. It’s logging data at end of day – how do I access that data such that I can perform a lookback? I’m hoping to load all the data upon starting a process so that when data pumps in I’m able to reference live data and historical (really only looking for high/low/close).
I’ve looked for links refering to HDB and loading, but there doesn’t seem to be much information.
Is it the same process as loading a normal table?
Any help appreciated!
Am I better off saving the data to a basic table and then loading that table as part of my starting process for the function?
Hi paintrade,
If I understand your question correctly I think you would like to have access to data both in memory and on disk.
In the tick set up in kdb data is partitioned into multiple process so it is not straight forward to access both historical and real-time data in one query. Multiple reasons exist for this, such as memory footprint, available storage and hardware.
The standard approach to this problem is to create a gateway process which will query your real-time and historical data and join them together. A good description of this process can be seen at:
https://code.kx.com/q/wp/common_design_principles_for_kdb_gateways.pdf
Examples of such implementations can be found on github.
Best regards,
Joseph Griffiths
For this type of task people will generally build a gateway to serve both the real-time data (from the rdb) and the historical data (from the hdb).
Some good examples here: https://code.kx.com/q/wp/common_design_principles_for_kdb_gateways.pdf
Terry
Thanks, Terry. I’m going to read through the PDF this morning.
Hey Joe,
Thanks for hoping in. Please correct me if I’m wrong, couldn’t I just write a process to load the data into memory at EOD flush? The memory usage would be a downside, but if I’m only looking to hold high/low/close I don’t think it’ll be that big of a deal.
Obviously not the prettiest way to do it, but perhaps a quick hackaround while I try and wade through the gateway stuff.
Hi paintrade,
If I understand correctly you could just read in your log files at the end of day and from there create a table and select high, low etc. from the table. I’d avoid doing this directly in the TP as you may end up broadcasting the things coming in from the log.
I think a better idea would be to create a new simple subscriber that just defines upd, just t insert x, to collect all the data. Then from there you could just create your summary table as part of .u.end. From this you could use :
dpft[directory;partition;p#field;tablename]} `
to save this down.
I hope this can be of use,
Joseph Griffiths
Hey Joseph,
Thanks!
Oddly enough, as I was playing around with the historical logs I found that my symbols were saving as numbers.
Going to have to investigate this before I move forward.
Load in the sym file into that process
That did it. Thanks!