I have multiple compressed (zipped) kdb files on disk, sorted by timestamp. Is there a way to open all of these files and stream all rows to some processing function in time sorted order across all files? in other words, i would need to block the read until next_timestamp (in file)<=clock_timestamp (in processor)
I think this is easiest done using a similar design of kdb+tick (or even, just use kdb+tick directly). The key is to suspend data publishing from reader to processor until certain event. Therefore, one solution is to write a simple feedhandler (as a file reader), which reads the input files, find out the next timestamp in order, wait until that time before publishing the data into tickerplant (if you don’t need recovery, etc., you may even publish directly into the processor). On the recipient side, on the other hand, there is no extra handling required as you can safely assumed that the data is arriving at the expected time.