Has anybody done an Asof join (aj) over HDB?

Hi All,

I’ve found I need to do an aj over historical database, to retrieve either points in time or intervals of data. Realized quickly that aj doesn’t work straight out of the box. Has anybody done anything like this?

The logic I have is to group by points in time or intervals by sym and date, then for points, run a regular aj on the HDB, for the day but check if it doesn’t find anything, in which case, I need to decrement the date by one and run a where close of i=last i, till I find a date that actually has data (got to cover weekends). For intervals, I retreive the interval by aj and check if the first record has data and if not, do the same as described previously.

I’m also observing complications if trying to optimize retrieval by whole day per symbol.

Currently working on the code but wondering if somebody already done it and can save me a few hours.

For a real example, my HDB data has the following structure:

flip datesymtimeaskbidvaskvbid!(date$();0#;time$();real$();real$();real$();real$()) // this is the Dukascopy FX tick data HDB

while the retrieval is either a list of dates or a table with 2 dates, OpenDate and CloseDate (this being a set of FX trades).

Thanks,

John

Hi John

When you do an asof join on the HDB with symtime then it’s usually best to do it date at a time, and not do any subselects e.g. 

aj[symtime;mytrades;select from quotes where date=x]

One way to do it would be to start at the latest date and do the aj, then work backwards until the dataset is completely filled, populating more values as you go

// sample data set - small number of trades in the middle of the quotes

q)nt:10                                                                                                                                                                                        

q)trade:(date:2010.01.01+nt?5;sym:nt?ab`c)                                                                                                                                                 

q)nq:1000                                                                                                                                                                                      

q)quote:(date:asc 2009.12.01+nq?60;sym:g#nq?abc;bid:nq?100)      

// aj the data from quotes for a specific day against the t table

ajq:{[t;q;d] aj[symdate;t;select from q where date=d]}

// where t has a null bid value (that check will probably need to be changed) 

// aj on the quotes and fill in the current table in the correction positions

ajqr:{[t;q;d] @[t;i;:;ajq[t i:where null t`bid;q;d]]}

// using a seed value of the trade table joined with the quotes for the maximum trade date, iterate until all trades are filled (all bids are non-null)

// start at the maximum date and move backwards

// (you’ll need to change the stop condition to make sure it doesn’t run infinitely)

q)first {[c;q] (ajqr[c 0;q;c 1]; -1+c 1)}[;quote]/[{0<count select from x[0] where null bid};(ajq[trade;quote;max tradedate];-1+max tradedate)]                                                

date       sym bid


2010.01.02 a   36 

2010.01.01 a   43 

2010.01.03 c   63 

2010.01.05 b   56 

2010.01.02 c   54 

2010.01.03 b   49 

2010.01.01 c   15 

2010.01.02 c   54 

2010.01.02 b   17 

2010.01.03 a   48 

You can see the intermediate results as it builds with

show each first each {[c;q] (ajqr[c 0;q;c 1]; -1+c 1)}[;quote][{0<count select from x[0] where null bid};(ajq[trade;quote;max tradedate];-1+max tradedate)] 

Thanks 

Jonny

AquaQ Analytics