running an aj on a very large data set. command keeps failing. Is there a better/more efficient want to aj? Thx.
when you say ‘failing’ does that mean it throws an exception? please provide an example
do you mean it returns the wrong results? ensure your data is sorted by time (within equal sym partitions)
do you mean it runs for a very long time?
ensure that the second table (third argument) has a p or g attribute on the column corresponding to the first join key (first argument).
assuming you have trades in a table t, and quotes in a table q, the typical aj command would be:
t:aj[symtime;t] q
in case you lost your `p attribute on q:
t:aj[symtime;t] update `p#sym from q
quoting from “Q Tips”
Joining with a table that is lacking a p or g attribute on the first exact match column will lead to
extremely slow joins. In the best case, this will take minutes if not hours to complete, in the worst case it
will crash your machine.
…
Again, as we discussed in Section 8.2 [97], applying proper attributes is critical for good join performance.
While kdb+s optimizations can typically compensate for missing attributes when using lj, pj,
ij and uj, both aj and wj are unforgiving.
a simple solution is just divide it into smaller groups, like
{ aj[`time;select from trade where date=x,sym=y;select from quote where date=x,sym=y } ./: dates cross syms