Running user defined aggregation on partitioned tables

Community Support
,
1

https://learninghub.kx.com/forums/topic/running-user-defined-aggregation-on-partitioned-tables

Hello

I want to run a user defined aggregating function getPercentile (to get say 99% or 50% or 1% percentile of a list of value) on a partitioned table. From this link map-reduce , only few types of aggregation can KDB decompose into map step and reduce step (Although med is listed, it will throw a part error if being executed across partitions). Since I got lots of data in each partition, I don’t want to load them all into memory and apply the aggregation, so is there any way to apply user specific map-reduce for aggregation on partitioned tables?

Thanks!

2

To my knowledge, the best way you can do is to fetch the data into memory and apply the custom aggregate function to the retrieved data, like the below

 

update percentile:getPercentile price from select sym, price from trade where date>=.z.d-7 // and other filters

 

Certain function like sum and prd can reduce in a memory-effective way as they don’t need to keep the original values from each partition. However med or your getPercentile function cannot.