real time processing of comma delimited files

The following is a comma delimited file that is collected in real timefor the stock UURAF. A detailed quote is collected every time thevolume changes (meaning a trade has happened).10:14:44,0.3540,-0.0290,134988,-7.57,0.3610,0.3420,0.3343,0.3830,0.3500,0.371010:41:47,0.3540,-0.0290,135988,-7.57,0.3610,0.3420,0.3343,0.3830,0.3500,0.371011:04:49,0.3542,-0.0288,145488,-7.52,0.3610,0.3420,0.3343,0.3830,0.3500,0.356011:06:49,0.3500,-0.0330,148988,-8.62,0.3610,0.3420,0.3343,0.3830,0.3500,0.357011:08:49,0.3500,-0.0330,157688,-8.62,0.3610,0.3420,0.3343,0.3830,0.3500,0.357011:15:50,0.3491,-0.0339,163688,-8.85,0.3610,0.3420,0.3343,0.3830,0.3500,0.357011:17:50,0.3570,-0.0260,164428,-6.79,0.3610,0.3420,0.3343,0.3830,0.3500,0.3570I want to calculate the moving average of the most recent 5 and 10trades.How would I create an empty array and then update the array every timea new trade happens?This has been a sticking point for me and my novice understanding ofkdb.thanks

Hi elmer,how come that you post this question also in the J-Forum.Nevertheless, what you need to do is to send the data to a kdb instance everytime there is an update.You can implement this using either python, c++, java or excel.Regards,KimAm 18.12.2011 17:37, schrieb butch:> The following is a comma delimited file that is collected in real time> for the stock UURAF. A detailed quote is collected every time the> volume changes (meaning a trade has happened).>> 10:14:44,0.3540,-0.0290,134988,-7.57,0.3610,0.3420,0.3343,0.3830,0.3500,0.3710> 10:41:47,0.3540,-0.0290,135988,-7.57,0.3610,0.3420,0.3343,0.3830,0.3500,0.3710> 11:04:49,0.3542,-0.0288,145488,-7.52,0.3610,0.3420,0.3343,0.3830,0.3500,0.3560> 11:06:49,0.3500,-0.0330,148988,-8.62,0.3610,0.3420,0.3343,0.3830,0.3500,0.3570> 11:08:49,0.3500,-0.0330,157688,-8.62,0.3610,0.3420,0.3343,0.3830,0.3500,0.3570> 11:15:50,0.3491,-0.0339,163688,-8.85,0.3610,0.3420,0.3343,0.3830,0.3500,0.3570> 11:17:50,0.3570,-0.0260,164428,-6.79,0.3610,0.3420,0.3343,0.3830,0.3500,0.3570>> I want to calculate the moving average of the most recent 5 and 10> trades.>> How would I create an empty array and then update the array every time> a new trade happens?>> This has been a sticking point for me and my novice understanding of> kdb.>> thanks>

.z.ts in combo with .Q.fs (if the file is too big)

On Sun, Dec 18, 2011 at 11:39 PM, Kim Kuen Tang wrot=
e:
>
> Hi elmer,
>
> how come that you post this question also in the J-Forum.
>
> Nevertheless, what you need to do is to send the data to a kdb instance
> everytime there is an update.
> You can implement this using either python, c++, java or excel.
>
> Regards,
> Kim
>
> Am 18.12.2011 17:37, schrieb butch:
>
>> The following is a comma delimited file that is collected in real time
>> for the stock UURAF. =A0A detailed quote is collected every time the
>> volume changes (meaning a trade has happened).
>>
>>
>> 10:14:44,0.3540,-0.0290,134988,-7.57,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3710
>>
>> 10:41:47,0.3540,-0.0290,135988,-7.57,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3710
>>
>> 11:04:49,0.3542,-0.0288,145488,-7.52,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3560
>>
>> 11:06:49,0.3500,-0.0330,148988,-8.62,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3570
>>
>> 11:08:49,0.3500,-0.0330,157688,-8.62,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3570
>>
>> 11:15:50,0.3491,-0.0339,163688,-8.85,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3570
>>
>> 11:17:50,0.3570,-0.0260,164428,-6.79,0.3610,0.3420,0.3343,0.3830,0.3500,=
0.3570
>>
>> I want to calculate the moving average of the most recent 5 and 10
>> trades.
>>
>> How would I create an empty array and then update the array every time
>> a new trade happens?
>>
>> This has been a sticking point for me and my novice understanding of
>> kdb.
>>
>> thanks
>>
>
> –
>

Submitted via Google Groups

If i understood your requirement correctly, the csv file is being
written by a separate process from the kdb process responsible for
computing moving avgs. For simplicity assume that the csv file has
only these columns sym, price, size and date. I also believe that you
are only interested in the price column, in case you require other
columns the below approach can be amended.

So here is my approach :-

calculateMovingAvgs :{[oldSize;newSize;fileHandle] data:raze (" F
“;”,") 0: (fileHandle;oldSize;newSize); .cache.latestPrices:-5

.cache.latestPrices,data;.cache.currentSize:newSize;.cache.movingAvgs:

avgs .cache.latestPrices}

calculateMovingAvgs[0;hcount :path/to/file ; :path/to/file]

Now set a timer function

.z.ts:{if[not .cache.currentSize =3D newSize:hcount :path/to/file; calculateMovingAvgs[.cache.currentSize;newSize;:path/to/file]]}

The advantage of this approach is that it will only read the new bytes
and not the entire file