How to replay csv file one line by one line

Community Support
1

Hi, Masters:
  I have some csv files, the format such as:

    timestamp, bid, ask, volumn

    09:00:00,2,3,100

    09:00:01,3,4,200

    09:00:02,3,5,300

    …

  How to replay the data one line by one line into kdb? Or do you have suggestions? Thanks so much!

Zheng

2

Depending on the size of your file, you could read it all into memory at once, as a table. http://code.kx.com/q/cookbook/faq/#how-do-i-import-a-csv-file-into-a-table

3

Why exactly do you want to replay the .csv file line by line? When you say replay do you mean load into kdb+?

Starting with this table: 

table:(time:asc 1000000?.z.P;sym:1000000?abcdef`ghi;price:1000000?100f)

and saving it down: 

`:table.csv 0: csv 0: table

The following options are available to you: 

If you just want to load the csv file into kdb you can do: 

(“SSF”;enlist",")0:`table.csv

If your .csv file is really big you can use .Q.fs or .Q.fsn 

.Q.fs[{table::(“SSF”;enlist",")0:x}]`:table.csv

If you really want to load it line by line, even though I wouldn’t recommend this, you can use over and do it this way: 

{x:x,y}/[0#(“SSF”;enlist",")0::table.csv;("SSF";enlist",")0:table.csv]


Let me know if this helps you!

Regards!

Alex

4

You could either read the csv into memory in one go, or read through it trunk by trunk. (Both of these methods are described in the documentations.)

Once you get the data, may it be all in one go or trunk by trunk, you can replay it whichever way you prefer.

5

Alex, thanks!
Yes, the process of your method is working. It can be load into kdb+.

You asked “Why you want to replay the .csv file line by line?”. It is because I want to implement a replay service to playback what I had recorded based on the market real tick-by-tick data. Actually, I can load the whole csv file into kdb+, but my purpose is replay the data line-by-line based on the timestamp, such as: 09:00:00, replay line 1; 09:00:01, replay line 2; 09:00:05, replay line 3; and so on. I prefer to have another table to save the streaming data.

6

Yes, you’re right. I can read the whole csv file into kdb+ without problem.
I just want to replay it line by line based on the recorded timestamp.

7

Flying, thanks!
Do you mean the document that Collin mentioned?

http://code.kx.com/q/cookbook/faq/#how-do-i-import-a-csv-file-into-a-table