Hello Team,
I am working on the process to import the huge csv file data into kdb. I would like to add the error/exception handling logic to my script.
Please find the below code snippet that i am using.
chunks:{.Q.en[`$dir] (“DIJSS*”; enlist “,”) 0:x}
.Q.fs[{0N! .[path;();,; chunks x]} `$csvfile];
Here, in the above code how can i add the Error handling logic. How can i log the Error/Exception in to log.
Please help me on this.
Thanks in Advance.
Hi Victor,
Q supports try catch functionality using the . and @ operators in one of their many overloaded forms. Sample usage is given below, noting the distinction usage between functions monadic and polyadic functions.
q)f:{x+1}
q)g:{[x;y]x+y}
q)
q)
q)@[f;1;{ show x;}]
2
q)@[f;`2;{ show x;}]
“type”
q)
q)
q)f[`2]
{x+1}
'type
`2
1
q))
q))\
q)
q)
q)
q).[g;(1;2);{ show x;}]
3
q).[g;(1;`2);{ show x;}]
“type”
q)
q)
This simple example should extend to what you want.
You can find more information on try catch functionality here: http://code.kx.com/wiki/JB:QforMortals/execution\_control
under “Protected Evaluation”
Regards,
Connor
Hi Connor,
Thank you for your response. However, i am still having confusion that how can i use . and @ operators with .Q.fs function.
Do you have any idea ?
Regards,
Victor
Hi Victor,
To expand a little more:
.Q.fs takes two arguments. The first being a function to deal with chunks of a file and the second, the file name.
The file specified will be read in, as a string, and essentially each chunk passed as a string to the first argument, at which point you specify what logic is to be used to deal with the data. In your example I think you should do something like this (Ive chosen to insert data to a table in memory for the save of clarity) and I’ve created a counter so you can see exactly whats happening:
/Without error trapping initially.
/Note that the fancy logic is to account that my cvs contains a header row (which yours may or may not have.);
q)c
timesymsrcpriceamountside
q)p:();i:0;.Q.fs[{ p insert $[i=0;1_;::] flip c!"TSSFFS"$flip "," vs' x;i+:1};:trade.csv]
1494745
q)meta p
c | t f a
------| -----
time | t
sym | s
src | s
price | f
amount| f
side | s
q)
q)count p
40000
q)\wc -l trade.csv
" 40001 trade.csv"
q)
/With error trapping (around the main function) that will exit the routine as soon as it sees an issue without attempting to carry on.
q)p:();i:0;.[.Q.fs;({if[i=10;‘`oops]; `p insert $[i=0;1_;::] flip c!“TSSFFS”$flip “,” vs’ x;i+:1};`:trade.csv);{ show “error stopping”}]
“error stopping”
q)count p
346
q)i
10
q)/Note error trapping could be added to the first argument of .Q.fs if you still wished to processes the rest of the file, assuming one chunk had issues.
Regards,
Connor