Saving with auto partition

Community Support
1

Hi.
At current moment I’m saving all incoming data directly to partitions(Date) which selected from my c# application(every record checking Date)…

Can it possible to save automatically all data?

Simple example what I mean, look at table:

2014.07.06 23:50:00.000 somedata

2014.07.06 23:55:00.000 somedata

2014.07.07 00:05:00.000 somedata

2014.07.06 23:57:00.000 somedata

2014.07.06 00:07:00.000 somedata

Save it to 2 partition 2014.07.06 and 2014.07.07 by generic ability of q?

If not, can it possible to do with premade func(example please)?

2

yes…have a look at .Q.dcfgnt…http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles.

More explanation here: https://groups.google.com/forum/#!searchin/personal-kdbplus/save$20all$20partitions/personal-kdbplus/bb_AosPGMHk/71JDykKS4HUJ


Thanks,

Sean

3

Thank, it what I wanted.

One thing I saw that after saving on disk, partition column saved too.

Better will be if it will be deleted because partition already contains that information.

But I have not skills to update this function -)

4

Vadim…this is untested, but works on my test db.

tab:(date:10?2014.07.06 2014.07.07;time:10?.z.t;vol:10?100f);

\d .Q

k)dpfnt:{[d;p;f;n;t];if[~&/qm’r:+en[d]t;'unmappable];{[d;t;i;x]@[d;x;:;t[x]]}[d:par[d;p;n];r;\<r f]'!r;@[;f;p#]@[d;`.d;:;r@&~f=r:!r];n}

k)dcft:{[d;c;f;t]*(dpfnt[d;;f;t]).‘{[t;c;p](p;?[. t;,(=;c;p);0b;a!a:.q.except[.q.cols . t;c]])}[t;c]’?[;();();c]?[`. t;();1b;(,c)!,c]}

\d .

.Q.dcft[:.;date;;tab]

`tab

Slight mods to Aarons…removed the reference to the partition column in the .d file, and the writing process. If you have time can you test and get back to me.

Cheers,

Sean

5

Not works at my pc. Got error:

6

Hi Vadim

You could use the attached script to save down data directly to directory. The first step you should perform in your q session is to load in
your raw data with the correct data types. We can use your data layout in the initial post as an example:

e.g. rawData: flip (datetimesymprice)!0:[(“DTSF”;“,”);<rawDataFileLocation>]

Then once the raw data has been loaded into q, this must be enumerated like below where the result is saved to a directory path:

e.g. loadedData: .Q.en[dirPath;rawData]

dirPath could be in the form `:hdb/testdata/ for example.

You can now load the attached script by performing the below command into your q session:

\l dataSave.q

This script will save the data down to directory but without having the partition (date) column listed within your table. The data will by sectioned by date when saved under the directory path. The savefunc function uses five parameters which you should input when running the function (one of these being the loadedData enumerated table above).
Some features are that the function can sort and apply attributes to the data within your table. The function will append data to the generated datapath directory in the correct format. If there is a problem with saving the data down, the error will be caught and displayed.

Once the script is loaded, you can run the function in the following format:

savefunc[loadedData;loadedDate;hdbdir;tableNamePath;sortCol]

e.g. to save down data for one particular date, run:

savefunc[loadedData;2014.04.01;`:hdb/testdata;`trade;`sym]

e.g. to save down data for all dates within your table, run:

savefunc[loadedData;;:hdb/testdata;trade;sym] each exec distinct date$date from loadedData

Expected output can be seen below

Saving data to directory :hdb/testdata/2014.04.01/trade/
Saving data to directory :hdb/testdata/2014.04.02/trade/
:hdb/testdata/2014.04.01/trade/:hdb/testdata/2014.04.02/trade/

Hope this helps!
(Please see script attached)

Kind Regards

Natalie Inkpin

Financial Software Developer

AquaQAnalytics

7

copy and paste the code I wrote exactly as it is and it will work. The funcs are k funcs, and you have missed the k) prefix.

Thanks,

Sean

8

Hello Natalie
Thank you for your script, it helped me and I can modify it further.