Subject: can hdb symbol cols be enumerated to file NOT called sym ?Message-Id: Date: Mon, 29 Jan 2018 03:57:59 -0500To: personal-kdbplus@googlegroups.comMime-Version: 1.0 (Mac OS X Mail 9.3 (3124))X-Mailer: Apple Mail (2.3124)`:sym_2017 ? table.sym_colWill this work ?
On Monday, January 29, 2018 at 8:58:55 AM UTC, Science Student wrote:
`:sym_2017 ? table.sym_col
Will this work ?
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym. It is possible to have different sym files across different directories in a database which should cover most cases of saving down data. Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration.
See inline
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym( does \l /db only look for a filename sym ? ). It is possible to have different sym files across different directories in a database which should cover most cases of saving down data ( how? ). Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. ( you mean instead of using a \l /db ? )
Kx says it is possible to have a different named file. I was wondering on how is it possible without too much customization ?
It actually is possible to fairly simply enumerate against a different named sym file. Take the following for example:<o:p></o:p>
<o:p> </o:p>
q)\mkdir splay<o:p></o:p>
q)\cd splay<o:p></o:p>
q)where 11=type@'flip trades // unenumerated symbol columns in trades table<o:p></o:p>
symsrc<o:p></o:p>
q):trades/ set @[trades;where 11=type@'flip trades;:sym_2017?] // use functional apply to enumerate symbol columns<o:p></o:p>
`:trades/<o:p></o:p>
q)\<o:p></o:p>
<o:p> </o:p>
~ $ cd splay/<o:p></o:p>
~/splay $ ls # no sym file present<o:p></o:p>
sym_2017 trades<o:p></o:p>
<o:p> </o:p>
~/splay $ q .<o:p></o:p>
KDB+ 3.5 2017.08.22 Copyright (C) 1993-2017 Kx Systems<o:p></o:p>
l64/ 8()core 16048MB jmcmurray homer.aquaq.co.uk 127.0.1.1 EXPIRE 2018.06.30 AquaQ #50170<o:p></o:p>
<o:p> </o:p>
q)5#trades<o:p></o:p>
time sym src price size<o:p></o:p>
-------------------------------------------------<o:p></o:p>
2018.01.29D08:00:53.319000000 ORCL N 32.23 1099<o:p></o:p>
2018.01.29D08:01:26.649000000 YHOO O 35.52 471<o:p></o:p>
2018.01.29D08:01:36.413000000 YHOO L 35.52 438<o:p></o:p>
2018.01.29D08:01:41.763000000 YHOO N 35.51 1947<o:p></o:p>
2018.01.29D08:02:09.322000000 AAPL N 25.31 513<o:p></o:p>
q)5#exec src from trades // note custom enumeration on src column<o:p></o:p>
sym_2017$NOLNN<o:p></o:p>
q)5#exec sym from trades<o:p></o:p>
sym_2017$ORCLYHOOYHOOYHOOAAPL<o:p></o:p>
<o:p> </o:p>
Basically when you load a directory in KDB (e.g. an HDB), KDB will attempt to load every file in there, so any KDB binary files (e.g. sym vectors) will get loaded into memory, and will be loaded into variables with the same name as the filename. In this example sym & src are enumerated against sym_2017 so when this file is loaded with this name, it works.<o:p></o:p>
<o:p> </o:p>
The commonly used .Q.en function is hardcoded to enumerate against a file called sym, so if you want to use that function with a different filename, you’d have to create a modified version.<o:p></o:p>
<o:p> </o:p>
q).Q.en<o:p></o:p>
k){[s;d;x]if[#f@:&{$[11h=@*x;&/11h=@:'x;11h=@x]}‘x f:!+x;(/:d,sym)??,/?:’{$[0h=@x;,/x;x]}'x f];@[x;f;{$[0h=@y;(-1_+\0,#:'y)_x[sym;,/y];x[sym;y]]}s]}[?]<o:p></o:p>
<o:p> </o:p>
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Science Student
Sent: 29 January 2018 09:32
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] Re: can hdb symbol cols be enumerated to file NOT called sym ?<o:p></o:p>
<o:p> </o:p>
See inline<o:p></o:p>
<o:p> </o:p>
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym( does \l /db only look for a filename sym ? ). It is possible to have different sym files across different directories in a database which should cover most cases of saving down data ( how? ). Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. ( you mean instead of using a \l /db ? )<o:p></o:p>
<o:p> </o:p>
Kx says it is possible to have a different named file. I was wondering on how is it possible without too much customization ?<o:p></o:p>
<o:p> </o:p>
On Jan 29, 2018, at 4:19 AM, Sean Hehir <seanhehir15@gmail.com> wrote:<o:p></o:p>
<o:p> </o:p>
On Monday, January 29, 2018 at 8:58:55 AM UTC, Science Student wrote:<o:p></o:p>
`:sym_2017 ? table.sym_col
Will this work ? <o:p></o:p>
<o:p> </o:p>
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym. It is possible to have different sym files across different directories in a database which should cover most cases of saving down data. Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. <o:p></o:p>
<o:p> </o:p>
–
Submitted via Google Groups
Great :-) Thanks.
I see you are using the ? operator, right.
Yes, the ? operator is used for enumerating a vector, either against another in-memory vector (passed as a symbol e.g. sym_2017) or an on-disk vector (passed as an hsym e.g. :sym_2017)<o:p></o:p>
<o:p> </o:p>
$ can also be used for enumeration, however this will not extend the enum vector with new symbols encountered i.e. to use $ you would need to do something like:
:sym_2017 set distinct raze (tradessym;trades`src)<o:p></o:p>
<o:p> </o:p>
To populate the enum vector before enumerating.<o:p></o:p>
<o:p> </o:p>
With ?, any symbols that arent already in the enum vector will be added automatically.<o:p></o:p>
<o:p> </o:p>
<o:p> </o:p>
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Science Student
Sent: 29 January 2018 10:17
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] can hdb symbol cols be enumerated to file NOT called sym ?<o:p></o:p>
<o:p> </o:p>
Great :-) Thanks.<o:p></o:p>
I see you are using the ? operator, right.<o:p></o:p>
<o:p> </o:p>
On Jan 29, 2018, at 5:09 AM, Jonathon McMurray <jonathon.mcmurray@aquaq.co.uk> wrote:<o:p></o:p>
<o:p> </o:p>
It actually is possible to fairly simply enumerate against a different named sym file. Take the following for example:<o:p></o:p>
<o:p></o:p>
q)\mkdir splay<o:p></o:p>
q)\cd splay<o:p></o:p>
q)where 11=type@'flip trades // unenumerated symbol columns in trades table<o:p></o:p>
symsrc<o:p></o:p>q)
:trades/ set @[trades;where 11=type@'flip trades;:sym_2017?] // use functional apply to enumerate symbol columns<o:p></o:p>`:trades/<o:p></o:p>
q)\<o:p></o:p>
<o:p></o:p>
~ $ cd splay/<o:p></o:p>
~/splay $ ls # no sym file present<o:p></o:p>
sym_2017 trades<o:p></o:p>
<o:p></o:p>
~/splay $ q .<o:p></o:p>
KDB+ 3.5 2017.08.22 Copyright (C) 1993-2017 Kx Systems<o:p></o:p>
l64/ 8()core 16048MB jmcmurray homer.aquaq.co.uk 127.0.1.1 EXPIRE 2018.06.30 AquaQ #50170<o:p></o:p>
<o:p></o:p>
q)5#trades<o:p></o:p>
time sym src price size<o:p></o:p>
-------------------------------------------------<o:p></o:p>
2018.01.29D08:00:53.319000000 ORCL N 32.23 1099<o:p></o:p>
2018.01.29D08:01:26.649000000 YHOO O 35.52 471<o:p></o:p>
2018.01.29D08:01:36.413000000 YHOO L 35.52 438<o:p></o:p>
2018.01.29D08:01:41.763000000 YHOO N 35.51 1947<o:p></o:p>
2018.01.29D08:02:09.322000000 AAPL N 25.31 513<o:p></o:p>
q)5#exec src from trades // note custom enumeration on src column<o:p></o:p>
sym_2017$NOLNN<o:p></o:p>q)5#exec sym from trades<o:p></o:p>
sym_2017$ORCLYHOOYHOOYHOOAAPL<o:p></o:p><o:p></o:p>
Basically when you load a directory in KDB (e.g. an HDB), KDB will attempt to load every file in there, so any KDB binary files (e.g. sym vectors) will get loaded into memory, and will be loaded into variables with the same name as the filename. In this example sym & src are enumerated against sym_2017 so when this file is loaded with this name, it works.<o:p></o:p>
<o:p></o:p>
The commonly used .Q.en function is hardcoded to enumerate against a file called sym, so if you want to use that function with a different filename, you’d have to create a modified version.<o:p></o:p>
<o:p></o:p>
q).Q.en<o:p></o:p>
k){[s;d;x]if[#f@:&{$[11h=@*x;&/11h=@:'x;11h=@x]}‘x f:!+x;(
/:d,sym)??,/?:’{$[0h=@x;,/x;x]}'x f];@[x;f;{$[0h=@y;(-1_+\0,#:'y)_x[sym;,/y];x[sym;y]]}s]}[?]<o:p></o:p><o:p></o:p>
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Science Student
Sent: 29 January 2018 09:32
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] Re: can hdb symbol cols be enumerated to file NOT called sym ?<o:p></o:p><o:p></o:p>
See inline<o:p></o:p>
<o:p></o:p>
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym( does \l /db only look for a filename sym ? ). It is possible to have different sym files across different directories in a database which should cover most cases of saving down data ( how? ). Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. ( you mean instead of using a \l /db ? )<o:p></o:p>
<o:p></o:p>
Kx says it is possible to have a different named file. I was wondering on how is it possible without too much customization ?<o:p></o:p>
<o:p></o:p>
On Jan 29, 2018, at 4:19 AM, Sean Hehir <seanhehir15@gmail.com> wrote:<o:p></o:p>
<o:p></o:p>
On Monday, January 29, 2018 at 8:58:55 AM UTC, Science Student wrote:<o:p></o:p>
`:sym_2017 ? table.sym_col
Will this work ? <o:p></o:p>
<o:p></o:p>
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym. It is possible to have different sym files across different directories in a database which should cover most cases of saving down data. Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. <o:p></o:p>
<o:p></o:p>
–
Submitted via Google Groups
Well with $ there will manual code into writing the new list back to file which I presume ? is doing underneath the covers (or perhaps using native code). Your thoughts ?
Yes, if you wanted to use $ youd have to also have some code to extend the enumeration vector as you go. But I dont think theres any reason to do so when you can simply use ? and have this done for you.<o:p></o:p>
<o:p> </o:p>
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Science Student
Sent: 29 January 2018 10:30
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] can hdb symbol cols be enumerated to file NOT called sym ?<o:p></o:p>
<o:p> </o:p>
Well with $ there will manual code into writing the new list back to file which I presume ? is doing underneath the covers (or perhaps using native code). Your thoughts ?<o:p></o:p>
<o:p> </o:p>
On Jan 29, 2018, at 5:24 AM, Jonathon McMurray <jonathon.mcmurray@aquaq.co.uk> wrote:<o:p></o:p>
<o:p> </o:p>
Yes, the ? operator is used for enumerating a vector, either against another in-memory vector (passed as a symbol e.g.
sym_2017) or an on-disk vector (passed as an hsym e.g.:sym_2017)<o:p></o:p><o:p></o:p>
$ can also be used for enumeration, however this will not extend the enum vector with new symbols encountered i.e. to use $ you would need to do something like:
:sym_2017 set distinct raze (tradessym;trades`src)<o:p></o:p><o:p></o:p>
To populate the enum vector before enumerating.<o:p></o:p>
<o:p></o:p>
With ?, any symbols that arent already in the enum vector will be added automatically.<o:p></o:p>
<o:p></o:p>
<o:p></o:p>
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Science Student
Sent: 29 January 2018 10:17
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] can hdb symbol cols be enumerated to file NOT called sym ?<o:p></o:p><o:p></o:p>
Great :-) Thanks.<o:p></o:p>
I see you are using the ? operator, right.<o:p></o:p>
<o:p></o:p>
On Jan 29, 2018, at 5:09 AM, Jonathon McMurray <jonathon.mcmurray@aquaq.co.uk> wrote:<o:p></o:p>
<o:p></o:p>
It actually is possible to fairly simply enumerate against a different named sym file. Take the following for example:<o:p></o:p>
<o:p></o:p>
q)\mkdir splay<o:p></o:p>
q)\cd splay<o:p></o:p>
q)where 11=type@'flip trades // unenumerated symbol columns in trades table<o:p></o:p>
symsrc<o:p></o:p>q)
:trades/ set @[trades;where 11=type@'flip trades;:sym_2017?] // use functional apply to enumerate symbol columns<o:p></o:p>`:trades/<o:p></o:p>
q)\<o:p></o:p>
<o:p></o:p>
~ $ cd splay/<o:p></o:p>
~/splay $ ls # no sym file present<o:p></o:p>
sym_2017 trades<o:p></o:p>
<o:p></o:p>
~/splay $ q .<o:p></o:p>
KDB+ 3.5 2017.08.22 Copyright (C) 1993-2017 Kx Systems<o:p></o:p>
l64/ 8()core 16048MB jmcmurray homer.aquaq.co.uk 127.0.1.1 EXPIRE 2018.06.30 AquaQ #50170<o:p></o:p>
<o:p></o:p>
q)5#trades<o:p></o:p>
time sym src price size<o:p></o:p>
-------------------------------------------------<o:p></o:p>
2018.01.29D08:00:53.319000000 ORCL N 32.23 1099<o:p></o:p>
2018.01.29D08:01:26.649000000 YHOO O 35.52 471<o:p></o:p>
2018.01.29D08:01:36.413000000 YHOO L 35.52 438<o:p></o:p>
2018.01.29D08:01:41.763000000 YHOO N 35.51 1947<o:p></o:p>
2018.01.29D08:02:09.322000000 AAPL N 25.31 513<o:p></o:p>
q)5#exec src from trades // note custom enumeration on src column<o:p></o:p>
sym_2017$NOLNN<o:p></o:p>q)5#exec sym from trades<o:p></o:p>
sym_2017$ORCLYHOOYHOOYHOOAAPL<o:p></o:p><o:p></o:p>
Basically when you load a directory in KDB (e.g. an HDB), KDB will attempt to load every file in there, so any KDB binary files (e.g. sym vectors) will get loaded into memory, and will be loaded into variables with the same name as the filename. In this example sym & src are enumerated against sym_2017 so when this file is loaded with this name, it works.<o:p></o:p>
<o:p></o:p>
The commonly used .Q.en function is hardcoded to enumerate against a file called sym, so if you want to use that function with a different filename, you’d have to create a modified version.<o:p></o:p>
<o:p></o:p>
q).Q.en<o:p></o:p>
k){[s;d;x]if[#f@:&{$[11h=@*x;&/11h=@:'x;11h=@x]}‘x f:!+x;(
/:d,sym)??,/?:’{$[0h=@x;,/x;x]}'x f];@[x;f;{$[0h=@y;(-1_+\0,#:'y)_x[sym;,/y];x[sym;y]]}s]}[?]<o:p></o:p><o:p></o:p>
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Science Student
Sent: 29 January 2018 09:32
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] Re: can hdb symbol cols be enumerated to file NOT called sym ?<o:p></o:p><o:p></o:p>
See inline<o:p></o:p>
<o:p></o:p>
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym( does \l /db only look for a filename sym ? ). It is possible to have different sym files across different directories in a database which should cover most cases of saving down data ( how? ). Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. ( you mean instead of using a \l /db ? )<o:p></o:p>
<o:p></o:p>
Kx says it is possible to have a different named file. I was wondering on how is it possible without too much customization ?<o:p></o:p>
<o:p></o:p>
On Jan 29, 2018, at 4:19 AM, Sean Hehir <seanhehir15@gmail.com> wrote:<o:p></o:p>
<o:p></o:p>
On Monday, January 29, 2018 at 8:58:55 AM UTC, Science Student wrote:<o:p></o:p>
`:sym_2017 ? table.sym_col
Will this work ? <o:p></o:p>
<o:p></o:p>
While it is possible to save the symbols down in that way as a file you won’t be able to load back in the table as you would if you had a file called sym. It is possible to have different sym files across different directories in a database which should cover most cases of saving down data. Another option would be to create a custom load function that looks specifically for a user defined sym file for the enumeration. <o:p></o:p>
<o:p></o:p>
–
Submitted via Google Groups
Appreciate your help Jonathon :-)