Hi All,
Can someone please help me using q for data manipulation
My Requirement:
I have a huge logfile and i need to grep log file for a particular
word, if that particular word is present then the whole line need to
be fetched and should be manipulated further to get individual values
which has to be inserted into KDB.
Please post me few built in functions that kdb/q has for data
manipulation.
Also please advice me how far q is going to help me in data
manipulation before inserting it into KDB
–
Submitted via Google Groups
If it’s small enough to fit in memory (and the two billion list
limit), you can do
{x where x like “*foo*”}read0 f
to get lines of f containing foo.
If not, look into .Q.fs.
Once you have the lines, you can use “vs” to split them along
delimiters, “cut” to split them at fixed offsets, etc.
I’ve often found q to be an excellent tool for manipulating text, as
long as it’s in fairly fixed form. (For really free-form text, use
perl. :) )
The only real limitation is that there’s no built-in regexp support,
only very basic pattern-matching.
Thanks a lotOn May 5, 8:16?pm, Aaron Davies <aaron.dav…> wrote:> If it’s small enough to fit in memory (and the two billion list> limit), you can do>> {x where x like “foo”}read0 f>> to get lines of f containing foo.>> If not, look into .Q.fs.>> Once you have the lines, you can use “vs” to split them along> delimiters, “cut” to split them at fixed offsets, etc.>> I’ve often found q to be an excellent tool for manipulating text, as> long as it’s in fairly fixed form. (For really free-form text, use> perl. :) )>> The only real limitation is that there’s no built-in regexp support,> only very basic pattern-matching.>>>>>> On Wednesday, May 5, 2010, Pravin <pravin198…> wrote:> > Hi All,>> > Can someone please help me using q for data manipulation>> > My Requirement:>> > I have a huge logfile and i need to grep log file for a particular> > word, if that particular word is present then the whole line need to> > be fetched and should be manipulated further to get individual values> > which has to be inserted into KDB.> > Please post me few built in functions that kdb/q has for data> > manipulation.> > Also please advice me how far q is going to help me in data> > manipulation before inserting it into KDB>> > –> >
Submitted via Google Groups</pravin198…></aaron.dav…>
Hi Aaron and all,Your above idea is working fine, could you please help me in findingout the same for more than one matching strings in same line. I triedwith various option but not working{x where x like “*foo*”, x like “*hoo*”}read0 fThanks in AdvanceOn May 10, 12:38?pm, Pravin <pravin198…> wrote:> Thanks a lot>> On May 5, 8:16?pm, Aaron Davies <aaron.dav…> wrote:>>>>>> > If it’s small enough to fit in memory (and the two billion list> > limit), you can do>> > {x where x like “foo”}read0 f>> > to get lines of f containing foo.>> > If not, look into .Q.fs.>> > Once you have the lines, you can use “vs” to split them along> > delimiters, “cut” to split them at fixed offsets, etc.>> > I’ve often found q to be an excellent tool for manipulating text, as> > long as it’s in fairly fixed form. (For really free-form text, use> > perl. :) )>> > The only real limitation is that there’s no built-in regexp support,> > only very basic pattern-matching.>> > On Wednesday, May 5, 2010, Pravin <pravin198…> wrote:> > > Hi All,>> > > Can someone please help me using q for data manipulation>> > > My Requirement:>> > > I have a huge logfile and i need to grep log file for a particular> > > word, if that particular word is present then the whole line need to> > > be fetched and should be manipulated further to get individual values> > > which has to be inserted into KDB.> > > Please post me few built in functions that kdb/q has for data> > > manipulation.> > > Also please advice me how far q is going to help me in data> > > manipulation before inserting it into KDB>> > > –> > >
Submitted via Google Groups</pravin198…></aaron.dav…></pravin198…>
Hi Aaron/all,Your above idea is working fine, Could you plese help me to match morethan one string in the same line. I tried with various option its notwoking, only option working for me is to copy output to another fileusing 0: and repeat above syntax with different string,Could you please help to do this in same lineThanks in advanceOn May 10, 12:38?pm, Pravin <pravin198…> wrote:> Thanks a lot>> On May 5, 8:16?pm, Aaron Davies <aaron.dav…> wrote:>>>>>> > If it’s small enough to fit in memory (and the two billion list> > limit), you can do>> > {x where x like “foo”}read0 f>> > to get lines of f containing foo.>> > If not, look into .Q.fs.>> > Once you have the lines, you can use “vs” to split them along> > delimiters, “cut” to split them at fixed offsets, etc.>> > I’ve often found q to be an excellent tool for manipulating text, as> > long as it’s in fairly fixed form. (For really free-form text, use> > perl. :) )>> > The only real limitation is that there’s no built-in regexp support,> > only very basic pattern-matching.>> > On Wednesday, May 5, 2010, Pravin <pravin198…> wrote:> > > Hi All,>> > > Can someone please help me using q for data manipulation>> > > My Requirement:>> > > I have a huge logfile and i need to grep log file for a particular> > > word, if that particular word is present then the whole line need to> > > be fetched and should be manipulated further to get individual values> > > which has to be inserted into KDB.> > > Please post me few built in functions that kdb/q has for data> > > manipulation.> > > Also please advice me how far q is going to help me in data> > > manipulation before inserting it into KDB>> > > –> > >
Submitted via Google Groups</pravin198…></aaron.dav…></pravin198…>