Hi,
suppose I have a zip file containing a huge txt file with fixed length records. The zip file is 2GB, the uncompressed txt would be 20GB.
One way to load it would be to uncompress the zip and load the huge txt in chunks. This works and is great, but I would love to omit the uncompressing step and make kdb uncompress and parse directly (chunk by chunk).
v2.7 and above has some zipping magic but I did not see anywhere if parsing zipped files is possible.
Is it?
Knowing that all TAQ data from NYSE comes zipped these days, I would be surprised if noone else had the same wish before. Were they heard by the q gods?:)
And finally an even longer shot: what if the zip has two (more) txt files?
Thanks in advance,
Andras
You could use zlib (gzread) and wrap it in an so.On Dec 5, 1:45?am, andrew wall <aw…> wrote:> Hi Andras,> Most customers pay per core for a kdb licence. As such, they would usually> prefer to designate such OS style jobs to the OS via a wrapping script> using another application, which then invokes q. Do any of the standard> unzipping apps read chunks?> I think kdb’s capabilities with zip files is limited to splayed tables so> far - as far as I know, kx haven’t exposed read functionality for zips.> Cheers,> Andrew> On Dec 5, 2012 12:30 AM, “andrasbme” <andras.bo…> wrote:>>>> > Hi,> > suppose I have a zip file containing a huge txt file with fixed length> > records. The zip file is 2GB, the uncompressed txt would be 20GB.> > One way to load it would be to uncompress the zip and load the huge txt in> > chunks. This works and is great, but I would love to omit the uncompressing> > step and make kdb uncompress and parse directly (chunk by chunk).> > v2.7 and above has some zipping magic but I did not see anywhere if> > parsing zipped files is possible.> > Is it?> > Knowing that all TAQ data from NYSE comes zipped these days, I would be> > surprised if noone else had the same wish before. Were they heard by the q> > gods?:)> > And finally an even longer shot: what if the zip has two (more) txt files?>> > Thanks in advance,> > Andras>> > –> >
Submitted via Google Groups</andras.bo…></aw…>