Question about streaming file compression

Community Support
1

Hi all

From http://code.kx.com/wiki/Cookbook/FileCompression , We can use “set” the stream file compressed. However, in our existing data loader, the data of same date come in difference time, so we are not using “set” but “upsert” to write data to HDB folder, like this

// r is the table with data like to write to HDB, dst is the HDB root folder

s:update `#sym from .Q.en[DST] r;

dst: sv .Q.par[DST;d;tabName],;

dst upsert s;

For HDB persist logic like that, how can I still do streaming file compression?

Regards,

Carfield

2

Hi Carfield<o:p></o:p>

<o:p> </o:p>

If you upsert to a directory on disk that is already compressed, the upsert will also be compressed<o:p></o:p>

<o:p> </o:p>

In other words, where you upsert, you can instead check if the directory exists, and if it doesn’t, use set with the compression parameters, something like e.g.<o:p></o:p>

<o:p> </o:p>

$[()~key dst;(dst;17;2;6) set;dst upsert] s;<o:p></o:p>

<o:p> </o:p>

Hope that helps<o:p></o:p>

Jonathon<o:p></o:p>

<o:p> </o:p>

From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Carfield Yim
Sent: 27 November 2017 09:36
To: Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Subject: [personal kdb+] Question about streaming file compression<o:p></o:p>

<o:p> </o:p>

Hi all<o:p></o:p>

<o:p> </o:p>

From http://code.kx.com/wiki/Cookbook/FileCompression , We can use “set” the stream file compressed. However, in our existing data loader, the data of same date come in difference time, so we are not using “set” but “upsert” to write data to HDB folder, like this<o:p></o:p>

<o:p> </o:p>

// r is the table with data like to write to HDB, dst is the HDB root folder<o:p></o:p>

s:update `#sym from .Q.en[DST] r;<o:p></o:p>

dst: sv .Q.par[DST;d;tabName],;<o:p></o:p>

dst upsert s;<o:p></o:p>

<o:p> </o:p>

For HDB persist logic like that, how can I still do streaming file compression?<o:p></o:p>

<o:p> </o:p>

Regards,<o:p></o:p>

Carfield<o:p></o:p>

<o:p> </o:p>

–
Submitted via Google Groups

3

Thanks a lot, it is very helpful