# Question about streaming file compression

**URL:** <https://forum.kx.com/t/question-about-streaming-file-compression/11640>\
**Category:** Community Support\
**Tags:** kdb-and-q\
**Created:** [November 27, 2017, 9:35am UTC](https://forum.kx.com/t/question-about-streaming-file-compression/11640 "2017-11-27T09:35:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![carfield11](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@carfield11](https://forum.kx.com/u/carfield11)\
**Post date:** [November 27, 2017, 9:35am UTC](https://forum.kx.com/t/question-about-streaming-file-compression/11640/1 "2017-11-27T09:35:00Z")

</div>

Hi all

From&nbsp;[http://code.kx.com/wiki/Cookbook/FileCompression](http://code.kx.com/wiki/Cookbook/FileCompression) , We can use “set” the stream file compressed. However, in our existing data loader, the data of same date come in difference time, so we are not using “set” but “upsert” to write data to HDB folder, like this

// r is the table with data like to write to HDB, dst is the HDB root folder

s:update `#sym from .Q.en[DST] r;

dst:` sv .Q.par[DST;d;tabName],`;

dst upsert s;

For HDB persist logic like that, how can I still do streaming file compression?

Regards,

Carfield

---

<div class="post-metadata">

**Author:** ![Jonathon\_McMurr](https://avatars.discourse-cdn.com/v4/letter/j/f17d59/32.png) [@Jonathon\_McMurr](https://forum.kx.com/u/Jonathon_McMurr)\
**Post date:** [November 27, 2017, 10:56am UTC](https://forum.kx.com/t/question-about-streaming-file-compression/11640/2 "2017-11-27T10:56:00Z")

</div>

Hi Carfield\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

If you upsert to a directory on disk that is already compressed, the upsert will also be compressed\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

In other words, where you upsert, you can instead check if the directory exists, and if it doesn’t, use set with the compression parameters, something like e.g.\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

$[()~key dst;(dst;17;2;6) set;dst upsert] s;\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

Hope that helps\<o:p\>\</o:p\>

Jonathon\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

**From:** [personal-kdbplus@googlegroups.com](mailto:personal-kdbplus@googlegroups.com) [[mailto:personal-kdbplus@googlegroups.com](mailto:personal-kdbplus@googlegroups.com)] **On Behalf Of** Carfield Yim  
**Sent:** 27 November 2017 09:36  
**To:** Kdb+ Personal Developers \<personal-kdbplus@googlegroups.com\>  
**Subject:** [personal kdb+] Question about streaming file compression\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

Hi all\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

From&nbsp;http://code.kx.com/wiki/Cookbook/FileCompression , We can use “set” the stream file compressed. However, in our existing data loader, the data of same date come in difference time, so we are not using “set” but “upsert” to write data to HDB folder, like this\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

// r is the table with data like to write to HDB, dst is the HDB root folder\<o:p\>\</o:p\>

s:update `#sym from .Q.en[DST] r;\<o:p\>\</o:p\>

dst:` sv .Q.par[DST;d;tabName],`;\<o:p\>\</o:p\>

dst upsert s;\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

For HDB persist logic like that, how can I still do streaming file compression?\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

Regards,\<o:p\>\</o:p\>

Carfield\<o:p\>\</o:p\>

\<o:p\> \</o:p\>

–  
Submitted via Google Groups

---

<div class="post-metadata">

**Author:** ![carfield11](https://avatars.discourse-cdn.com/v4/letter/c/ec9cab/32.png) [@carfield11](https://forum.kx.com/u/carfield11)\
**Post date:** [December 1, 2017, 2:57pm UTC](https://forum.kx.com/t/question-about-streaming-file-compression/11640/3 "2017-12-01T14:57:00Z")

</div>

Thanks a lot, it is very helpful