Can I clear the symbol in the process

Community Support
1

Hi, I have a data loading job load data from external sources to HDB. After the data loaded, the memory usage growth a lot, and the memory doesn’t return to the system even after I delete all the temp table created. 

I suspect during the data loading process, a lot of symbol created and it doesn’t release after the job finish, just wonder, can I clear the symbol in the Q process?

2

no..consider redesigning your schema if this is the case. 

further clarification here..

https://groups.google.com/forum/#!searchin/personal-kdbplus/symw$20clean/personal-kdbplus/wclPCspiEYs/h6jeA0zZraQJ

Thanks,

Sean

3

does the result from

.Q.wsymssymw

confirm your suspicions?

4

Thanks Charles, yes, it is, the `symw take up most of the space

5

Hi all, like to bring up an old thread. 

We spent sometime and identified few column that better to saving in Character array ( string ) rather than symbol. However, long like we need to export all old records to CSV and then import again, is it? Is there better way to solve this issue?

We’ve check for this script http://code.kx.com/wiki/Contrib/CompactingHdbSym , but it run very slow and doesn’t that useful. Will anyone have better suggestion for me?

6

To perform data conversion you don’t have to go thru CSV as an intermediate format. You can perform in-memory conversion and write the results to disk. You can also send the old data to a new server over the network.

7

Right, sure, but what I mean is if there any tool that can do something like 

castcol:{[dbdir;table;col;newtype] 

And it will also clean up the sym file, rather than dump the data and load it again in whatever way. 

8

<canlwsqxmjf_hhreegpshxzzduqf_kdyjzkm>
<cacfbovcqoet0cum4capgid8v58y-xw2_l4cj_h2rkqo3qno9tw>


<84f92eb9-2c2b-4072-9eb9-91be764ae669@googlegroups.com>
To: Carfield Yim , personal-kdbplus@googlegroups.com

Hi Carfield,

A few months ago, I put a script at the bottom of the page you posted.=E2=80=8E It's multithreaded (so run with -s) and quite performant.

But fundamentally you have to start the sym file from scratch, which is non-trivial.

Feel free to try out the script and contribute any suggestions by editing the page.

David

From: Carfield Yim
Sent: Monday, June 15, 2015 18:52
To: personal-kdbplus@googlegroups.com
Reply To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] Can I clear the symbol in the process

Right, sure, but what I mean is if there any tool that can do something like

castcol:{[dbdir;table;col;newtype]

And it will also clean up the sym file, rather than dump the data and load it again in whatever way.

On Monday, June 15, 2015 at 8:18:11 PM UTC+8, Yan Yan wrote:
To perform data conversion you don't have to go thru CSV as an intermediate format. You can perform in-memory conversion and write the results to disk. You can also send the old data to a new server over the network.