See memory of table with attributes

Community Support
1

Is there a way to determine how much memory a table with attributes occupies?

So here’s an example of what I’m talking about:

I start off with the following memory:

q).Q.w

used| 119680

heap| 67108864

peak| 67108864

wmax| 0

mmap| 0

mphy| 9419714560

syms| 574

symw| 18675

I then create a table with the partition attribute:

q)t:(a:`p#til 1000000;b:1000000?0 1 2)

I then have the following memory profile:

q).Q.w

used| 75617312

heap| 201326592

peak| 201326592

wmax| 0

mmap| 0

mphy| 9419714560

syms| 574

symw| 18675

So there’s been an increase of 75497632 bytes. However the function -22! returns a much smaller amount:

q)-22!t

16000039

Is there a function similar to -22! that would return the full amount of memory t (and overhead related to t )takes up in memory? I know that I could delete t from the global namespace and check .Q.w before and after, but I’m hoping for a way to see the memory without destroying t. 

Any help is greatly appreciated!

Thanks!

2

Hi Mike,

-22! just gives the memory used by values of t and does not include memory usage by attributes. For your table you can calculate the same value as below:

total integer values=2000000

total memory=8*2000000=16000000

And there will be some overhead to store table info.

Not sure if there is any function to get full memory info but one way is to store the table on disk as splayed table and check the memory usage of it either directly from disk or by mapping the table to service (mmap value in .Q.w). That will give you total memory usage including memory used by attributes data structure.

For this, you can create your own function which will save table on disk, get the memory usage and deletes the table from disk.

3

Would count[t]#t also copy the attributes?

If so, monitoring memory usage before and after a # would give you some insight. 

4

Thinking about it again, it requires a slightly more involved solution. A better function is:

f: { memBefore: .Q.w[used] ; t1:{@[x;y;#[z]]}/[1 _ 0!x;key attributes;attributes:exec last a by c from meta x where not null a]; memChange: .Q.w[][used] - memBefore; memChange }

t:(a:`p#til 1000000;b:1000000?0 1 2)

f t

75497664

This function will create a full, temporary copy of the table in memory.

Memory is allocated in blocks of 2^, increasing the table size to 1200000 will dramatically increase the size in memory. Comparing here on a table with and without attributes.

q)f:{memBefore: .Q.w[used] ; t1:{@[x;y;#[z]]}/[1 _ 0!x;key attributes;attributes:exec last a by c from meta x where not null a]; memChange: .Q.w[][used] - memBefore; memChange }

q)t:(a:til 1200000;b:1200000?0 1 2)

q)-22!t

19200039

q)f t

33554592

q)update p#a from t

`t

q)-22!t

19200039

q)f t

150995136

Thanks,
Matthew

5

for a vector with no attr, vector capacity is power of 2
`long$2 xexp ceiling 2 xlog 16+elementSize*count

e.g. til 100000000
q)`long$2 xexp ceiling 2 xlog 16+8*100000000
1073741824

see section 42 of http://kx.com/q/d/q.htm for info on attr space usage.

42 Attributes

example overhead
s#2 2 3 sorted 0 u#2 4 5 unique 16*u
p#2 2 1 parted (4\*u;16\*u;4\*u+1) g#2 1 2 grouped (4*u;16*u;4*u+1;4*n)
The byte overheads use n(number of elements) and u(number of uniques).

Those attr usage numbers need updating for v3.x. I think they would be
u#2 4 5 unique 32\*u p#2 2 1 parted (8*u;32*u;8*u+1)
`g#2 1 2 grouped (8*u;32*u;8*u+1;8*n)

So for your example
q){long$2 xexp ceiling 2 xlog 16+(8*count x)+sum 8 32 8*count distinct x}ta
67108864
q){long$2 xexp ceiling 2 xlog 16+8*count x)}tb
8388608
q)67108864+8388608
75497472

if you’re targeting the general case, ref counting allows same object to be present in multiple containers; observe ref count with -16!x.

This should be sufficient information as a basis for someone to cook up a function that works for any object. If you do, please share it with the community.

6

Hi all,

I’ve been working on this functionality for our TorQ framework and wrote a blog about this problem:

http://www.aquaq.co.uk/q/adventure-in-retrieving-memory-size-of-kdb-object/

It has been quite enjoyable working on this and I’ll be excited to see what you guys think about it :)

Best,

WooiKent Lee

Financial Software Developer

AQUAQ Analytics

"This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. 

If you are not the named addressee or if you have received this email in error, you may not, without the consent of AquaQ Analytics, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. 

Unless separately agreed, AquaQ Analytics does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of AquaQ Analytics and AquaQ Analytics shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed"

On Wednesday, November 25, 2015 at 2:32:51 PM UTC, Charles Skelton wrote:

for a vector with no attr, vector capacity is power of 2
`long$2 xexp ceiling 2 xlog 16+elementSize*count

e.g. til 100000000
q)`long$2 xexp ceiling 2 xlog 16+8*100000000
1073741824

see section 42 of http://kx.com/q/d/q.htm for info on attr space usage.

42 Attributes

example overhead
s#2 2 3 sorted 0 u#2 4 5 unique 16*u
p#2 2 1 parted (4\*u;16\*u;4\*u+1) g#2 1 2 grouped (4*u;16*u;4*u+1;4*n)
The byte overheads use n(number of elements) and u(number of uniques).

Those attr usage numbers need updating for v3.x. I think they would be
u#2 4 5 unique 32\*u p#2 2 1 parted (8*u;32*u;8*u+1)
`g#2 1 2 grouped (8*u;32*u;8*u+1;8*n)

So for your example
q){long$2 xexp ceiling 2 xlog 16+(8*count x)+sum 8 32 8*count distinct x}ta
67108864
q){long$2 xexp ceiling 2 xlog 16+8*count x)}tb
8388608
q)67108864+8388608
75497472

if you’re targeting the general case, ref counting allows same object to be present in multiple containers; observe ref count with -16!x.

This should be sufficient information as a basis for someone to cook up a function that works for any object. If you do, please share it with the community.