Memory mapped -23!

Community Support
1

When you use -23!select from :t/** does it not copy the individual columns into memory like a plane   **select from :t/ ?

2

Hi, Science Student!

I had a look at the two methods of querying of what I understand to be an on-disk table.
I would like to point out that:

select from `:t/

does not load the columns into memory:

i.e:
  t:([]a:30000000?50000; b:3000000?500); /- new table

  :t/ set t**                              /- saving the table on disk       **.Q.w[**]    **used| 326160**        /- this is what we are looking at    **heap| 67108864    peak| 67108864    wmax| 0    mmap| 0    mphy| 16827965440    syms| 635    symw| 27543**       **tab:select from :t/
  
   .Q.w[]
   used| 328064              /- there is a very small increase in used memory
   heap| 67108864
   peak| 67108864
   wmax| 0
   mmap| 480000032
   mphy| 16827965440
   syms| 664
   symw| 28392
   
   tab:reverse select from :t** /  /- in this case the table is loaded into memory and as we can see                                  /- the memory used has increased a lot more       .**Q.w[]    used| 537198912    heap| 603979776    peak| 603979776    wmax| 0    mmap| 0    mphy| 16827965440    syms| 638    symw| 27628**       the **-23!select from :t/ does not seem to do anything different than the simple select from :t/.    I think both commands do just about the same thing.       here is an example of memory usage with **-23!select from :t/**

   .Q.w[]
   used| 326160
   heap| 67108864
   peak| 67108864
   wmax| 0
   mmap| 0
   mphy| 16827965440
   syms| 635
   symw| 27543

   tab:-23!select from `:t/

   .Q.w[]
   used| 328000
   heap| 67108864
   peak| 67108864
   wmax| 0
   mmap| 480000032
   mphy| 16827965440
   syms| 638
   symw| 27628

As you can see the increase in used memory does no increase much (the increase being caused by the table
being mapped into memory rather than loaded)
  
  
I hope this answers your question!
Regards!
Alex

3

See my comments inline and let me know what you think.

4

-23!x sequentially faults any mapped memory backing x, to attempt it being fully resident in core on completion.

This is useful when x is not already in the page cache, further use (e.g. sort) of x entails random access over much of x and the disk technology from where x is mapped does not perform well for random access (e.g. spinning rust) relative to sequential access.


5

Hi! 

When using 

select from `:t/ 

You only map the table into memory not load it, the reverse forces the table to actually be loaded into memory so we can see the difference in memory used and that is why I used it. You are right about the fact that the memory used increases as a result of using reverse but the increase would be the same if we would load the table into memory.

The -23! select from :t/ and select from :t/ have indeed the same effect on memory used. 

I was just trying to point out that select from :t/ does not load anything into memory and that -23! select from :t/ is not any different than select from `:t/. 

Cheers! 

Alex

6

>The -23! select from :t/ and select from :t/ have indeed the same effect on memory used.


as measured by .Q.w.

But the effect of -23! is visible on resident vs virt


q).z.i / pid

24070i

q)t:select from quote where date=last date

q)\top -n 1 -b -p 24070

"  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND        ..

"24070 charlie 20 0 27.4g 3.6g 3728 S 0.0 1.4 0:01.14 q ..

q)-23!t;

q)\top -n 1 -b -p 24070

"  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND        ..

"24070 charlie 20 0 27.4g 26g 23g S 0.0 10.7 1:01.40 q ..