dealing with opaque type in C API

Community Support
1

Suppose there is a C library whose API is written with either void* or sometype_t and functions tied to these types. For example,

void* init(int x); // returns an opaque type variable

int f(void* x, int y); // uses the init’ed thing

http://code.kx.com/wiki/Cookbook/InterfacingWithC doesn’t quite tell me what to do. I have a little-hack-that-works-for-me™ but before going further with it, I I better ask for a more proper and proven way of doing things. (My hack involves intptr_t conversion.)

thanks,

-jay

2

just store it as int or long atom depending on the architecture.
when you pass it back convert it to the desired pointer type.

on thing: this is dangerous approach to opaque types. better hide these pointers as global static vars in your c files.

 

3

On Thu, Feb 23, 2012 at 1:47 AM, Felix Lungu <felix.lungu@gmail.com> wrote:

just store it as int or long atom depending on the architecture.
when you pass it back convert it to the desired pointer type.

That’s what intptr_t can do. I can make it “safe” by converting to/from symbol and prevent accidental modification.

on thing: this is dangerous approach to opaque types. better hide these pointers as global static vars in your c files.

Because init() can be called at different times, static initialization seems infeasible.

Thanks for your suggestions!

 

 

On 23 Feb 2012, at 11:14, Jay Han wrote:

Suppose there is a C library whose API is written with either void* or sometype_t and functions tied to these types. For example,

void* init(int x); // returns an opaque type variable

int f(void* x, int y); // uses the init’ed thing

http://code.kx.com/wiki/Cookbook/InterfacingWithC doesn’t quite tell me what to do. I have a little-hack-that-works-for-me™ but before going further with it, I I better ask for a more proper and proven way of doing things. (My hack involves intptr_t conversion.)

thanks,

-jay

–

Submitted via Google Groups

4

To: personal-kdbplus@googlegroups.com
X-Mailer: Apple Mail (2.1257)
X-Gm-Message-State: ALoCoQmhGIighWXZW70SWkPSdFM9sr5C1osvfu4OLXUj9BXOySwORhifTbzMyR+1GB4IgPcqCR2z

Hi All,

I am trying to get the number of trades for each sym and date for a =
partitioned (very big) historical database.
If I .Q.view to one date, select count i by sym from trade executes =
quite fast, time depends on the date.
However if I .Q.view to, let’s say, a month, then select count i by =
date, sym from trade took more than I wanted to wait (never finished, =
but lasted at least 2 minutes).

Then I tried this:

likviditas:flip symdatedb!(sym$();date$();int$());
{likviditas::likviditas, select sym, date:x, db from select db:count i =
by sym from trade where date=x;0N!`$"OK - ",(string x)} each dates

where “dates” is a list of all the dates in the DB. It works really =
fast, but the kernel kills q after processing about only 100 days…
The kernel log is at the end of my message.
Do you have any idea what goes wrong here? It seems that the process =
runs out of memory, but why? One day of data fits in the memory, and the =
above function never uses more than one day, does it?

Thanks for any help,
Andras

The kernel log:

Feb 24 08:59:54 f1 kernel: [1320010.284407] q invoked oom-killer: =
gfp_mask=0x280da, order=0, oom_adj=0
Feb 24 08:59:54 f1 kernel: [1320010.284422] q cpuset=/ mems_allowed=0
Feb 24 08:59:54 f1 kernel: [1320010.284430] Pid: 4363, comm: q Not =
tainted 2.6.32-5-xen-amd64 #1
Feb 24 08:59:54 f1 kernel: [1320010.284436] Call Trace:
Feb 24 08:59:54 f1 kernel: [1320010.284449] [] ? =
oom_kill_process+0x7f/0x23f
Feb 24 08:59:54 f1 kernel: [1320010.284458] [] ? =
__out_of_memory+0x12a/0x141
Feb 24 08:59:54 f1 kernel: [1320010.284466] [] ? =
out_of_memory+0x140/0x172
Feb 24 08:59:54 f1 kernel: [1320010.284475] [] ? =
__alloc_pages_nodemask+0x4ec/0x5fd
Feb 24 08:59:54 f1 kernel: [1320010.284486] [] ? =
handle_mm_fault+0x27a/0x80f
Feb 24 08:59:54 f1 kernel: [1320010.284495] [] ? =
get_unmapped_area+0xd7/0x139
Feb 24 08:59:54 f1 kernel: [1320010.284505] [] ? =
do_page_fault+0x2e0/0x2fc
Feb 24 08:59:54 f1 kernel: [1320010.284515] [] ? =
page_fault+0x25/0x30
Feb 24 08:59:54 f1 kernel: [1320010.284522] Mem-Info:
Feb 24 08:59:54 f1 kernel: [1320010.284526] Node 0 DMA per-cpu:
Feb 24 08:59:54 f1 kernel: [1320010.284533] CPU 0: hi: 0, btch: =
1 usd: 0
Feb 24 08:59:54 f1 kernel: [1320010.284539] CPU 1: hi: 0, btch: =
1 usd: 0
Feb 24 08:59:54 f1 kernel: [1320010.284544] Node 0 DMA32 per-cpu:
Feb 24 08:59:54 f1 kernel: [1320010.284551] CPU 0: hi: 186, btch: =
31 usd: 30
Feb 24 08:59:54 f1 kernel: [1320010.284557] CPU 1: hi: 186, btch: =
31 usd: 0
Feb 24 08:59:54 f1 kernel: [1320010.284566] active_anon:60341 =
inactive_anon:60379 isolated_anon:0
Feb 24 08:59:54 f1 kernel: [1320010.284568] active_file:7 =
inactive_file:8 isolated_file:0
Feb 24 08:59:54 f1 kernel: [1320010.284569] unevictable:0 dirty:0 =
writeback:0 unstable:0
Feb 24 08:59:54 f1 kernel: [1320010.284570] free:1178 =
slab_reclaimable:737 slab_unreclaimable:905
Feb 24 08:59:54 f1 kernel: [1320010.284572] mapped:29 shmem:1 =
pagetables:952 bounce:0
Feb 24 08:59:54 f1 kernel: [1320010.284592] Node 0 DMA free:2024kB =
min:80kB low:100kB high:120kB active_anon:6436kB inactive_anon:6484kB =
active_file:28kB inactive_file:0kB unevictable:0kB isolated(anon):0kB =
isolated(file):0kB present:14868kB mlocked:0kB dirty:0kB writeback:0kB =
mapped:36kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB =
kernel_stack:0kB pagetables:48kB unstable:0kB bounce:0kB =
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 24 08:59:54 f1 kernel: [1320010.284626] lowmem_reserve: 0 489 489 =
489
Feb 24 08:59:54 f1 kernel: [1320010.284639] Node 0 DMA32 free:2688kB =
min:2788kB low:3484kB high:4180kB active_anon:234928kB =
inactive_anon:235032kB active_file:0kB inactive_file:32kB =
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB =
mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:4kB =
slab_reclaimable:2948kB slab_unreclaimable:3612kB kernel_stack:592kB =
pagetables:3760kB unstable:0kB bounce:0kB writeback_tmp:0kB =
pages_scanned:207 all_unreclaimable? yes
Feb 24 08:59:54 f1 kernel: [1320010.284674] lowmem_reserve: 0 0 0 0
Feb 24 08:59:54 f1 kernel: [1320010.284685] Node 0 DMA: 04kB 18kB =
016kB 132kB 164kB 1128kB 1256kB 1512kB 11024kB 02048kB 04096kB =
= 2024kB
Feb 24 08:59:54 f1 kernel: [1320010.284711] Node 0 DMA32: 1144kB 118kB =
816kB 132kB 164kB 1128kB 1256kB 1512kB 11024kB 02048kB 04096kB =
= 2688kB
Feb 24 08:59:54 f1 kernel: [1320010.284736] 21449 total pagecache pages
Feb 24 08:59:54 f1 kernel: [1320010.284741] 21405 pages in swap cache
Feb 24 08:59:54 f1 kernel: [1320010.284747] Swap cache stats: add =
3888544, delete 3867139, find 1550765/1779704
Feb 24 08:59:54 f1 kernel: [1320010.284754] Free swap = 0kB
Feb 24 08:59:54 f1 kernel: [1320010.284759] Total swap = 524280kB
Feb 24 08:59:54 f1 kernel: [1320010.286336] 131072 pages RAM
Feb 24 08:59:54 f1 kernel: [1320010.286345] 4078 pages reserved
Feb 24 08:59:54 f1 kernel: [1320010.286350] 168 pages shared
Feb 24 08:59:54 f1 kernel: [1320010.286354] 125554 pages non-shared
Feb 24 08:59:54 f1 kernel: [1320010.286360] Out of memory: kill process =
4362 (rlwrap) score 158020 or a child
Feb 24 08:59:54 f1 kernel: [1320010.286367] Killed process 4363 (q)=

5

how many symbols per day?

6

To: personal-kdbplus@googlegroups.com
X-Mailer: Apple Mail (2.1257)
X-Gm-Message-State: ALoCoQn0N5zj5TgVQcqIcCgsMtNYHLBJpscieG78Eyv9+sMzbZFTADG7NIiTWOzOayoOgXKR20WI

around 10000, this is the full taq database

2012.02.24. d=E1tummal, 12:05 id=F5pontban Charles Skelton =EDrta:

> how many symbols per day?
>
> On Fri, Feb 24, 2012 at 11:30 AM, Boh=E1k Andr=E1s =
wrote:
> Hi All,
>
> I am trying to get the number of trades for each sym and date for a =
partitioned (very big) historical database.
> If I .Q.view to one date, select count i by sym from trade executes =
quite fast, time depends on the date.
> However if I .Q.view to, let’s say, a month, then select count i by =
date, sym from trade took more than I wanted to wait (never finished, =
but lasted at least 2 minutes).
>
> Then I tried this:
>
> likviditas:flip symdatedb!(sym$();date$();int$());
> {likviditas::likviditas, select sym, date:x, db from select db:count i =
by sym from trade where date=x;0N!`$"OK - ",(string x)} each dates
>
> where “dates” is a list of all the dates in the DB. It works really =
fast, but the kernel kills q after processing about only 100 days…
> The kernel log is at the end of my message.
> Do you have any idea what goes wrong here? It seems that the process =
runs out of memory, but why? One day of data fits in the memory, and the =
above function never uses more than one day, does it?
>
> Thanks for any help,
> Andras
>
>
> The kernel log:
>
> Feb 24 08:59:54 f1 kernel: [1320010.284407] q invoked oom-killer: =
gfp_mask=0x280da, order=0, oom_adj=0
> Feb 24 08:59:54 f1 kernel: [1320010.284422] q cpuset=/ =
mems_allowed=0
> Feb 24 08:59:54 f1 kernel: [1320010.284430] Pid: 4363, comm: q Not =
tainted 2.6.32-5-xen-amd64 #1
> Feb 24 08:59:54 f1 kernel: [1320010.284436] Call Trace:
> Feb 24 08:59:54 f1 kernel: [1320010.284449] [] ? =
oom_kill_process+0x7f/0x23f
> Feb 24 08:59:54 f1 kernel: [1320010.284458] [] ? =
__out_of_memory+0x12a/0x141
> Feb 24 08:59:54 f1 kernel: [1320010.284466] [] ? =
out_of_memory+0x140/0x172
> Feb 24 08:59:54 f1 kernel: [1320010.284475] [] ? =
__alloc_pages_nodemask+0x4ec/0x5fd
> Feb 24 08:59:54 f1 kernel: [1320010.284486] [] ? =
handle_mm_fault+0x27a/0x80f
> Feb 24 08:59:54 f1 kernel: [1320010.284495] [] ? =
get_unmapped_area+0xd7/0x139
> Feb 24 08:59:54 f1 kernel: [1320010.284505] [] ? =
do_page_fault+0x2e0/0x2fc
> Feb 24 08:59:54 f1 kernel: [1320010.284515] [] ? =
page_fault+0x25/0x30
> Feb 24 08:59:54 f1 kernel: [1320010.284522] Mem-Info:
> Feb 24 08:59:54 f1 kernel: [1320010.284526] Node 0 DMA per-cpu:
> Feb 24 08:59:54 f1 kernel: [1320010.284533] CPU 0: hi: 0, btch: =
1 usd: 0
> Feb 24 08:59:54 f1 kernel: [1320010.284539] CPU 1: hi: 0, btch: =
1 usd: 0
> Feb 24 08:59:54 f1 kernel: [1320010.284544] Node 0 DMA32 per-cpu:
> Feb 24 08:59:54 f1 kernel: [1320010.284551] CPU 0: hi: 186, btch: =
31 usd: 30
> Feb 24 08:59:54 f1 kernel: [1320010.284557] CPU 1: hi: 186, btch: =
31 usd: 0
> Feb 24 08:59:54 f1 kernel: [1320010.284566] active_anon:60341 =
inactive_anon:60379 isolated_anon:0
> Feb 24 08:59:54 f1 kernel: [1320010.284568] active_file:7 =
inactive_file:8 isolated_file:0
> Feb 24 08:59:54 f1 kernel: [1320010.284569] unevictable:0 dirty:0 =
writeback:0 unstable:0
> Feb 24 08:59:54 f1 kernel: [1320010.284570] free:1178 =
slab_reclaimable:737 slab_unreclaimable:905
> Feb 24 08:59:54 f1 kernel: [1320010.284572] mapped:29 shmem:1 =
pagetables:952 bounce:0
> Feb 24 08:59:54 f1 kernel: [1320010.284592] Node 0 DMA free:2024kB =
min:80kB low:100kB high:120kB active_anon:6436kB inactive_anon:6484kB =
active_file:28kB inactive_file:0kB unevictable:0kB isolated(anon):0kB =
isolated(file):0kB present:14868kB mlocked:0kB dirty:0kB writeback:0kB =
mapped:36kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB =
kernel_stack:0kB pagetables:48kB unstable:0kB bounce:0kB =
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> Feb 24 08:59:54 f1 kernel: [1320010.284626] lowmem_reserve: 0 489 =
489 489
> Feb 24 08:59:54 f1 kernel: [1320010.284639] Node 0 DMA32 free:2688kB =
min:2788kB low:3484kB high:4180kB active_anon:234928kB =
inactive_anon:235032kB active_file:0kB inactive_file:32kB =
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB =
mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:4kB =
slab_reclaimable:2948kB slab_unreclaimable:3612kB kernel_stack:592kB =
pagetables:3760kB unstable:0kB bounce:0kB writeback_tmp:0kB =
pages_scanned:207 all_unreclaimable? yes
> Feb 24 08:59:54 f1 kernel: [1320010.284674] lowmem_reserve: 0 0 0 0
> Feb 24 08:59:54 f1 kernel: [1320010.284685] Node 0 DMA: 04kB 18kB =
016kB 132kB 164kB 1128kB 1256kB 1512kB 11024kB 02048kB 04096kB =
= 2024kB
> Feb 24 08:59:54 f1 kernel: [1320010.284711] Node 0 DMA32: 1144kB =
118kB 816kB 132kB 164kB 1128kB 1256kB 1512kB 11024kB 02048kB =
04096kB = 2688kB
> Feb 24 08:59:54 f1 kernel: [1320010.284736] 21449 total pagecache =
pages
> Feb 24 08:59:54 f1 kernel: [1320010.284741] 21405 pages in swap cache
> Feb 24 08:59:54 f1 kernel: [1320010.284747] Swap cache stats: add =
3888544, delete 3867139, find 1550765/1779704
> Feb 24 08:59:54 f1 kernel: [1320010.284754] Free swap = 0kB
> Feb 24 08:59:54 f1 kernel: [1320010.284759] Total swap = 524280kB
> Feb 24 08:59:54 f1 kernel: [1320010.286336] 131072 pages RAM
> Feb 24 08:59:54 f1 kernel: [1320010.286345] 4078 pages reserved
> Feb 24 08:59:54 f1 kernel: [1320010.286350] 168 pages shared
> Feb 24 08:59:54 f1 kernel: [1320010.286354] 125554 pages non-shared
> Feb 24 08:59:54 f1 kernel: [1320010.286360] Out of memory: kill =
process 4362 (rlwrap) score 158020 or a child
> Feb 24 08:59:54 f1 kernel: [1320010.286367] Killed process 4363 (q)
>
> –
> You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> To post to this group, send email to =
personal-kdbplus@googlegroups.com.
> To unsubscribe from this group, send email to =
personal-kdbplus+unsubscribe@googlegroups.com.
> For more options, visit this group at =
http://groups.google.com/group/personal-kdbplus?hl=en.
>
>
>
> –
> You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> To post to this group, send email to =
personal-kdbplus@googlegroups.com.
> To unsubscribe from this group, send email to =
personal-kdbplus+unsubscribe@googlegroups.com.
> For more options, visit this group at =
http://groups.google.com/group/personal-kdbplus?hl=en.


7

can you say more about your setup?

kdb+ 32 or 64bit? Which version/release date?

using par.txt?

partitioned by date and sym, or just by date?

multithreaded?

using built-in file compression?

How much RAM on system (physical+swap)

thanks

8

To: personal-kdbplus@googlegroups.com
X-Mailer: Apple Mail (2.1257)
X-Gm-Message-State: ALoCoQmeM0sJJ+eWyiYgEGdTiTD00VVubgutiV1lgVcLFG/quShwlXCcBHLKhEp4KnHQCqV5qub0

Of course:

> can you say more about your setup?
> kdb+ 32 or 64bit? Which version/release date?
kdb+ 32bit
KDB+ 2.8 2012.01.20 Copyright (C) 1993-2012 Kx Systems
l32/ 2()core 507MB x.x.x.x PLAY 2012.04.19
> using par.txt?
No
> partitioned by date and sym, or just by date?
Just by date
> multithreaded?
No.
> using built-in file compression?
No.
> How much RAM on system (physical+swap)
(It should be 2GB. This is a virtual server, the admin probably took =
away some ram recently. But when the problem occured, it had 2GB ram)

cat /proc/meminfo
MemTotal: 520028 kB
MemFree: 410012 kB
Buffers: 2124 kB
Cached: 68496 kB
SwapCached: 1292 kB
Active: 7952 kB
Inactive: 64924 kB
Active(anon): 364 kB
Inactive(anon): 1856 kB
Active(file): 7588 kB
Inactive(file): 63068 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524280 kB
SwapFree: 517712 kB
Dirty: 16 kB
Writeback: 0 kB
AnonPages: 1272 kB
Mapped: 1872 kB
Shmem: 0 kB
Slab: 12688 kB
SReclaimable: 8816 kB
SUnreclaim: 3872 kB
KernelStack: 576 kB
PageTables: 1848 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 784292 kB
Committed_AS: 46808 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 2380 kB
VmallocChunk: 34359733712 kB
HardwareCorrupted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 524288 kB
DirectMap2M: 0 kB

> thanks
>
> On Fri, Feb 24, 2012 at 1:16 PM, Boh=E1k Andr=E1s =
wrote:
> around 10000, this is the full taq database
>
> 2012.02.24. d=E1tummal, 12:05 id=F5pontban Charles Skelton =EDrta:
>
> > how many symbols per day?
> >
> > On Fri, Feb 24, 2012 at 11:30 AM, Boh=E1k Andr=E1s =
wrote:
> > Hi All,
> >
> > I am trying to get the number of trades for each sym and date for a =
partitioned (very big) historical database.
> > If I .Q.view to one date, select count i by sym from trade executes =
quite fast, time depends on the date.
> > However if I .Q.view to, let’s say, a month, then select count i by =
date, sym from trade took more than I wanted to wait (never finished, =
but lasted at least 2 minutes).
> >
> > Then I tried this:
> >
> > likviditas:flip symdatedb!(sym$();date$();int$());
> > {likviditas::likviditas, select sym, date:x, db from select db:count =
i by sym from trade where date=x;0N!`$"OK - ",(string x)} each dates
> >
> > where “dates” is a list of all the dates in the DB. It works really =
fast, but the kernel kills q after processing about only 100 days…
> > The kernel log is at the end of my message.
> > Do you have any idea what goes wrong here? It seems that the process =
runs out of memory, but why? One day of data fits in the memory, and the =
above function never uses more than one day, does it?
> >
> > Thanks for any help,
> > Andras
> >
> >
> > The kernel log:
> >
> > Feb 24 08:59:54 f1 kernel: [1320010.284407] q invoked oom-killer: =
gfp_mask=0x280da, order=0, oom_adj=0
> > Feb 24 08:59:54 f1 kernel: [1320010.284422] q cpuset=/ =
mems_allowed=0
> > Feb 24 08:59:54 f1 kernel: [1320010.284430] Pid: 4363, comm: q Not =
tainted 2.6.32-5-xen-amd64 #1
> > Feb 24 08:59:54 f1 kernel: [1320010.284436] Call Trace:
> > Feb 24 08:59:54 f1 kernel: [1320010.284449] [] ? =
oom_kill_process+0x7f/0x23f
> > Feb 24 08:59:54 f1 kernel: [1320010.284458] [] ? =
__out_of_memory+0x12a/0x141
> > Feb 24 08:59:54 f1 kernel: [1320010.284466] [] ? =
out_of_memory+0x140/0x172
> > Feb 24 08:59:54 f1 kernel: [1320010.284475] [] ? =
__alloc_pages_nodemask+0x4ec/0x5fd
> > Feb 24 08:59:54 f1 kernel: [1320010.284486] [] ? =
handle_mm_fault+0x27a/0x80f
> > Feb 24 08:59:54 f1 kernel: [1320010.284495] [] ? =
get_unmapped_area+0xd7/0x139
> > Feb 24 08:59:54 f1 kernel: [1320010.284505] [] ? =
do_page_fault+0x2e0/0x2fc
> > Feb 24 08:59:54 f1 kernel: [1320010.284515] [] ? =
page_fault+0x25/0x30
> > Feb 24 08:59:54 f1 kernel: [1320010.284522] Mem-Info:
> > Feb 24 08:59:54 f1 kernel: [1320010.284526] Node 0 DMA per-cpu:
> > Feb 24 08:59:54 f1 kernel: [1320010.284533] CPU 0: hi: 0, =
btch: 1 usd: 0
> > Feb 24 08:59:54 f1 kernel: [1320010.284539] CPU 1: hi: 0, =
btch: 1 usd: 0
> > Feb 24 08:59:54 f1 kernel: [1320010.284544] Node 0 DMA32 per-cpu:
> > Feb 24 08:59:54 f1 kernel: [1320010.284551] CPU 0: hi: 186, =
btch: 31 usd: 30
> > Feb 24 08:59:54 f1 kernel: [1320010.284557] CPU 1: hi: 186, =
btch: 31 usd: 0
> > Feb 24 08:59:54 f1 kernel: [1320010.284566] active_anon:60341 =
inactive_anon:60379 isolated_anon:0
> > Feb 24 08:59:54 f1 kernel: [1320010.284568] active_file:7 =
inactive_file:8 isolated_file:0
> > Feb 24 08:59:54 f1 kernel: [1320010.284569] unevictable:0 dirty:0 =
writeback:0 unstable:0
> > Feb 24 08:59:54 f1 kernel: [1320010.284570] free:1178 =
slab_reclaimable:737 slab_unreclaimable:905
> > Feb 24 08:59:54 f1 kernel: [1320010.284572] mapped:29 shmem:1 =
pagetables:952 bounce:0
> > Feb 24 08:59:54 f1 kernel: [1320010.284592] Node 0 DMA free:2024kB =
min:80kB low:100kB high:120kB active_anon:6436kB inactive_anon:6484kB =
active_file:28kB inactive_file:0kB unevictable:0kB isolated(anon):0kB =
isolated(file):0kB present:14868kB mlocked:0kB dirty:0kB writeback:0kB =
mapped:36kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB =
kernel_stack:0kB pagetables:48kB unstable:0kB bounce:0kB =
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> > Feb 24 08:59:54 f1 kernel: [1320010.284626] lowmem_reserve: 0 489 =
489 489
> > Feb 24 08:59:54 f1 kernel: [1320010.284639] Node 0 DMA32 free:2688kB =
min:2788kB low:3484kB high:4180kB active_anon:234928kB =
inactive_anon:235032kB active_file:0kB inactive_file:32kB =
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:500960kB =
mlocked:0kB dirty:0kB writeback:0kB mapped:80kB shmem:4kB =
slab_reclaimable:2948kB slab_unreclaimable:3612kB kernel_stack:592kB =
pagetables:3760kB unstable:0kB bounce:0kB writeback_tmp:0kB =
pages_scanned:207 all_unreclaimable? yes
> > Feb 24 08:59:54 f1 kernel: [1320010.284674] lowmem_reserve: 0 0 0 =
0
> > Feb 24 08:59:54 f1 kernel: [1320010.284685] Node 0 DMA: 04kB 18kB =
016kB 132kB 164kB 1128kB 1256kB 1512kB 11024kB 02048kB 04096kB =
= 2024kB
> > Feb 24 08:59:54 f1 kernel: [1320010.284711] Node 0 DMA32: 1144kB =
118kB 816kB 132kB 164kB 1128kB 1256kB 1512kB 11024kB 02048kB =
04096kB = 2688kB
> > Feb 24 08:59:54 f1 kernel: [1320010.284736] 21449 total pagecache =
pages
> > Feb 24 08:59:54 f1 kernel: [1320010.284741] 21405 pages in swap =
cache
> > Feb 24 08:59:54 f1 kernel: [1320010.284747] Swap cache stats: add =
3888544, delete 3867139, find 1550765/1779704
> > Feb 24 08:59:54 f1 kernel: [1320010.284754] Free swap = 0kB
> > Feb 24 08:59:54 f1 kernel: [1320010.284759] Total swap = 524280kB
> > Feb 24 08:59:54 f1 kernel: [1320010.286336] 131072 pages RAM
> > Feb 24 08:59:54 f1 kernel: [1320010.286345] 4078 pages reserved
> > Feb 24 08:59:54 f1 kernel: [1320010.286350] 168 pages shared
> > Feb 24 08:59:54 f1 kernel: [1320010.286354] 125554 pages non-shared
> > Feb 24 08:59:54 f1 kernel: [1320010.286360] Out of memory: kill =
process 4362 (rlwrap) score 158020 or a child
> > Feb 24 08:59:54 f1 kernel: [1320010.286367] Killed process 4363 (q)
> >
> > –
> > You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> > To post to this group, send email to =
personal-kdbplus@googlegroups.com.
> > To unsubscribe from this group, send email to =
personal-kdbplus+unsubscribe@googlegroups.com.
> > For more options, visit this group at =
http://groups.google.com/group/personal-kdbplus?hl=en.
> >
> >
> >
> > –
> > You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> > To post to this group, send email to =
personal-kdbplus@googlegroups.com.
> > To unsubscribe from this group, send email to =
personal-kdbplus+unsubscribe@googlegroups.com.
> > For more options, visit this group at =
http://groups.google.com/group/personal-kdbplus?hl=en.
>
> –
> You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> To post to this group, send email to =
personal-kdbplus@googlegroups.com.
> To unsubscribe from this group, send email to =
personal-kdbplus+unsubscribe@googlegroups.com.
> For more options, visit this group at =
http://groups.google.com/group/personal-kdbplus?hl=en.
>
>
>
> –
> You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> To post to this group, send email to =
personal-kdbplus@googlegroups.com.
> To unsubscribe from this group, send email to =
personal-kdbplus+unsubscribe@googlegroups.com.
> For more options, visit this group at =
http://groups.google.com/group/personal-kdbplus?hl=en.


9

2GB is quite a tight system for taq, 500MB an even more challenging environment ;-)

select db:count i by date,sym from trade where date in dates

should be as optimal as any.

try using the command line option

-g 1

to free memory as soon as it can.

Although it may be that you simply have too little ram to process a particular day’s data - can you see which date it fails on, and whether that is unusually large or can indeed be loaded as a single date?

Typically you’d need 8-16GB to query taq in any meaningful form.

thanks