Hi list,
We are trying to estimate RAM usage for a specific table and would
like to derive some kind of general formula for any table based on the
schema (and maybe some sample data).
We have tried an approach below based on having a sizable chunk of
sample data and and comparing .Q.w[`used]
with or without each column in our sample:
checkColumnRAMUsage:{[table]
columns : (key flip value table);
sizePerColumn :: ()!();
{[table;columns;column]
getRowSize : {[t]
t2 :: t;
/ delete a single row to make sure we have a copy of the table data
delete from `t2 where time = min(time);
tableCount : count t2;
/ check the amount of used memory
.Q.gc;
before: .Q.w[`used];
/ drop the newly created table
delete t2 from `.;
/ check the amount of used memory, again
.Q.gc;
after: .Q.w[`used];
: (before - after) % tableCount;
};
.log.out[.z.h;"Checking column: ";column];
$[column <> `time;
[
baseline : getRowSize[?[table;();0b;columns!columns]];
symIndex : columns ? column;
columns : columns _ symIndex;
smaller : getRowSize[?[table;();0b;columns!columns]];
sizePerColumn[column] :: baseline - smaller;
];
]
}[table;columns;] each columns;
temp : sizePerColumn;
delete sizePerColumn from `.;
: temp;
}