Hi,Say I have a large single file or splayed table (not partitioned) thatwill not fit in memory. Is there any way to update the table to besorted by a given column without needing to have the entire table inmemory?Thanks,Leon
yes:// flat tableq)t:(a:2 1 2;b:2 2 3)q)savet:tq)@[`:.;`t;{`a xasc x}]`:.q)get `:ta b—1 22 22 3q)// splayedq)t:(a:2 1 2;b:2 2 3)q)`a xasc `:t/`:t/q)get`:t/a b—1 22 3q)t:(a:2 1 2;b:2 2 3)q)`:t/ set t`:t/q)`a xasc `:t/`:t/q)get`:t/a b—1 22 22 3q)On Sun, Nov 17, 2013 at 4:51 PM, Leon Baum wrote:>> Hi,>> Say I have a large single file or splayed table (not partitioned) that> will not fit in memory. Is there any way to update the table to be> sorted by a given column without needing to have the entire table in> memory?>>> Thanks,> Leon>> –>
Submitted via Google Groups
Thanks, those examples do result in the tables being updated on disk,but the sorting still appears to use as much memory as loading theentire table. I suppose that’s the only efficient way to do it.Leonwp writes:> yes:>> // flat table> q)t:(a:2 1 2;b:2 2 3)> q)savet\> :t> q)@[`:.;`t;{`a xasc x}]> `:.> q)get `:t> a b> —> 1 2> 2 2> 2 3> q)>> // splayed> q)t:(a:2 1 2;b:2 2 3)> q)`a xasc `:t/> `:t/> q)get`:t/> a b> —> 1 2> 2 3> q)t:(a:2 1 2;b:2 2 3)> q)`:t/ set t> `:t/> q)`a xasc `:t/> `:t/> q)get`:t/> a b> —> 1 2> 2 2> 2 3> q)>>> On Sun, Nov 17, 2013 at 4:51 PM, Leon Baum wrote:>>>> Hi,>>>> Say I have a large single file or splayed table (not partitioned) that>> will not fit in memory. Is there any way to update the table to be>> sorted by a given column without needing to have the entire table in>> memory?>>>>>> Thanks,>> Leon>>>> –>>
Submitted via Google Groups
a xasc :t/
calculates the index permuation, and then applies it column by column.
You may not notice a difference in mem usage unless you have >2 columns.