Hi,After loading a table from a csv file (large record count), I amsaving it to the disk. In Script0 IDE I want to query the data and seeall the rows.But when I run a query like:select from value `:tabloadI still am getting the ‘Out of Memory’ error. I thought that sincewhile querying kdb+ would pull the data from the disk, so this errorshould not occur. kdb+ is loading the table from the disk to memoryevery time I am accessing it, and so is the memory outage.How can I see all the rows in a big table at one shot?Thanks,Sayan
In-Reply-To: Sensitivity: NormalImportance: NormalSubject: Re: [personal kdb+] Querying a table in kdb+To: personal-kdbplus@googlegroups.comFrom: “Manish Patel” <manni.patel>Date: Tue, 28 Apr 2009 08:12:43 +0000U3R1ZGlvIGlzIGFuIGVudGlyZWx5IHNlcGFyYXRlIGFwcCBtYWtpbmcgYSBjb25uZWN0aW9uIHRvIGtkYiAtIHNvIHllcywgaXQgbWFrZXMgc2Vuc2UgdGhhdCBpdCB0aHJvd3MgdGhlIGVudGlyZSB0YWJsZSBhdCB0aGUgY2FsbGVyIGJlY2F1c2UgdGhhdCdzIHdoYXQgdGhlIGNhbGxlciBpcyBhc2tpbmcgZm9yLg0KDQpEaWQgeW91IHRyeSBtYXhpbmcgdGhlIG1lbW9yeSBmb3Igc3R1ZGlvPyBJZiB5b3Ugc3RpbGwgZ2V0IGFuIGVycm9yIHlvdSBjYW4gbG9vayBhdCBpdCBiaXQgYnkgYml0IHdpdGggdGhlIGkgaW5kZXggb3IgY3V0IGRvd24gdGhlIG51bWJlciBvZiBjb2x1bW5zIHRvIHdoYXQgeW91IHJlYWxseSBuZWVkIHRvIHNlZSBpbiB5b3VyIHNlbGVjdCBzdGF0ZW1lbnQNCg0KU2VudCB1c2luZyBCbGFja0JlcnJ5riBmcm9tIE9yYW5nZQ0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogU2F5YW5vc2F1cmFzIDxzYXlhbmRlZXA1MkBnbWFpbC5jb20+DQoNCkRhdGU6IFR1ZSwgMjggQXByIDIwMDkgMDA6Mzc6MDIgDQpUbzogS2RiKyBQZXJzb25hbCBEZXZlbG9wZXJzPHBlcnNvbmFsLWtkYnBsdXNAZ29vZ2xlZ3JvdXBzLmNvbT4NClN1YmplY3Q6IFtwZXJzb25hbCBrZGIrXSBRdWVyeWluZyBhIHRhYmxlIGluIGtkYisNCg0KDQoNCkhpLA0KDQpBZnRlciBsb2FkaW5nIGEgdGFibGUgZnJvbSBhIGNzdiBmaWxlIChsYXJnZSByZWNvcmQgY291bnQpLCBJIGFtDQpzYXZpbmcgaXQgdG8gdGhlIGRpc2suIEluIFNjcmlwdDAgSURFIEkgd2FudCB0byBxdWVyeSB0aGUgZGF0YSBhbmQgc2VlDQphbGwgdGhlIHJvd3MuDQoNCkJ1dCB3aGVuIEkgcnVuIGEgcXVlcnkgbGlrZToNCg0Kc2VsZWN0IGZyb20gdmFsdWUgYDp0YWJsb2FkDQoNCkkgc3RpbGwgYW0gZ2V0dGluZyB0aGUgJ091dCBvZiBNZW1vcnknIGVycm9yLiBJIHRob3VnaHQgdGhhdCBzaW5jZQ0Kd2hpbGUgcXVlcnlpbmcga2RiKyB3b3VsZCBwdWxsIHRoZSBkYXRhIGZyb20gdGhlIGRpc2ssIHNvIHRoaXMgZXJyb3INCnNob3VsZCBub3Qgb2NjdXIuIGtkYisgaXMgbG9hZGluZyB0aGUgdGFibGUgZnJvbSB0aGUgZGlzayB0byBtZW1vcnkNCmV2ZXJ5IHRpbWUgSSBhbSBhY2Nlc3NpbmcgaXQsIGFuZCBzbyBpcyB0aGUgbWVtb3J5IG91dGFnZS4NCg0KSG93IGNhbiBJIHNlZSBhbGwgdGhlIHJvd3MgaW4gYSBiaWcgdGFibGUgYXQgb25lIHNob3Q/DQoNClRoYW5rcywNClNheWFuDQoNCg0KDQo</manni.patel>
Thanks Manish. Yes I did try maximizing a couple of times, but nothinghappens. Yes I guess I have to take a call about that. But when I runthe query from the console, then only a part of it appears at atime…is there a way to view all the records bunch by bunch?SayanOn Apr 28, 1:12?pm, “Manish Patel” <manni.pa…> wrote:> Studio is an entirely separate app making a connection to kdb - so yes, it makes sense that it throws the entire table at the caller because that’s what the caller is asking for.>> Did you try maxing the memory for studio? If you still get an error you can look at it bit by bit with the i index or cut down the number of columns to what you really need to see in your select statement>> Sent using BlackBerry? from Orange>> -----Original Message-----> From: Sayanosauras <sayandee…>>> Date: Tue, 28 Apr 2009 00:37:02> To: Kdb+ Personal Developers> Subject: [personal kdb+] Querying a table in kdb+>> Hi,>> After loading a table from a csv file (large record count), I am> saving it to the disk. In Script0 IDE I want to query the data and see> all the rows.>> But when I run a query like:>> select from value `:tabload>> I still am getting the ‘Out of Memory’ error. I thought that since> while querying kdb+ would pull the data from the disk, so this error> should not occur. kdb+ is loading the table from the disk to memory> every time I am accessing it, and so is the memory outage.>> How can I see all the rows in a big table at one shot?>> Thanks,> Sayan</sayandee…></manni.pa…>
X-Mailer: Apple Mail (2.930.3)On 28.04.2009, at 09:37, Sayanosauras wrote:> But when I run a query like:>> select from value :tabload\>\> I still am getting the 'Out of Memory' error. I thought that since\> while querying kdb+ would pull the data from the disk, so this error\> should not occur. kdb+ is loading the table from the disk to memory\> every time I am accessing it, and so is the memory outage.value :tabloadis asking kdb+ to get the complete value stored at `:tabloadso it’s doing exactly what you asked, but that’s not what you wantif you start your q session pointing at your saved table..$ q tabload -p 1234you’ll have a session where you can get data loaded on demand and run queries likeq) select aa,bb,cc from tabload where dd?