I am now doing a project to download the price quote to KDB+ . And
figure out how to use “peach” to parallel download web data and store
in a table . I am using 32b trial version and start q with “C:\q
\w32\q.exe -s 5” .
The codes showed as below. Basically, I define a global “webdb” table
to hold the content got from web for later processing , function
“httpget” is used to fetch the web data, function “load_and_upsert”
receives stock “sym” to get the quote and save to “webdb” .
As I have many stocks (“stklist” holds a few here) and my internet
connection is not stable, I am trying to use “peach” instead of “each”
to speed up the web connection, but it does not work. I understand
“peach” not allow to modify the global variable. but I really want
“peach”'s parallel feature (“each” is much slow for it download one by
one) and do need to store the fetched content into a global table. So
How to solve or bypass that conflict ?
Thanks for your help.
Halley.
// **** This script fetch stk quote from web ********
httpget:{[host;location] r:(`$“:http://”,host)“GET “,location,” HTTP/
1.1\r\nHost:”,host,“\r\n\r\n”;i:first r ss “\r\n\r\n”; (i+4) _ r};
return the data from peach in the main thread and do the parsing and =
loading.
it is fast enough.
or, spawn worker processes that you command and control async to =
download and parse the data and update async the table in the main =
process.
i’ve done something like that in the past and works very well.
On 9 Mar 2012, at 10:59, bigbug wrote:
> Hi, there,
>
> I am now doing a project to download the price quote to KDB+ . And
> figure out how to use “peach” to parallel download web data and store
> in a table . I am using 32b trial version and start q with “C:\q
> \w32\q.exe -s 5” .
>
> The codes showed as below. Basically, I define a global “webdb” table
> to hold the content got from web for later processing , function
> “httpget” is used to fetch the web data, function “load_and_upsert”
> receives stock “sym” to get the quote and save to “webdb” .
>
> As I have many stocks (“stklist” holds a few here) and my internet
> connection is not stable, I am trying to use “peach” instead of “each”
> to speed up the web connection, but it does not work. I understand
> “peach” not allow to modify the global variable. but I really want
> “peach”‘s parallel feature (“each” is much slow for it download one by
> one) and do need to store the fetched content into a global table. So
> How to solve or bypass that conflict ?
>
> Thanks for your help.
>
> Halley.
>
>
>
> // **** This script fetch stk quote from web ********
>
> httpget:{[host;location] r:($":http://",host)"GET ",location," HTTP/ \> 1.1\r\nHost:",host,"\r\n\r\n";i:first r ss "\r\n\r\n"; (i+4) \_ r}; \> \> \> webdb: ([]webcontent:symbol$());
> stklist:
> = sh510050sh510060sh510880sh510030sz159902sz159901sz150019sz150013s= z150007sz150001sh601179sh601186sh601188sh601199;
>
>
> load_and_upsert: {[sym]
> webdata: httpget[“hq.sinajs.cn”;“/list=s_”,string sym];
> 1 ((string sym),“\t”,webdata);
> upsert[webdb]($webdata); // conflict with “peach”
> }
>
> peach [load_and_upsert] stklist; // not work !!! , give " k){x’:y}
> 'noupdate … " error ; but " each[load_and_upsert]stklist; " works =
!!
>
> –
> You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
> To post to this group, send email to = personal-kdbplus@googlegroups.com.
> To unsubscribe from this group, send email to = personal-kdbplus+unsubscribe@googlegroups.com.
> For more options, visit this group at = http://groups.google.com/group/personal-kdbplus?hl=en.
>
Felix, would you mind to show some demo sample code of main/slaveprocess to handle such case ?Simon, your code works very well . Thanks,I had figure out to write the data to a file, and then reload it laterto bypass the “peach” prohibition of changing global variable. Ithink it should work, though an ugly way.It is a pity that “peach” has such constrain (“peach [load_and_upsert]stklist” looks elegant. right?) I suppose it might be related to"lock" issue of parallel program and Kx takes a shortcut to avoidcomplex implementation. Is it possible to fine tune the “peach"mechanism in the future?On 3?9?, ??5?45?, Simon Garland <si…> wrote:> or as the text returned seems to include the stock symbol:>> q)f2:{13_httpget[“hq.sinajs.cn”;”/list=s_“,string x]}> q)flip(“S*”;”=“)0:peach[f2]stklist>>>>>>>> On Fri, Mar 9, 2012 at 10:35, Simon Garland <si…> wrote:> > it’s the fetch/load step that you want to do in parallel> > so if you simplify it to:>> > fetch:{(x;httpget[“hq.sinajs.cn”;”/list=s_“,string x])}>> > and then:>> > peach[fetch] stklist>> > you’ll get the fast download with the result in a format you can easily> > turn into a table>> > (note: you should use HTTP/1.0 in httpget)>> > On Fri, Mar 9, 2012 at 09:59, bigbug <matlab…> wrote:>> >> Hi, there,>> >> I am now doing a project to download the price quote to KDB+ . And> >> figure out how to use “peach” to parallel download web data and store> >> in a table . I am using 32b trial version and start q with “C:\q> >> \w32\q.exe -s 5” .>> >> The codes showed as below. Basically, I define a global “webdb” table> >> to hold the content got from web for later processing , function> >> “httpget” is used to fetch the web data, function “load_and_upsert”> >> receives stock “sym” to get the quote and save to “webdb” .>> >> As I have many stocks (“stklist” holds a few here) and my internet> >> connection is not stable, I am trying to use “peach” instead of “each”> >> to speed up the web connection, but it does not work. I understand> >> “peach” not allow to modify the global variable. but I really want> >> “peach”'s parallel feature (“each” is much slow for it download one by> >> one) and do need to store the fetched content into a global table. So> >> How to solve or bypass that conflict ?>> >> Thanks for your help.>> >> Halley.>> >> // This script fetch stk quote from web**** >> >> httpget:{[host;location] r:($":http://",host)"GET ",location," HTTP/> >> 1.1\r\nHost:",host,"\r\n\r\n";i:first r ss "\r\n\r\n"; (i+4) _ r};>> >> webdb: ([]webcontent:symbol$());> >> stklist:>> >> sh510050sh510060sh510880sh510030sz159902sz159901sz150019sz150013sz 150007sz150001sh601179sh601186sh601188sh601199;>> >> load_and_upsert: {[sym]> >> webdata: httpget[“hq.sinajs.cn”;”/list=s_“,string sym];> >> 1 ((string sym),”\t",webdata);> >> upsert[webdb]($webdata); // conflict with “peach”> >> }>> >> peach [load_and_upsert] stklist; // not work !!! , give " k){x’:y}> >> 'noupdate … " error ; but " each[load_and_upsert]stklist; " works !!>> >> –> >>
slvd:int$()<br><br>slvreg:{if[not .z.w in slvd;slvd,::.z.w]}<br>.z.pc:{if[x in slvd;slvd::slvd except x]} / remove the worker from list =<br>when it goes away<br><br>sndjob:{[f;args;cb]@'[neg count[args]#slvd;flip(job;f;args;cb)];}