Get Data from Web Services

Community Support
1

Received: by 10.100.10.15 with SMTP id 15mr89765anj.0.1219353099060; Thu, 21
Aug 2008 14:11:39 -0700 (PDT)
Date: Thu, 21 Aug 2008 14:11:39 -0700 (PDT)
X-IP: 199.89.64.176
User-Agent: G2/1.0
X-Google-Token: dH4yqAwAAABI1HQl4x9IdlmJ_zyeSrg7
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 7.0; MSBrowserIE7; Windows NT
5.1; msie6xpv1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727;
.NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; MS-RTC LM
8; MSIE6ENV21; MSIE6ENV21),gzip(gfe),gzip(gfe)
X-HTTP-Via: 1.1 piipx1:8080 (DataReactor/4.1.0p24d2)
Message-ID: <19d8d5e7-abdf-4167-8a33-769a532c3c7a@k37g2000hsf.googlegroups.com>
Subject: Get Data from Web Services
From: Ruslan
To: “Kdb+ Personal Developers”
X-Google-Approved: simon.garland@gmail.com via email at 2008-08-22 06:03:34

What is the best way to get data via http://server-foo/page?parm=value
curl type a query?
Using the cookbook example, https://code.kx.com/svn/cookbook_code/yahoo.q
I run into timeout issues if the query/data pull takes longer than 1
min and an interpreter issue if my url contains special characters
such as a dash(-).
I fixed the special char issues with type casting from string:
“urlget: `$”:http://server-foo:8080";" before concatenating it to the
query string, “GET /page?parm=value\r\n\r\n”, but I am still having
issues with the request timeouts.

Any suggestions?

-Ruslan

2
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Menlo; color: #000000; background-color: #ffffff}span.s1 {font-variant-ligatures: no-common-ligatures}

yahoo[1; (ibm,goog,cat,intc,msft,aapl)]

Is there a way to keep the socket open and run multiple requests? 

3

the yahoo function described here http://code.kx.com/wiki/Cookbook/ProgrammingExamples uses the “single shot” request: 

  `:http://ichart.finance.yahoo.com “GET /table.csv?s=”,(string stock:stocks[i]),parms," http/1.0\r\nhost:ichart.finance.yahoo.com\r\n\r\n"

the single shot request closes the connection after the result arrives (described here http://code.kx.com/wiki/Reference/hopen) .

if you wanted to heep the connection, the first step you would need to try is to keep a reference to the connection to yahoo:

q)hopen `:http://ichart.finance.yahoo.com

'type

that doesn’t work…

q)hopen `:ichart.finance.yahoo.com:80

3i

'parse

(i think kdb doesn’t like a text response sent from yahoo.com.  kdb only expects messages in it’s IPC format)

at this point, i’d google “yahoo table.csv” - or investigate a websocket interface to yahoo..

if you’re want live data, you could look at the many CFD markets out there.  they are happy to send you live data.

jack