How does the http request in yahoo.q work?
txt: `:http://ichart.finance.yahoo.com “GET /table.csv?s=” , (string
stock) , params , " http/1.0\r\nhost:ichart.finance.yahoo.com\r\n\r
\n";
I couldn’t find any documentation for making http requests through q
elsewhere (or have I missed?)
thanks!
On Jan 19, 2011, at 2:19 PM, K4 Monk wrote:
> How does the http request in yahoo.q work?
>
> txt: `:http://ichart.finance.yahoo.com “GET /table.csv?s=” , (string
stock) , params , " http/1.0 host:ichart.finance.yahoo.com ";
>
> I couldn’t find any documentation for making http requests through q
elsewhere (or have I missed?)
not sure if it’s documented for q specifically, but it’s pretty standard
make a filehandle of the protocol and hostname, and “send” (apply) to
that an http 1.1 request
(not sure why the example has 1.0 in the version since host: is a
1.1-only header…)
anything you can do in telnet .. 80 will work here
pro% telnet ichart.finance.yahoo.com 80
Trying 76.13.116.133…
Connected to any-chart.finance-global.a01.yahoodns.net.
Escape character is ‘^]’.
GET /table.csv?s=AAPL http/1.1
host:ichart.finance.yahoo.com
HTTP/1.1 200 OK
Date: Thu, 20 Jan 2011 00:31:03 GMT
P3P: policyref=“http://info.yahoo.com/w3c/p3p.xml”, CP=“CAO DSP COR
CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi
PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC
GOV”
Cache-Control: private
Connection: close
Transfer-Encoding: chunked
Content-Type: text/csv
fe67
Date,Open,High,Low,Close,Volume,Adj Close
2011-01-19,348.35,348.60,336.88,338.84,40494700,338.84
2011-01-18,329.52,344.76,326.00,340.65,66831600,340.65
2011-01-14,345.89,348.48,344.44,348.48,10998600,348.48
2011-01-13,345.16,346.64,343.85,345.68,10599300,345.68
2011-01-12,343.25,344.43,342.00,344.42,10790700,344.42
2011-01-11,344.88,344.96,339.47,341.64,15814500,341.64
2011-01-10,338.83,343.23,337.17,342.45,16000400,342.45
2011-01-07,333.99,336.35,331.90,336.12,11096800,336.12
2011-01-06,334.72,335.25,332.90,333.73,10709500,333.73
2011-01-05,329.55,334.34,329.50,334.00,9058700,334.00
2011-01-04,332.44,332.50,328.15,331.29,11038600,331.29
2011-01-03,325.64,330.26,324.84,329.57,15883600,329.57
2010-12-31,322.95,323.48,321.31,322.56,6911000,322.56
2010-12-30,325.48,325.51,323.05,323.66,5624800,323.66
..
pro% q
q):http://ichart.finance.yahoo.com "GET /table.csv?s=AAPL http/1.0 host:ichart.finance.yahoo.com " "HTTP/1.1 200 OK Date: Thu, 20 Jan 2011 00:33:51 GMT P3P: policyref="h.. q)-1:http://ichart.finance.yahoo.com "GET /table.csv?s=AAPL
http/1.0 host:ichart.finance.yahoo.com ";
HTTP/1.1 200 OK
Date: Thu, 20 Jan 2011 00:32:56 GMT
P3P: policyref=“http://info.yahoo.com/w3c/p3p.xml”, CP=“CAO DSP COR
CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi
PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC
GOV”
Cache-Control: private
Connection: close
Content-Type: text/csv
Date,Open,High,Low,Close,Volume,Adj Close
2011-01-19,348.35,348.60,336.88,338.84,40494700,338.84
2011-01-18,329.52,344.76,326.00,340.65,66831600,340.65
2011-01-14,345.89,348.48,344.44,348.48,10998600,348.48
2011-01-13,345.16,346.64,343.85,345.68,10599300,345.68
2011-01-12,343.25,344.43,342.00,344.42,10790700,344.42
2011-01-11,344.88,344.96,339.47,341.64,15814500,341.64
2011-01-10,338.83,343.23,337.17,342.45,16000400,342.45
2011-01-07,333.99,336.35,331.90,336.12,11096800,336.12
2011-01-06,334.72,335.25,332.90,333.73,10709500,333.73
2011-01-05,329.55,334.34,329.50,334.00,9058700,334.00
2011-01-04,332.44,332.50,328.15,331.29,11038600,331.29
2011-01-03,325.64,330.26,324.84,329.57,15883600,329.57
2010-12-31,322.95,323.48,321.31,322.56,6911000,322.56
2010-12-30,325.48,325.51,323.05,323.66,5624800,323.66
..
1.0 is used to avoid ‘chunk’ errors
On Jan 20, 12:34?am, Aaron Davies <aaron.dav…> wrote:> On Jan 19, 2011, at 2:19 PM, K4 Monk wrote:>> > How does the http request in yahoo.q work?>> > txt: :http://ichart.finance.yahoo.com"GET /table.csv?s=" , (string stock) , params , " http/1.0\r\nhost:ichart.finance.yahoo.com\r\n\r\n";>> > I couldn't find any documentation for making http requests through q elsewhere (or have I missed?)>> not sure if it's documented for q specifically, but it's pretty standard>> make a filehandle of the protocol and hostname, and "send" (apply) to that an http 1.1 request>Thanks for the reply. I am confused not by the request sent throughHTTP, but by the call within q itself. (GET also happens to be asystem command in Linux). I understand using telnet for the purpose aswell.What I want to know is that when reading the line left to right itjust appears as a string, but giving it a prefix of :http causes q toactually parse this as a HTTP request?</aaron.dav…>
To: personal-kdbplus@googlegroups.com
X-Mailer: Apple Mail (2.1081)
nope. it is special case of call be name.
ex:
f:{1+x}
`f 2
on the same idea when the symbol starts with `:http:… if will open a =
socket to that host and port and will send the text that’s passed as =
argument.
it reads the reply and closes the connexion.
you can call any http methods on the remote server as long you =
understand the working of the http and formatting of the messages.
this feature is there because it is useful in some applications but =
there is no other raw socket operations in q.
felix
On Jan 20, 10:32?am, Felix Lungu <felix.lu…> wrote:> nope. it is special case of call be name.> ex:>> f:{1+x}>> f 2>> on the same idea when the symbol starts with :http:… if will open a socket to that host and port and will send the text that’s passed as argument.> it reads the reply and closes the connexion.>> you can call any http methods on the remote server as long you understand the working of the http and formatting of the messages.>> this feature is there because it is useful in some applications but there is no other raw socket operations in q.>> felix>thank you Felix. So, you are saying that the http GET request isactually a special case of an implicit “hopen”? Sorry, but I stilldon’t see it. To add to my confusion, ichart.finance.yahoo.com occurstwice.</felix.lu…>
The second “ichart.finance.yahoo.com” is the host that holds the
resource you are requesting.
See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23
The first “ichart.finance.yahoo.com” is the host (and implicit port)
to which the q process is connecting.
This may be the host holding the resource, but more interestingly, it
may be a proxy server.
This is not using “hopen” since it is not speaking q’s wire protocol
which has an initial handshake involving the passing of username and
password and determining whether IPC compression can be used. The
socket from hopen stays open until closed or until q thinks there’s a
problem with the stream. (There’s other stuff in the protocol with
respect to endianness, compression, sync vs async, etc.)
The http hsym mechanism basically opens a raw socket on which the
argument string is sent.
Once the payload has been sent, it slurps up the bytes of the
response, closes the socket, and returns the bytes as a string..
Quite different from hopen.
ACS
txt: `:http://ichart.finance.yahoo.com “GET /table.csv?s=” , (string
stock) , params , " http/1.0\r\nhost:ichart.finance.yahoo.com\r\n\r
\n";