Hi, KDB FolksI am trying to understand the example for QForMortals. Can you explainwhat is the execution algorithm and why it is faster than each[(1#)]?Thanks flip enlist 1001 1002 1004 10031001100210041003
To: personal-kdbplus@googlegroups.com
X-Mailer: Apple Mail (2.1084)
> I am trying to understand the example for QForMortals. Can you explain
> what is the execution algorithm and why it is faster than each[(1#)]?
>
> q)flip enlist 1001 1002 1004 1003
> 1001
> 1002
> 1004
> 1003
first, “each[(1#)]” is a very odd way to write the operation–using the =
“each” form of the iteration adverb, normal would be to infix as =
“(1#)each”, e.g. "(1#)each 1001 1002 1004 1003, which is somewhat =
faster.
q)\t do[100000;each[1#]1001 1002 1004 1003]
188
q)\t do[100000;(1#)each 1001 1002 1004 1003]
155
the speedup here probably comes from not having to create the projection =
“each[(1#)]” first–note that calling it directly as a prefix dyadic, =
rather than as a projection with missing argument provided, has =
identical performance to infix:
q)\t do[100000;each[1#;1001 1002 1004 1003]]
155
much more common would be to use dyadic each as “1#'”, e.g. “1#'1001 =
1002 1004 1003”, which is more than twice as fast as “each[(1#)]”
q)\t do[100000;1#'1001 1002 1004 1003]
73
the performance improvement here may have to do with the fact that =
“each” is a compiled function (aka “lambda”), not a primitive, so it =
could be function call overhead we’re seeing.
that said, the most common way to do what you’re doing would be “enlist =
each”, e.g. “enlist each 1001 1002 1003 1004”, which is only 60 =
*nano*seconds slower than “flip enlist” in this case:
q)\t do[100000;enlist each 1001 1002 1004 1003]
59
q)\t do[100000;flip enlist 1001 1002 1004 1003]
53
note that using the k syntax for iteration of a monadic function has =
identical performance to flip enlist
q)\t do[100000;enlist’[1001 1002 1004 1003]]
53
as to why “flip enlist” has its particular performance profile, i’m not =
entirely sure. it’s possible that “flip” is a “more primitive” call, =
i.e. its C implementation calls fewer other functions than “#”'s.
of course, these are all more or less O(n)
q)\t do[100000;enlist each til 10]
139
q)\t do[100000;enlist each til 100]
728
q)\t do[100000;enlist each til 1000]
6613
q)\t do[100000;flip enlist til 10]
119
q)\t do[100000;flip enlist til 100]
578
q)\t do[100000;flip enlist til 1000]
5144=
Return-Path: rohit.tripathi@capgemini.com
Do you think 1# also invokes C’s internal pointer arithmetic which doesn’t =
happen in the other case?
-----Original Message-----
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegrou=
ps.com] On Behalf Of Aaron Davies
Sent: Thursday, October 06, 2011 10:18 AM
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] Try to understand enlist
> I am trying to understand the example for QForMortals. Can you explain
> what is the execution algorithm and why it is faster than each[(1#)]?
>=20
> q)flip enlist 1001 1002 1004 1003
> 1001
> 1002
> 1004
> 1003
first, “each[(1#)]” is a very odd way to write the operation–using the “ea=
ch” form of the iteration adverb, normal would be to infix as “(1#)each”, e=
.g. "(1#)each 1001 1002 1004 1003, which is somewhat faster.
q)\t do[100000;each[1#]1001 1002 1004 1003]
188
q)\t do[100000;(1#)each 1001 1002 1004 1003]
155
the speedup here probably comes from not having to create the projection “e=
ach[(1#)]” first–note that calling it directly as a prefix dyadic, rather =
than as a projection with missing argument provided, has identical performa=
nce to infix:
q)\t do[100000;each[1#;1001 1002 1004 1003]]
155
much more common would be to use dyadic each as “1#'”, e.g. “1#'1001 1002 1=
004 1003”, which is more than twice as fast as “each[(1#)]”
q)\t do[100000;1#'1001 1002 1004 1003]
73
the performance improvement here may have to do with the fact that “each” i=
s a compiled function (aka “lambda”), not a primitive, so it could be funct=
ion call overhead we’re seeing.
that said, the most common way to do what you’re doing would be “enlist eac=
h”, e.g. “enlist each 1001 1002 1003 1004”, which is only 60 *nano*seconds =
slower than “flip enlist” in this case:
q)\t do[100000;enlist each 1001 1002 1004 1003]
59
q)\t do[100000;flip enlist 1001 1002 1004 1003]
53
note that using the k syntax for iteration of a monadic function has identi=
cal performance to flip enlist
q)\t do[100000;enlist’[1001 1002 1004 1003]]
53
as to why “flip enlist” has its particular performance profile, i’m not ent=
irely sure. it’s possible that “flip” is a “more primitive” call, i.e. its =
C implementation calls fewer other functions than “#”'s.
of course, these are all more or less O(n)
q)\t do[100000;enlist each til 10]
139
q)\t do[100000;enlist each til 100]
728
q)\t do[100000;enlist each til 1000]
6613
q)\t do[100000;flip enlist til 10]
119
q)\t do[100000;flip enlist til 100]
578
q)\t do[100000;flip enlist til 1000]
5144
–=20
Submitted via Google Groups
> as to why “flip enlist” has its particular performance profile, i’m not e=
ntirely sure. it’s possible that “flip” is a “more primitive” call, i.e. it=
s C implementation calls fewer other functions than “#”'s.
Ah, sorry missed this part
-----Original Message-----
From: Tripathi, Rohit=20
Sent: Thursday, October 06, 2011 11:39 AM
To: ‘personal-kdbplus@googlegroups.com’
Subject: RE: [personal kdb+] Try to understand enlist
Do you think 1# also invokes C’s internal pointer arithmetic which doesn’t =
happen in the other case?
-----Original Message-----
From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegrou=
ps.com] On Behalf Of Aaron Davies
Sent: Thursday, October 06, 2011 10:18 AM
To: personal-kdbplus@googlegroups.com
Subject: Re: [personal kdb+] Try to understand enlist
> I am trying to understand the example for QForMortals. Can you explain
> what is the execution algorithm and why it is faster than each[(1#)]?
>=20
> q)flip enlist 1001 1002 1004 1003
> 1001
> 1002
> 1004
> 1003
first, “each[(1#)]” is a very odd way to write the operation–using the “ea=
ch” form of the iteration adverb, normal would be to infix as “(1#)each”, e=
.g. "(1#)each 1001 1002 1004 1003, which is somewhat faster.
q)\t do[100000;each[1#]1001 1002 1004 1003]
188
q)\t do[100000;(1#)each 1001 1002 1004 1003]
155
the speedup here probably comes from not having to create the projection “e=
ach[(1#)]” first–note that calling it directly as a prefix dyadic, rather =
than as a projection with missing argument provided, has identical performa=
nce to infix:
q)\t do[100000;each[1#;1001 1002 1004 1003]]
155
much more common would be to use dyadic each as “1#'”, e.g. “1#'1001 1002 1=
004 1003”, which is more than twice as fast as “each[(1#)]”
q)\t do[100000;1#'1001 1002 1004 1003]
73
the performance improvement here may have to do with the fact that “each” i=
s a compiled function (aka “lambda”), not a primitive, so it could be funct=
ion call overhead we’re seeing.
that said, the most common way to do what you’re doing would be “enlist eac=
h”, e.g. “enlist each 1001 1002 1003 1004”, which is only 60 *nano*seconds =
slower than “flip enlist” in this case:
q)\t do[100000;enlist each 1001 1002 1004 1003]
59
q)\t do[100000;flip enlist 1001 1002 1004 1003]
53
note that using the k syntax for iteration of a monadic function has identi=
cal performance to flip enlist
q)\t do[100000;enlist’[1001 1002 1004 1003]]
53
as to why “flip enlist” has its particular performance profile, i’m not ent=
irely sure. it’s possible that “flip” is a “more primitive” call, i.e. its =
C implementation calls fewer other functions than “#”'s.
of course, these are all more or less O(n)
q)\t do[100000;enlist each til 10]
139
q)\t do[100000;enlist each til 100]
728
q)\t do[100000;enlist each til 1000]
6613
q)\t do[100000;flip enlist til 10]
119
q)\t do[100000;flip enlist til 100]
578
q)\t do[100000;flip enlist til 1000]
5144
–=20
Submitted via Google Groups