How can I efficiently iterate over some values while making small updates to a large state vector (in my actual application it’s a dictionary).
For example, if I use over it’s very slow because the entire state vector is copied at each iteration:
q)n:prd 7#10
q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]
2108 402655360
But if I were to use a global state (what I’m trying to avoid), it’s much more efficient:
q)xx:n#1f
q)\ts {xx:rand 1f}'[100?n]
0 2912
Thanks,
John
Hi John,
You can store your large state vector in the global state and then update it in place.
q)n:prd 7#10
q)`xx set n#1f
q)\ts @[;;:;rand 1f]/[`xx;100?n]
0 1872
Best,
Osvaldo
No need to create the projection in this case
q)\ts @/[`xx;100?n;:;rand 1f]
0 1648
-Ajay
If you can generate the values you’d like to update to separately then it’d be easier to use replace as demonstrated in https://code.kx.com/q/ref/amend/#replacement
q)n:prd 7#10
q)v:n#1f // the initial state of your vector
q)indx:100?n // the indices to update
q)nv:count[indx]?1f // new values
q)\ts @[v;indx;:;nv]
//11 134219232
q)\ts {@[x;y;:;rand 1f]}/[`xx;100?n]
0 2160
q)count distinct xx
101
Any other idea with over?
Osvaldo
Thanks for the suggestions, but I should have clarified two points earlier:
The actual calculation is path-dependent and non-trivial, so the solution should work for any arbitrary function of the current state and an element of the iterated vector.
I want to avoid updating globals because ultimately I want to be able to run this calculation many times in parallel with peach. And AFAIK, it’s not possible to use thread local globals directly from q.
I think you are not including the creation of the variable. when you make it a global. When you do your timings.
Making it hard to see what your really comparing and wanting to optimize.
Using global will reduce memory but dont think there is much difference in speed.
q)\ts {x[y]:rand 1f; x}/[n#1f;100?n]
1117 402655360
q)\ts xx:n#1f ; {x[y]:rand 1f; x}/[xx;100?n]
1168 268437936