Hello
I have an issue with a function running at EOD in a wdb process with multiprocessing enabled.
The process starts with "-s -2".
We have a function that loads data from a cache and appends to data from our iHDB.
We have 3 columns which are symbol columns which we enumerate against the sym file, but when we load the cache data in, the 3 columns are type long and the syms are not converted back to their values as if the sym file is not loaded properly in the process.
I am 100% sure this is a multiprocessing issue because we have multiprocessing enabled in one environment and disabled in another and in the environment where it is disabled it works fine.
This is a bit of our endofday function running in wdb
Then these are 2 functions in question:
// this function is not that important but is added here for reference as it is used
.wdb.bMSRCExtractCrtData:{[dt;rawTab]
ihdbDir:hsym .utils.checkForEnvVar .eod.IHDB;
ihdbPar:.Q.par[ihdbDir;dt;rawTab];
$[()~key ihdbPar;
`criteria xkey errorNullSchema;
{exec last (([] publishTime;injectorTime;idApp;service;serviceType),'stats) by criteria:criteria from x } ihdbPar]
}
// ISSUE --- this is the function with the problem
.wdb.buildSomeCache:{[x]
// added this because the sym file is only loaded on the worker processes
.wdb.reloadsymfile .Q.dd[.wdb.hdbsettings`hdbdir;`sym];
dt:$[null x;.z.d-1;x];
cacheTab:`tableLatest; rawTab:`table;
hdbDir:.wdb.hdbsettings`hdbdir;
prevCacheDir:.Q.par[hdbDir;dt-1;cacheTab];
.log.out[.z.h;"Getting previous cache data";rawTab];
prevCacheData:@[get;prevCacheDir;{[e].log.err[.z.h;"No directory found for previous day's table running HDB cache: ",e;()];0b}]; <<<--- here data is loaded with columns as long instead of symbols
if[not 0b ~ prevCacheData;prevCacheData:.ut.Unenumerate prevCacheData];
.log.out[.z.h;"Getting latest table data to upsert to cache";dt];
newData:.wdb.bMSRCExtractCrtData[dt;rawTab];
// see def of bMSRCExtractCrtData above
.log.out[.z.h;"Done";dt];
if[0b~prevCacheData; /if getting previous cache failed.
prevDt:last dtPars where not 0=count each key each dtPars:.Q.par[hdbDir;;cacheTab] each {[s;e]s+til 1+e-s}[.z.d-7;.z.d-2]; /check if any files in last few dates. Chose the latest par if so.
prevCacheData:$[null prevDt;[.log.err[.z.h;"Cannot find any previous cache table - using blank schema instead. WARNING: This may require manual backfill!";rawTab];0#newData];prevCacheData:get prevDt]];
latestCache:$[count newData;.[upsert;((`criteria xkey prevCacheData);newData);{.lg.e[`proc;"Failed to upsert newData to prevCacheData. Check schemas match!"]}];prevCacheData];
saveDownLoc:.Q.par[hdbDir;dt;`$string[cacheTab],"/"];
data:latestCache;
// In testing, output parameters
if[`test in key .proc.params;:(saveDownLoc;data)];
//Write Cache.
.log.out[.z.h;"Data for latest cache updated - now writing to HDB";(cacheTab;.Q.par[hdbDir;dt;`$string[cacheTab],"/"])];
saveDownLoc set .Q.en[hdbDir;] data; /write data to disc. Data on
.log.out[.z.h;"Cache Data written to HDB ";(rawTab;cacheTab;dt)];
}
how data looks:
criteria publishTime injectorTime idApp service serviceType periodIn periodOut totalIn totalOut minTime avgTime maxTime
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(`symbol$())!() 2026.05.29D12:41:33.567818000 2026.05.29D12:41:33.588000000 8679035 8679934 4460763 0 0 1 1 -1 -1 -1
()!() 2026.06.02D23:40:25.382296000 2026.06.02D23:40:25.382000000 12076648 66195 4460763 0 0 1510 1510 -1 -1 -1
(,`AAA)!,"0#BBBBB=" 2026.06.02D14:41:46.317439000 2026.06.02D14:41:46.320000000 11161624 12918266 4460763 1 1 1 1 2.691 2.691 2.691
how data should look like:
criteria publishTime injectorTime idApp service serviceType periodIn periodOut totalIn totalOut minTime avgTime maxTime
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(`symbol$())!() 2026.05.29D12:41:33.567818000 2026.05.29D12:41:33.588000000 SYMBOL1 SYMBOL4 subscription 0 0 1 1 -1 -1 -1
()!() 2026.06.01D23:40:25.366653000 2026.06.01D23:40:25.367000000 SYMBOL2 SYMBOL5 subscription 0 0 1208 1208 -1 -1 -1
(,`AAA)!,"0#BBBBB=" 2026.05.31D07:01:12.900157000 2026.05.31D07:01:12.900000000 SYMBOL3 SYMBOL6 subscription 1 1 1 1 0.199 0.199 0.199