HQ Workers disconnect on hq job forget

[kyleabbott]

In the case that hq server has a very large memory footprint (e.g., in excess of 100GB), if you run hq job forget all to release memory for old finished jobs, it succeeds in forgetting the jobs and freeing much of the memory, but often times I have noticed some (or many) of the workers lose their connection to the server during the forget operation. If the workers are set to "finish-running" they will remain active on the slurm node but are dropped from the server's worker list. It is hard to replicate this without submitting many jobs with hundreds of thousands of tasks each via JDF, but it has happened to me multiple times. What logs would need to be provided to help diagnose the issue the next time this happens?

[Kobzol]

Hi, thanks for reporting the issue. Does it take a long time to execute the hq job forget command? I wonder if the large memory deallocation (which is synchronous and blocks I/O) can lead to work connections timing out. Also, how many jobs were you forgetting? At least some order of magnitude.

[kyleabbott]

Yes, it can take several seconds if the memory consumption of hq server is large, say 120-160GB or something like that.

Number of jobs on the order of a few thousand at most, but number of tasks in those jobs can be 500k each or more.

(Edited: to answer the second question above)

[Kobzol]

We tried a potential solution in #1065. Could you try it with HQ nightly (starting from tomorrow), if it fixes the issue? Thanks you.

[kyleabbott]

That's great. Thanks! I will take a look. It will be a while before I can give you any feedback, but I will get it in as soon as I can.