Hello everyone,
I’ve recently been doing development on my local machine and at some point have several jobs that get launched in parallel from a single workchain instance. It seems someone had a similar question Is there a Desktop Job Scheduler for MacOS - #3 by mbercx and I’m wondering if there is a simpler way to just tell AiiDA to only run one JobCalc at a time? Otherwise, I’ll just deal with it or install a scheduler as mentioned. Thanks.
Nathan
If your WorkChain is not nested (i.e., it only submits CalcJobs and does not submit other WorkChains), and you’re only running one WorkChain at a time, you can follow this simple approach:
# Allocate one slot for the WorkChain itself and another for a CalcJob.
# Any additional CalcJobs will wait until a slot is available.
verdi config set daemon.worker_process_slots 2
verdi daemon restart
If your WorkChain is nested (i.e., it submits other WorkChains), you should increase the worker_process_slots based on the depth of nesting. For example, if your WorkChain submits two nested WorkChains, you will need two more slots to ensure smooth execution without blocking.
This configuration should only be used for testing purposes. If the number of submitted WorkChains is equal to or greater than the available worker slots, the WorkChan can deadlock, as there will not be enough slots for all the processes to complete.
When moving to production, set the default value back for worker_process_slots:
verdi config set daemon.worker_process_slots 200
verdi daemon restart
I think it really depend on your goals.
If during the development, you want to check if every pieces of workchain steps are working as expect by running it as a whole without starving your laptop resource, I’d recommend to not start daemon using engine.run() instead of engine.submit().
If you want to check that some steps in your workchain can spin up calculations in parallel but again not starving your laptop resources with hundreds of system processes. Using the approach mentioned by @Xing above.
If you want to use your PC to run production workchain but not starving your laptop resources. I think it is better to have a scheduler and the hyperqueue is probably want you are looking for. We have aiida-hyperqueue plugin that developed from the team and being used also for aiidalab-qe deployment. (for how to set up this, I reply in another post by @nkeilbart at Implementing the Flux scheduler from LLNL - #4 by jusong.yu)