How to control the number of running subworkchains submitted by a bigger workchain

General Usage
1

Hi everyone

I have a workchain that basically submits 500 other subworkchains. To do so, I define a step in the outline of the workchain that loops over 500 cases and submits the corresponding subworkchains.

The issue I have with this is that all the subworkchains are submitted and start running without control, collapsing all the resources of my workstations.

Hence, I would like to find a way to control the number of subworkchains that are simulateneously running in such a way the big workchain does not submit more subworkchains if N subworkchains are simultaneously running.

My first idea to achieve this was to check the is_terminated attribute of the previously submitted subworkchains. I checked how many were still active in a while loop that does not end until the number of active subworkchains is below N.

With this idea, I manage to stop the workchain from submitting more than N subworkchains.

The problem I am facing now is that I realized that the submitted subworkchains remain in the state “created” forever, probably due to the fact that the step of the outline of the workchain in which they have been submitted is still running. Therefore, the submitted subworkchains never finish and hence, the rest of the subworkchains will never be submitted.

Any suggestion on how to proceed or about how to control the number of simultaneous subworkchains running?

1 Like
2

Good question @jgarridoa . I think your analysis of the problem is spot on. If you are not returning from the workchain step, but waiting (I imagine by keeping the loop running or even having a sleep() in there) forever, the daemon worker can never get around to run the subprocesses. The most immediate solution is to simply start another daemon worker. You can do this from the CLI with verdi daemon incr or verdi daemon incr N to start N additional workers. Note that you should not run more workers than CPUs on your machine typically.

Another approach that users of AiiDA typically use is asking the question whether the top level workchain really needs to be a workchain. Is it “just” launching the 500 subprocesses? If so, you could just do this in a script. Is there really that much value to have a workchain node in the provenance graph as the top caller? You can also group the subprocesses that are launched in a Group to keep track of them (think of it like a folder that you put files in, but instead of files, you put Nodes in).

When you go the later route, you might want to take a look at the aiida-submission-controller package that is designed actually for this purpose.

3

Hi, I’d like to add another approach to this discussion that we recently used, in case you need/want to use a WorkChain instead of a script as mentioned before.

You could store your different cases in the context and insert a while loop in your outline. You define a maximum number N of sub-processes to be submitted and as long as the list (or whatever datatype) in the context contains further entries, the corresponding method submits N sub-processes at a time.
One has to note that the next batch of sub-processes will only be submitted once all of the previous N sub-WorkChains are terminated. So on the contrary, in case your sub-processes have significantly different runtimes, you might waste some time as you don’t always fill up your running sub-processes to the maximum number. However, you ensure that the used resources are limited, accepting the previously mentioned disadvantage.

1 Like
4

I tried by increasing the number of workers, but it did not change the result.

Using a big workchain to run all the subworkchains is not strictly neccesary, but it was useful to exploit two things: travelling easily the provenance graph and the possibility of kill all the subworkchains by killing the big workchain.

By using groups (as you suggest), I can keep track of all the subworkchains to easily check their properties later. Thanks for the suggestion because I did not realize about that possibility.

However, I still have the issue of how to kill all the subworkchains (In case I need to). What I am doing now is running a python script that submits only N simultaneous subworkchains and groups all the subworkchains nodes in a group node. I am storing the PID of the python process to kill it if neccesary. However, I do not know an easy way to, for example, kill all the N running subworkchains if neccesary. Of course, I can always write verdi process kill PK , but that is tedious, obviously.

Any idea on how to solve that?

5

Hi.
Thanks for your suggestion.

My problem is that all subworkchains have significantly different running times. I am running DFT calculations with Siesta and the workchains can take from hours to literally days. This is the reason why it is essential for me not to wait for a batch of N subworkchains to finish.

6

I tried by increasing the number of workers, but it did not change the result.

My bad, you are right, it is possible that the worker that is running the main workchain picked up the subprocesses and it would then still be stuck. Forget this approach, it is indeed not viable.

My problem is that all subworkchains have significantly different running times. I am running DFT calculations with Siesta and the workchains can take from hours to literally days. This is the reason why it is essential for me not to wait for a batch of N subworkchains to finish.

I am afraid that if you want to keep the top-level workchain, there is currently not really a better approach than the one suggested by @t-reents . Is there any way you can “guess” the runtime of the subprocesses by their inputs? If so, you could group them by expected runtime and still run them in batches without losing too much time.

In the end, it would be great if AiiDA allowed your use case, becausa it is not the first time we have come across it, but it would require some new functionality. One solution would be if the workchain interface would become asynchronous code, instead of the current synchronous. Back in the day, we explicitly decided against this, because writing workchain code was already quite complicated for novice users, and forcing them to write asynchronous code would be worse. But that has improved a lot since and writing async code in Python is now relatively easy. We are thinking whether we can allow workchains to optionally write async code, but that is a feature for the future and don’t expect that to come very soon. Alternatively we could come up with a different solution that just relies on synchronous code that would allow a workchain step to temporarily “yield” control for the worker to run other processes and come back later, but not sure yet if that is possible Would have to think about it a bit.

7

Is there any way you can “guess” the runtime of the subprocesses by their inputs? If so, you could group them by expected runtime and still run them in batches without losing too much time.

No, there is not. At least, not to my knowdledge.

In order to close this topic, I write here what I have done. It is not elegant nor smart but it does the job and someone might find it useful in the future.

In the end I have written a python script that submits N subworkchains and adds them into a group. Then, it keeps track of the number of running subworkchains. When less than N subworkchains are simultaneously running, it submits another one and it adds it to the group. With this python script I get a controlled way to submit the subworkchains.

In order to have the chance of cancelling all the remaning submissions, I store in a file the PID of the python script and if I ever want to cancel, I simply kill the process with kill PID.

If I want to kill the already running subworkchains, I use the following python function:

import subprocess

def kill_workchains(workchain_pks):
    # Build the command to kill workchains
    command = ["verdi", "process", "kill"] + [str(pk) for pk in workchain_pks]

    # Execute the command
    try:
        result = subprocess.run(command, check=True, capture_output=True, text=True)
        print(result.stdout)
    except subprocess.CalledProcessError as e:
        print(f"Error: {e}")
        print(f"Command output:\n{e.output}")

The input is a list with the PKs of the running subworkchains. They can be easily obtained by loading the group in which the subworkchains are stored. Then, with a simple loop, all the PKs of the running subworkchains can be stored in a list.

With the combination of the submission script + the killing function I basically get the functionality I wanted to get from a big submission workchain: an easy way to submit and kill everything.

Hope it helps!

8

Thanks a lot for posting your solution @jgarridoa . What you describe is exactly the approach of aiida-submission-controller. But fine to use your own version of it of course.

Regarding the killing of processes. If you want to do this from Python, you might also just use the Python API directly:

from aiida.engine.processes.control import kill_processes
kill_processes([load_node(pk) for pk in workchain_pks])
9

You can use AiiDA-WorkTree, a framework for designing flexible node-based workflows using AiiDA.

WorkTree allows tasks to run as soon as their dependencies are met. Thus, it minimizes idle time as tasks don’t have to wait for an entire step (level) to complete, as in the WorkChain, as pointed out by @t-reents and @sphuber.

I just added a parameter max_number_jobs to limit the maximum number of subprocesses running simultaneously within the WorkTree.

One can set:

wt.max_number_jobs = 5

This ensures that no more than five subprocesses will be running concurrently within the WorkTree.

Please check this example.