# AiiDA Submission Controller Help **URL:** \ **Category:** New to AiiDA\ **Tags:** question\ **Created:** [November 29, 2023, 2:51pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182 "2023-11-29T14:51:55Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\ **Post date:** [November 29, 2023, 2:51pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/1 "2023-11-29T14:51:55Z")
Hi all, Finally getting around to implementing my WorkChain and could use some help with aiida\_submission\_controller First, I want to know if what I want is possible - it seems to be, by using the FromGroupSubmissionController My ideal steps are: 1. Provide a list of input files to my WorkChain 2. It does some processing and launches a number of Gaussian calculations on a remote cluster, this does not need to be limited as the slurm scheduler will take care of that 3. These need to then be processed on a local program, but I can’t run 100 at a time. This is where I’d like to use the FromGroupSubmissionController to run, e.g. 4 at a time. Might need to periodically check (or rerun) to see if new calculations need to be submitted 4. Then I want to resubmit a calculation using those results to the cluster, and don’t need the controller (and really, should not have the controller, as that would be limiting) **So, firstly, just want to verify that this is possible!** Next, I already have the base WorkChain for all the above steps, which I can break apart for the submission controller. I see these use Groups and extras. I imagine I can just modify the WorkChain to store all the Gaussian calculation nodes in a group with a label that I provide to the controller. **However, for my second question - can someone explain the ‘extras’ that this controller works with? How, or do I, need to modify my Gaussian calculation nodes for this?** Thanks in advance! ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [November 29, 2023, 9:40pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/2 "2023-11-29T21:40:52Z")
Hi @kmlefran , I am not the author of the `aiida-submission-controller` but I have had a quick look and think I have a good understanding of how it is supposed to work. In principle, you subclass one of the controllers and implement the necessary methods and then to use it, you repeatedly call `submit_new_batch` and it will submit as much new processes as there are slots available. The examples show that you can typically add this call in a Python script and then call this using a bash loop: `while true ; do verdi run add_in_batches.py ; sleep 5 ; done` Now I am not sure that this will work when integrating with a `WorkChain`. At some point you will have to add a step in your workchain that does this loop to submit new processes as slots come available, but this while-loop will be blocking and the workchain cannot do anything else. With the current API, it is not possible for you to relinquish control to the workchain and have it do other things, such as submit more Gaussian calculations or check on their status. But that being said, it is not _necessary_ to wrap all of your functionality in a single `WorkChain`. Rather, you can have a simple Python script that launches the gaussian calculations and then starts the submission controller to launch the post-processing jobs locally. > **However, for my second question - can someone explain the ‘extras’ that this controller works with? How, or do I, need to modify my Gaussian calculation nodes for this?** Extras in AiiDA function kind of like “tags”. Any node can have extras and they are essentially like a JSON dictionary. The submission controller uses these extras to tag certain nodes in order to figure out which nodes to submit next. ---
**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\ **Post date:** [December 1, 2023, 2:57pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/4 "2023-12-01T14:57:04Z")
Okay, I think I’ve figured it out. Thanks for your reply! Like you said, it doesn’t need to be in a single `WorkChain`, so I’ll have an initial `WorkChain` that does my initial processing then the Gaussian, and I’ll have a script similar to add\_in\_batches running, but using the `FromGroupSubmissionController`. This will launch a second `WorkChain`, and I’ll have a similar script looking for outputs from that `WorkChain` to submit back to Gaussian. Just need to split my code apart, and add some modifications, but I think it will work. For the extras, I was having trouble figuring out how to add them for the `FromGroupSubmissionController` so I’ll just put my solution here if anyone does a search in the future ```auto from aiida.orm.extras import EntityExtras my_node_extras = EntityExtras(my_node) my_node_extras.set('test_key','test_value') # verify extras added with my_node.extras ``` ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [December 1, 2023, 3:18pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/5 "2023-12-01T15:18:53Z")
Good to hear you found a solution. Regarding the extras, those you can access directly on the node: ```python my_node.base.extras.set('test_key','test_value') my_node.base.extras.set_many({ 'a': 1, 'b': 2, }) my_node.base.extras.get('a') my_node.base.extras.delete('a') ``` etc. ---
**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\ **Post date:** [December 1, 2023, 7:06pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/6 "2023-12-01T19:06:18Z")
Ah, I was missing the .base I had been trying my\_node.extras.set(…) etc, but obviously my\_node.extras is just a dictionary Thanks! ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 18, 2024, 11:44pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/7 "2024-01-18T23:44:03Z")
Hi @kmlefran, would you be open to sharing your final workflow / WorkChain? Is it per-chance available somewhere on GitHub? I am currently dealing with similar issue and am interested in technical details. Thanks! ---
**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\ **Post date:** [January 19, 2024, 5:00pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/8 "2024-01-19T17:00:12Z")
Hi! I just made the repository public and released. on PyPi. > **[GitHub - kmlefran/aiida-aimall](https://github.com/kmlefran/aiida-aimall)** > > Contribute to kmlefran/aiida-aimall development by creating an account on GitHub. Of interest to you: - I created an example folder with the notebook I’m using to run currently. - The controllers.py in aiida\_aimall/ shows how I defined the four controllers I use - The calculations.py and work chains.py show how I added the extras as tags in my output. (Search for “extras”) Here I used a chemical identifier, a SMILES code, as the tag. Note that when you are using the controller in your case the tags need to be unique! That’s all the time I have now to write an explanation, but take a look and let me know if I can help you any more! ---
**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\ **Post date:** [January 19, 2024, 5:02pm UTC](https://aiida.discourse.group/t/aiida-submission-controller-help/182/9 "2024-01-19T17:02:00Z")
Oh, and I didn’t include the data I’m running on, so the cell looking for the files won’t execute properly