# How to Reset/fix FromGroupSubmissionController

**URL:** <https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426>\
**Category:** General Usage\
**Tags:** question\
**Created:** [July 8, 2024, 7:26pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426 "2024-07-08T19:26:32Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 8, 2024, 7:26pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/1 "2024-07-08T19:26:32Z")

</div>

Hi all,

I have been working with an `aiida-submission-controller`, `FromGroupSubmissionController` and have come across an issue where it is saying that there are no available slots. However, I have a limit of 100, and only 6 processes of the group that the controller uses are showing in `verdi process list`.

Is there a way I can reset the controller/force it to update or recheck the active processes, or otherwise fix this issue?

I’m using the `SmilesToGaussianController` in my [aiida-aimall](https://github.com/kmlefran/aiida-aimall) package

```python
smile_controller = SmilesToGaussianController(
    parent_group_label = 'smiles', # A group of orm.Str for molecule SMILES
    group_label = 'g16_opt', # group to put Workchain nodes into
    code_label='gaussian@cedar',
    g16_opt_params=orm.Dict(dict={
        'link0_parameters': {
            '%chk':'aiida.chk',
            "%mem": "2000MB",
            "%nprocshared": 4,
        },
        'functional':'wb97xd',
        'basis_set':'aug-cc-pvtz',
        'route_parameters': {'opt': None, 'freq':None,'Output':'WFX'},
        "input_parameters": {"output.wfx\n": None, "output2.wfx":None},
    }),
    wfxgroup = "opt_wfx",
    nprocs = 4,
    mem_mb = 3200,
    time_s = 60*60*24*7,
    max_concurrent = 100
)

smile_controller.num_available_slots

```

prints 0,

Again, only 6 processes are showing in `verdi process list`, and the limit should be 100.

Any help would be appreciated, thanks!

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 8, 2024, 8:02pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/2 "2024-07-08T20:02:05Z")

</div>

So, working through and I just found all the processes that it thinks are active using the `get_query` part of the controller. However, the processes that returned have `process_state` as ‘excepted’, so I still need to see if there is a way to get these out of what the controller views as active processes

---

<div class="post-metadata">

**Author:** ![t-reents](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/t-reents/32/82_2.png) [@t-reents](https://aiida.discourse.group/u/t-reents)\
**Post date:** [July 9, 2024, 8:11am UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/3 "2024-07-09T08:11:00Z")

</div>

Hi @kmlefran!

The number of available slots is always updated/recalculated when you call `SubmissionController.num_available_slots` or `SubmissionController.submit_new_batch()`. Moreover, excepted processes should be `sealed` and therefore ignored in the `get_query(only_active=True)` call. Did you specify `only_active=True` when you checked the `get_query()` method? This is relevant because `get_query()` would otherwise list all submitted processes, which would be correct in that scenario.

It would be good to get more details. How many smiles, i.e. `Str` nodes, are in the initial group and how many processes are already in the final group? E.g. how many jobs do you actually expect that need to be submitted.

Another detail that I realized in your implementation:  
Does the following actually work? ([aiida-aimall/aiida\_aimall/controllers.py at main · kmlefran/aiida-aimall · GitHub](https://github.com/kmlefran/aiida-aimall/blob/main/aiida_aimall/controllers.py#L116C9-L116C66))

```auto
return inputs, WorkflowFactory(self.WORKFLOW_ENTRY_POINT)

```

The returned tuple is passed to `aiida.engine.submit` in the `BaseSubmissionController` which expects the opposite order in principle. I made a small test and in that case it actually failed.

Happy to investigate this further once you provide more information.

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 9, 2024, 6:53pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/4 "2024-07-09T18:53:17Z")

</div>

Hi,

I did include `only_active=True` in the `get_query` call, and it returned the expected 100 processes. The processes were excepted but I guess they weren’t sealed for some reason? Not sure why.

What I’ve done for now is remove those nodes from the destination group to free up the active slots. The initial group has ~27000 Str nodes and to date ~2300 have been run using the controller. So, yeah it actually does work! That may be because I might not have updated `aiida-submission-controller`. Looking at the old version of `add_in_batches`, before an update 2 months ago, I see:

```python
        code = orm.load_code(self.code_label)
        inputs = {
            "code": code,
            "x": orm.Int(extras_values[0]),
            "y": orm.Int(extras_values[1]),
        }
        return inputs, CalculationFactory(code.get_input_plugin_name())

```

Which has the same order as mine. I drew pretty heavily from the examples almost a year ago when I was originally writing these, so that makes sense. I guess I’ll update my installation of the `aiida-submission-controller` and update the code as well to match the newer version. I don’t think that’s related to the issue I’m having here.

Just seems like the issue here is that my processes weren’t sealed despite being excepted for some reason. Probably unrelated to `aiida-submission-controller` itself I imagine. Anyone have any idea why that might be?

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 9, 2024, 7:31pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/5 "2024-07-09T19:31:08Z")

</div>

Quick note that my `aiida-submission-controller` is the same as the latest PyPi release - the changes with the updated `submit_new_batch` which gets a `builder` from `get_inputs_and_processclass_from_extras` hasn’t been released there yet, so that’s why I’m not getting an error since I installed using `pip`.

I can still install from the repository and update my stuff anyways, but just making a note of that.

---

<div class="post-metadata">

**Author:** ![t-reents](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/t-reents/32/82_2.png) [@t-reents](https://aiida.discourse.group/u/t-reents)\
**Post date:** [July 9, 2024, 8:25pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/6 "2024-07-09T20:25:32Z")

</div>

Hi @kmlefran

Yes, you are right. I forgot about that update and it makes totally sense that your approach is working. I was also not expecting that this is related to the issue you encountered, but only a general comment out of curiosity.

It’s indeed strange that these processes are excepted but not sealed. You can run `node.is_sealed` (where `node` is your excepted `ProcessNode`) to confirm that they were indeed not sealed. However, it seems that this has to be the reason based on your observations and having in mind that your implementation already worked for a large number of submissions.

Pinging @mbercx and @sphuber, am I missing something about excepted processes not being sealed or a known problem concerning this issue? (at least I didn’t find any previous issue/PR on aiida-core)

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [July 10, 2024, 6:40am UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/7 "2024-07-10T06:40:46Z")

</div>

An excepted node _should_ be sealed. So @kmlefran if you do in fact have excepted processes that are not sealed, that should be considered a bug. It would be great if you could come up with a reproducible example so I could have a look at how to fix it.

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 10, 2024, 3:51pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/8 "2024-07-10T15:51:48Z")

</div>

Yeah, a query is showing that this is the case for me:

```python
qb = QueryBuilder()
qb.append(Node,
    filters = {
        "or": [
            {"attributes.sealed": False},
            {"attributes": {"!has_key": "sealed"}},
        ],
        'attributes.process_state': 'excepted',
        })
qb.all()

```

Returns some nodes. I don’t know if I have anything to reproduce this reliably yet.  
The error in some of these processes that caused the exception was that the daemon was using the wrong Python environment and tried to use a function that wasn’t in that function

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [July 10, 2024, 4:51pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/9 "2024-07-10T16:51:13Z")

</div>

Ok, that would indeed be an anamolous situation. For the time being, you can iterate over those inconsistent nodes and call `node.seal()` to seal them manually. I think that should work. At least that way they won’t interfere with your submission controller

---

<div class="post-metadata">

**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\
**Post date:** [July 10, 2024, 5:10pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/10 "2024-07-10T17:10:16Z")

</div>

Also, if you happen to still have or find in the logs the traceback, it might help seeing where the exception was raised and why the sealing was not triggered

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 10, 2024, 7:53pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/11 "2024-07-10T19:53:19Z")

</div>

Here’s the process report

````auto
2024-06-25 17:05:58 [6930 | REPORT]: [80542|SmilesToGaussianWorkchain|on_except]: Traceback (most recent call last):
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/plumpy/processes.py", line 185, in spec
    return cls. __getattribute__ (cls, '_spec')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Protect' object has no attribute '_spec'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/plumpy/base/state_machine.py", line 324, in transition_to
    self._enter_next_state(new_state)
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/plumpy/base/state_machine.py", line 388, in _enter_next_state
    self._fire_state_event(StateEventHook.ENTERED_STATE, last_state)
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/plumpy/base/state_machine.py", line 300, in _fire_state_event
    callback(self, hook, state)
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/plumpy/processes.py", line 334, in <lambda>
    lambda _s, _h, from_state: self.on_entered(cast(Optional[process_states.State], from_state)),
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/aiida/engine/processes/process.py", line 428, in on_entered
    self.update_outputs()
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/aiida/engine/processes/process.py", line 665, in update_outputs
    outputs_flat = self._flat_outputs()
                   ^^^^^^^^^^^^^^^^^^^^
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/aiida/engine/processes/process.py", line 851, in _flat_outputs
    return dict(self._flatten_outputs(self.spec().outputs, self.outputs))
                                      ^^^^^^^^^^^
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/aiida/engine/processes/workchains/workchain.py", line 135, in spec
    return super().spec() # type: ignore[return-value]
           ^^^^^^^^^^^^^^
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/aiida/engine/processes/process.py", line 88, in spec
    return super().spec() # type: ignore[return-value]
           ^^^^^^^^^^^^^^
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/plumpy/processes.py", line 190, in spec
    cls.define(cls._spec) # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chemlab/anaconda3/lib/python3.11/site-packages/aiida_aimall/workchains.py", line 440, in define
    cls.update_paramaters_with_cm,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'SmilesToGaussianWorkchain' has no attribute 'update_paramaters_with_cm'```
````

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [July 24, 2024, 2:37pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/12 "2024-07-24T14:37:59Z")

</div>

Hi, tangentially related to the issue in this thread, but in talking with a coworker, I found that he also had an unsealed excepted WorkChain. I was able to reproduce this and have raised an issue on the `aiida-core` GitHub

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\
**Post date:** [July 24, 2026, 2:38pm UTC](https://aiida.discourse.group/t/how-to-reset-fix-fromgroupsubmissioncontroller/426/13 "2026-07-24T14:38:23Z")

</div>

This topic was automatically closed 730 days after the last reply. New replies are no longer allowed.