Using thread to watch workchain's status in AiiDAlab

AiiDAlab
1

Hello,

I have an AiiDAlab interface with several QE workchains running. I need to check regularly if they are all finished to advance to the next step automatically. To do that I use threading with my target function as follows

def _check_workchain_finished(self, workchains):
    checking_interval = 3
    while True:
        time.sleep(checking_interval)
        done = True
        for workchain in workchains:
            if not workchain.is_finished:
                done = False
        if done:
            self.state = self.State.SUCCESS
            self.validation_nodes = workchains
            break

where workchain is the list of runnings QE workchains that I have submitted.

I create a thread as

    thread = threading.Thread(target=self._check_workchain_finished,
                              args=[nodes])
    thread.start()

But there is exception raised at if not workchain.is_finished

Exception in thread Thread-12 (_check_workchain_finished):
Traceback (most recent call last):
  File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hungdt/works/IMMAD/codes/IMMAD/substitution.py", line 326, in _check_workchain_finished
    if not workchain.is_finished:
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/aiida/orm/nodes/process/process.py", line 375, in is_finished
    return self.process_state == ProcessState.FINISHED
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/aiida/orm/nodes/process/process.py", line 279, in process_state
    state = self.base.attributes.get(self.PROCESS_STATE_KEY, None)
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/aiida/orm/nodes/attributes.py", line 77, in get
    attribute = self._backend_node.get_attribute(key)
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/aiida/storage/psql_dos/orm/nodes.py", line 241, in get_attribute
    return self.model.attributes[key]
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/aiida/storage/psql_dos/orm/utils.py", line 85, in __getattr__
    self._ensure_model_uptodate(fields=(item,))
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/aiida/storage/psql_dos/orm/utils.py", line 162, in _ensure_model_uptodate
    self.session.expire(self._model, attribute_names=fields)
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 2431, in expire
    self._expire_state(state, attribute_names)
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 2434, in _expire_state
    self._validate_persistent(state)
  File "/home/hungdt/venv/aiida/lib64/python3.10/site-packages/sqlalchemy/orm/session.py", line 3161, in _validate_persistent
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Instance '<DbNode at 0x7f106e7804c0>' is not persistent within this Session

Do you know what is the problem here and how to solve it? Thank you.

1 Like
2

Hi @hungdt :wave:

The problem is that the DB connections in AiiDA are not thread safe, so you cannot pass Node ORM objects (in your case the workchains list) into a different thread.

What we do in AiiDAlab to get around this is to pass the Node UUIDs, and then use load_node inside the thread.

LMK if that makes sense.

3

Hi @hungdt,

As @danielhollas mentioned aiida is not thead-safe, so using threading with AiiDA might not be a good idea.

Regarding the use case you have “I need to check regularly if they are all finished to advance to the next step automatically.”, maybe you can try to create a large workchain to submit workchains, using self.to_context and append_ workchain node to the context for the next step.
Please find the example in append_ usage section of how to run calculations in parallel and post-process the result after all are finished. The AiiDA daemon can handle the wait for you, under the hood it uses async and there is no need to write multithreading code to do it by yourself.

I hope this helps,
Jason

1 Like
4

Besides the excellent replies from @danielhollas and @jusong.yu I have another tip. Instead of using workchain.is_finished you may want to consider using workchain.is_terminated. The latter also includes when a workchain ends up in the state KILLED or EXCEPTED. This part of the documentation will be useful: Concepts — AiiDA 2.4.0.post0 documentation

2 Likes
5

Thank you all very much. I have worked it out using thread just because of slightly changes in my code but @jusong.yu 's solution seems to be better that I should consider later.

@sphuber : thanks for the suggestion.