# How to debug a workflow stuck in "Waiting" state **URL:** \ **Category:** General Usage\ **Tags:** question\ **Created:** [January 17, 2024, 2:32pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220 "2024-01-17T14:32:24Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 2:32pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/1 "2024-01-17T14:32:24Z")
Hi all, I am trying to debug an issue where the top-level WorkChain is stuck in the “Waiting” Process State, even though all the sub-WorkChains have finished successfully (Exit Code 0). ```console $ verdi process list PK Created Process label Process State Process status ------ --------- ----------------- --------------- ---------------- 117419 34D ago ATMOSPEC workflow ⏵ Waiting Total results: 1 Report: last time an entry changed state: 1D ago (at 04:53:21 on 2024-01-16) Report: Checking daemon load... OK Report: Using 0% of the available daemon worker slots. ``` I’ve tried all the usual stuff - restarting the daemon and RabbitMQ, running `verdi devel rabbitmq tasks analyze --fix` with no luck. What else can I do? The workflow itself was launched with aiida-core 2.2. I’ve since upgraded to aiida-core 2.4 since I was seeing some other issue that was fixed in 2.4. The only other thing is that when this WorkChain was launched, the daemon workers were very busy, so I tried to pause it via `verdi process pause 117419`. But at that time I got an error that the process was unreachable. So I am not sure if this process was succesfully paused. But since then I’ve issued the `verdi process play` many times with no effect. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 3:08pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/2 "2024-01-17T15:08:33Z")
Another observation: When I try to pause to process via `verdi process pause`, I sometimes get: ```console Error: failed to pause Process<117419>: deep copying a base Node is not supported ``` Other times it says `Report: request to pause Process<117419> sent` ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 17, 2024, 3:11pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/3 "2024-01-17T15:11:58Z")
Could you run the following in a shell and report the output ```python load_node(117419).checkpoint ``` > Another observation: When I try to pause to process via `verdi process pause`, I sometimes get: ```auto Error: failed to pause Process<117419>: deep copying a base Node is not supported ``` That I have never seen before, very weird. Don’t have an idea what could be causing that. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 3:35pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/4 "2024-01-17T15:35:59Z")
> [@sphuber](#): > > `load_node(117419).checkpoint` The resulting output has 12k lines (there are 1000 Caljobs within this Workchain). I am attaching as a file. [checkpoint.txt](https://aiida.discourse.group/uploads/short-url/8SPVzlVTcVMHzyYPt2SKKS4h987.txt) (319.1 KB) (side note: I noticed that we were not allowing file uploads so I changed to settings to allow text/document and image files to be uploaded. CC @mbercx) ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 3:45pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/5 "2024-01-17T15:45:07Z")
> [@sphuber](#): > > That I have never seen before, very weird. Don’t have an idea what could be causing that. Back when I was using v2.2 I was getting the full stack trace. I’ll try to get it again so we can further debug this. But it does not seem to be the immediate cause of this issue I guess. For the record, the error message comes from here: > ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 17, 2024, 4:09pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/6 "2024-01-17T16:09:35Z")
> [@danielhollas](#): > > (side note: I noticed that we were not allowing file uploads so I changed to settings to allow text/document and image files to be uploaded. CC @mbercx) We disabled this because we have a 5 GB storage limit. Is there now at least a size limit? If not, I would revert the change to be honest. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 17, 2024, 4:14pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/7 "2024-01-17T16:14:24Z")
> [@danielhollas](#): > > The workflow itself was launched with aiida-core 2.2. I’ve since upgraded to aiida-core 2.4 since I was seeing some other issue that was fixed in 2.4. Actually, I think this may very well be the problem. We do not guarantee that processes that have been started with a specific version of AiiDA, will continue to work when AiiDA is upgraded. I now actually remember having addressed a bug in the last year that affected the `CalcJob` engine. The necessary fix would also change the expected layout of the process checkpoint, and so old checkpoints may no longer be compatible. I will try to remember what the change was and to see if I can remember what the change was, but I am not even sure if we can “reverse” engineer the correct checkpoint file. Do you have caching enabled for these subprocesses? In that case, your best bet would be to simply rerun the workflow I’m afraid. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 17, 2024, 4:22pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/8 "2024-01-17T16:22:36Z")
By the way, did you check the daemon logs for an exception? I expect there should be an exception when the daemon worker that picks up the workflow tries to reconstruct the `Process` instance from the checkpoint and continues it. Presumable, the checkpoint is corrupt due to the upgrade of `aiida-core` and the daemon worker fails to continue running it. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 4:28pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/9 "2024-01-17T16:28:44Z")
Hmm, but I had the issue already in the 2.2 version, i.e. before the upgrade. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 4:32pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/10 "2024-01-17T16:32:06Z")
> [@sphuber](#): > > By the way, did you check the daemon logs for an exception? I remember checking and did not see it. What’s the best way to double check? Currently the `verdi daemon logshow` doesn’t show any exceptions. Presumable, if I restart the daemon it should try to pick up the checkpoint right? ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 17, 2024, 4:34pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/11 "2024-01-17T16:34:33Z")
> [@danielhollas](#): > > I remember checking and did not see it. What’s the best way to double check? Currently the `verdi daemon logshow` doesn’t show any exceptions. Presumable, if I restart the daemon it should try to pick up the checkpoint right? Yes, if you hard restart the daemon `verdi daemon restart --reset` it should pick it up again and try to recreate the process and continue it. `verdi daemon logshow` essentially just does `tail -f` so this just shows the latest. You would want to look directly at the file. It should be in `.aiida/daemon/log/aiida-{profile}.log` ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 4:41pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/12 "2024-01-17T16:41:32Z")
> [@sphuber](#): > > Yes, if you hard restart the daemon `verdi daemon restart --reset` Hmm, after doing this I see nothing in the logs (at `info` verbosity level). > [@sphuber](#): > > Do you have caching enabled for these subprocesses? In that case, your best bet would be to simply rerun the workflow I’m afraid. I don’t unfortunately (this is all happening inside the AiiDAlab Docker image btw). If we convince ourselves that the checkpoint is indeed the problem, then I am happy to just relaunch. But right now I don’t think it is since all the symptoms were already present for the v2.2 aiida-core version. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 4:46pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/13 "2024-01-17T16:46:21Z")
Looking at the older logs, I see a bunch of these: ```auto 12/17/2023 03:16:05 AM <964422> kiwipy.rmq.tasks: [ERROR] Exception occurred while processing task. ``` But I don’t know if they correspond to the WorkChain in question. This was about the time where I was dealing with [How to kill a monster workflow 👹](https://aiida.discourse.group/t/how-to-kill-a-monster-workflow/206) ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 5:35pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/14 "2024-01-17T17:35:39Z")
Okay, this log entry is actually suspicious (I don’t see other entries for this PK) > Error: A subscriber with the process id\<117419\> already exists, which most likely means this worker is already working on it and this task was sent as a duplicate by mistake. Deleting the task now. Should I try to run ```console verdi devel rabbitmq tasks revive ``` ? ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 18, 2024, 7:26am UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/15 "2024-01-18T07:26:42Z")
But is that a recent message? That error means the tasks was sent out again, erroneously, and this check is preventing that a second daemon worker starts working on the same process, which would be very bad, most likely leading to it excepting. Reviving the task shouldn’t help in this case, because apparently there already was an existing task. I also wouldn’t ever use the `revive` command, because that has no security checks built-in. If you have to, stop the daemon and run `verdi devel rabbitmq tasks analyze --fix`. Or if you are on v2.5 you can run `verdi process repair`, that is the same command but easier to remember. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 18, 2024, 7:30am UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/16 "2024-01-18T07:30:17Z")
As for how to debug further. We would have to check what happens when the task gets picked up. Is that workchain 117419 the only one marked as active by `verdi process list`? Please stop the daemon and then run `verdi devel rabbitmq tasks analyze`. What is the output? It should say that there is one active process and it has a corresponding task. If that is true, then keep the daemon stopped and increase the log level: ```auto verdi config set logging.aiida_loglevel INFO ``` and start a single daemon worker in the foreground: ```auto verdi daemon worker ``` This should now “block” the shell as it is running a daemon runner. You should now see log messages that it receives the task of the active process and tries to continue the process. Please report the output. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 18, 2024, 10:16am UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/17 "2024-01-18T10:16:15Z")
> [@sphuber](#): > > Is that workchain 117419 the only one marked as active by `verdi process list`? Yep! ```console $ verdi process list PK Created Process label Process State Process status ------ --------- ----------------- --------------- ---------------- 117419 34D ago ATMOSPEC workflow ⏵ Waiting Total results: 1 ``` > [@sphuber](#): > > Please stop the daemon and then run `verdi devel rabbitmq tasks analyze`. What is the output? It should say that there is one active process and it has a corresponding task. ```console $ verdi daemon stop Profile: default Stopping the daemon... OK $ verdi devel rabbitmq tasks analyze -v info ... ... ... Info: RELEASE SAVEPOINT sa_savepoint_1 Info: [no key 0.00040s] {} Info: Active processes: [117419] Info: Process tasks: [117419] Success: No inconsistencies detected between database and RabbitMQ. $ verdi devel rabbitmq tasks list 117419 ``` > [@sphuber](#): > > and start a single daemon worker in the foreground: ```console $ verdi config set logging.aiida_loglevel INFO Success: 'logging.aiida_loglevel' set to INFO for 'default' profile $ verdi daemon worker / ``` The worker does not show any output. Perhaps the logging level config does not apply to it? When I run it with increased verbosity explicitly I get ```auto $ verdi daemon worker -v INFO ... ... ... Info: DbLink.output relationship direction symbol('MANYTOONE') Info: DbLink.output lazy loading clause :param_1 = db_dbnode.id Info: DbLink.output will use Session.get() to optimize instance loads Info: (DbLink|db_dblink) initialize prop id Info: (DbLink|db_dblink) initialize prop input_id Info: (DbLink|db_dblink) initialize prop output_id Info: (DbLink|db_dblink) initialize prop label Info: (DbLink|db_dblink) initialize prop type Info: (DbLink|db_dblink) _post_configure_properties() complete Info: SELECT db_dbsetting.val FROM db_dbsetting WHERE db_dbsetting.key = %(key_1)s Info: [generated in 0.00014s] {'key_1': 'repository|uuid'} Info: ROLLBACK Info: Pool disposed. Pool size: 5 Connections in pool: 0 Current Overflow: -5 Current Checked out connections: 0 Info: Pool recreating ``` > **Full output** > > ```auto > -Info: (Obj|db_object) _configure_property(id, Column) > Info: (Obj|db_object) _configure_property(hashkey, Column) > Info: (Obj|db_object) _configure_property(compressed, Column) > Info: (Obj|db_object) _configure_property(size, Column) > Info: (Obj|db_object) _configure_property(offset, Column) > Info: (Obj|db_object) _configure_property(length, Column) > Info: (Obj|db_object) _configure_property(pack_id, Column) > Info: (Obj|db_object) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (Obj|db_object) constructed > -Info: (DbSetting|db_dbsetting) _configure_property(id, Column) > Info: (DbSetting|db_dbsetting) _configure_property(key, Column) > Info: (DbSetting|db_dbsetting) _configure_property(val, Column) > Info: (DbSetting|db_dbsetting) _configure_property(description, Column) > Info: (DbSetting|db_dbsetting) _configure_property(time, Column) > Info: (DbSetting|db_dbsetting) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbSetting|db_dbsetting) constructed > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(_metadata, Column) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(aiidauser, RelationshipProperty) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(dbcomputer, RelationshipProperty) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(id, Column) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(aiidauser_id, Column) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(dbcomputer_id, Column) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(auth_params, Column) > Info: (DbAuthInfo|db_dbauthinfo) _configure_property(enabled, Column) > Info: (DbAuthInfo|db_dbauthinfo) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbAuthInfo|db_dbauthinfo) constructed > Info: (DbComputer|db_dbcomputer) _configure_property(_metadata, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(id, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(uuid, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(label, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(hostname, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(description, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(scheduler_type, Column) > Info: (DbComputer|db_dbcomputer) _configure_property(transport_type, Column) > Info: (DbComputer|db_dbcomputer) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbComputer|db_dbcomputer) constructed > Info: (DbUser|db_dbuser) _configure_property(id, Column) > Info: (DbUser|db_dbuser) _configure_property(email, Column) > Info: (DbUser|db_dbuser) _configure_property(first_name, Column) > Info: (DbUser|db_dbuser) _configure_property(last_name, Column) > Info: (DbUser|db_dbuser) _configure_property(institution, Column) > Info: (DbUser|db_dbuser) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbUser|db_dbuser) constructed > Info: (DbComment|db_dbcomment) _configure_property(dbnode, RelationshipProperty) > Info: (DbComment|db_dbcomment) _configure_property(user, RelationshipProperty) > Info: (DbComment|db_dbcomment) _configure_property(id, Column) > Info: (DbComment|db_dbcomment) _configure_property(uuid, Column) > Info: (DbComment|db_dbcomment) _configure_property(dbnode_id, Column) > Info: (DbComment|db_dbcomment) _configure_property(ctime, Column) > Info: (DbComment|db_dbcomment) _configure_property(mtime, Column) > Info: (DbComment|db_dbcomment) _configure_property(user_id, Column) > Info: (DbComment|db_dbcomment) _configure_property(content, Column) > Info: (DbComment|db_dbcomment) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbComment|db_dbcomment) constructed > Info: (DbGroupNode|db_dbgroup_dbnodes) _configure_property(id, Column) > Info: (DbGroupNode|db_dbgroup_dbnodes) _configure_property(dbnode_id, Column) > Info: (DbGroupNode|db_dbgroup_dbnodes) _configure_property(dbgroup_id, Column) > Info: (DbGroupNode|db_dbgroup_dbnodes) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbGroupNode|db_dbgroup_dbnodes) constructed > Info: (DbGroup|db_dbgroup) _configure_property(user, RelationshipProperty) > Info: (DbGroup|db_dbgroup) _configure_property(dbnodes, RelationshipProperty) > Info: (DbGroup|db_dbgroup) _configure_property(id, Column) > Info: (DbGroup|db_dbgroup) _configure_property(uuid, Column) > Info: (DbGroup|db_dbgroup) _configure_property(label, Column) > Info: (DbGroup|db_dbgroup) _configure_property(type_string, Column) > Info: (DbGroup|db_dbgroup) _configure_property(time, Column) > Info: (DbGroup|db_dbgroup) _configure_property(description, Column) > Info: (DbGroup|db_dbgroup) _configure_property(extras, Column) > Info: (DbGroup|db_dbgroup) _configure_property(user_id, Column) > Info: (DbGroup|db_dbgroup) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbGroup|db_dbgroup) constructed > Info: (DbLog|db_dblog) _configure_property(_metadata, Column) > Info: (DbLog|db_dblog) _configure_property(dbnode, RelationshipProperty) > Info: (DbLog|db_dblog) _configure_property(id, Column) > Info: (DbLog|db_dblog) _configure_property(uuid, Column) > Info: (DbLog|db_dblog) _configure_property(time, Column) > Info: (DbLog|db_dblog) _configure_property(loggername, Column) > Info: (DbLog|db_dblog) _configure_property(levelname, Column) > Info: (DbLog|db_dblog) _configure_property(dbnode_id, Column) > Info: (DbLog|db_dblog) _configure_property(message, Column) > Info: (DbLog|db_dblog) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbLog|db_dblog) constructed > Info: (DbNode|db_dbnode) _configure_property(dbcomputer, RelationshipProperty) > Info: (DbNode|db_dbnode) _configure_property(user, RelationshipProperty) > Info: (DbNode|db_dbnode) _configure_property(outputs_q, RelationshipProperty) > Info: (DbNode|db_dbnode) _configure_property(id, Column) > Info: (DbNode|db_dbnode) _configure_property(uuid, Column) > Info: (DbNode|db_dbnode) _configure_property(node_type, Column) > Info: (DbNode|db_dbnode) _configure_property(process_type, Column) > Info: (DbNode|db_dbnode) _configure_property(label, Column) > Info: (DbNode|db_dbnode) _configure_property(description, Column) > Info: (DbNode|db_dbnode) _configure_property(ctime, Column) > Info: (DbNode|db_dbnode) _configure_property(mtime, Column) > Info: (DbNode|db_dbnode) _configure_property(attributes, Column) > Info: (DbNode|db_dbnode) _configure_property(extras, Column) > Info: (DbNode|db_dbnode) _configure_property(repository_metadata, Column) > Info: (DbNode|db_dbnode) _configure_property(dbcomputer_id, Column) > Info: (DbNode|db_dbnode) _configure_property(user_id, Column) > Info: (DbNode|db_dbnode) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbNode|db_dbnode) constructed > Info: (DbLink|db_dblink) _configure_property(input, RelationshipProperty) > Info: (DbLink|db_dblink) _configure_property(output, RelationshipProperty) > Info: (DbLink|db_dblink) _configure_property(id, Column) > Info: (DbLink|db_dblink) _configure_property(input_id, Column) > Info: (DbLink|db_dblink) _configure_property(output_id, Column) > Info: (DbLink|db_dblink) _configure_property(label, Column) > Info: (DbLink|db_dblink) _configure_property(type, Column) > Info: (DbLink|db_dblink) Identified primary key columns: ColumnSet([Column('id', Integer(), table=, primary_key=True, nullable=False)]) > Info: (DbLink|db_dblink) constructed > Info: select pg_catalog.version() > Info: [raw sql] {} > Info: select current_schema() > Info: [raw sql] {} > Info: show standard_conforming_strings > Info: [raw sql] {} > Info: BEGIN (implicit) > Info: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s > Info: [generated in 0.00024s] {'name': 'alembic_version'} > Info: Context impl PostgresqlImpl. > Info: Will assume transactional DDL. > Info: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s > Info: [cached since 0.0306s ago] {'name': 'alembic_version'} > Info: SELECT alembic_version.version_num > FROM alembic_version > Info: [generated in 0.00011s] {} > Info: (DbSetting|db_dbsetting) _post_configure_properties() started > Info: (DbSetting|db_dbsetting) initialize prop id > Info: (DbSetting|db_dbsetting) initialize prop key > Info: (DbSetting|db_dbsetting) initialize prop val > Info: (DbSetting|db_dbsetting) initialize prop description > Info: (DbSetting|db_dbsetting) initialize prop time > Info: (DbSetting|db_dbsetting) _post_configure_properties() complete > Info: (DbAuthInfo|db_dbauthinfo) _post_configure_properties() started > Info: (DbAuthInfo|db_dbauthinfo) initialize prop _metadata > Info: (DbAuthInfo|db_dbauthinfo) initialize prop aiidauser > Info: DbAuthInfo.aiidauser setup primary join db_dbuser.id = db_dbauthinfo.aiidauser_id > Info: DbAuthInfo.aiidauser setup secondary join None > Info: DbAuthInfo.aiidauser synchronize pairs [(db_dbuser.id => db_dbauthinfo.aiidauser_id)] > Info: DbAuthInfo.aiidauser secondary synchronize pairs [] > Info: DbAuthInfo.aiidauser local/remote pairs [(db_dbauthinfo.aiidauser_id / db_dbuser.id)] > Info: DbAuthInfo.aiidauser remote columns [db_dbuser.id] > Info: DbAuthInfo.aiidauser local columns [db_dbauthinfo.aiidauser_id] > Info: DbAuthInfo.aiidauser relationship direction symbol('MANYTOONE') > Info: (DbUser|db_dbuser) _configure_property(authinfos, RelationshipProperty) > Info: DbUser.authinfos setup primary join db_dbuser.id = db_dbauthinfo.aiidauser_id > Info: DbUser.authinfos setup secondary join None > Info: DbUser.authinfos synchronize pairs [(db_dbuser.id => db_dbauthinfo.aiidauser_id)] > Info: DbUser.authinfos secondary synchronize pairs [] > Info: DbUser.authinfos local/remote pairs [(db_dbuser.id / db_dbauthinfo.aiidauser_id)] > Info: DbUser.authinfos remote columns [db_dbauthinfo.aiidauser_id] > Info: DbUser.authinfos local columns [db_dbuser.id] > Info: DbUser.authinfos relationship direction symbol('ONETOMANY') > Info: DbUser.authinfos lazy loading clause :param_1 = db_dbauthinfo.aiidauser_id > Info: DbAuthInfo.aiidauser lazy loading clause db_dbuser.id = :param_1 > Info: DbAuthInfo.aiidauser will use Session.get() to optimize instance loads > Info: (DbAuthInfo|db_dbauthinfo) initialize prop dbcomputer > Info: DbAuthInfo.dbcomputer setup primary join db_dbcomputer.id = db_dbauthinfo.dbcomputer_id > Info: DbAuthInfo.dbcomputer setup secondary join None > Info: DbAuthInfo.dbcomputer synchronize pairs [(db_dbcomputer.id => db_dbauthinfo.dbcomputer_id)] > Info: DbAuthInfo.dbcomputer secondary synchronize pairs [] > Info: DbAuthInfo.dbcomputer local/remote pairs [(db_dbauthinfo.dbcomputer_id / db_dbcomputer.id)] > Info: DbAuthInfo.dbcomputer remote columns [db_dbcomputer.id] > Info: DbAuthInfo.dbcomputer local columns [db_dbauthinfo.dbcomputer_id] > Info: DbAuthInfo.dbcomputer relationship direction symbol('MANYTOONE') > Info: (DbComputer|db_dbcomputer) _configure_property(authinfos, RelationshipProperty) > Info: DbComputer.authinfos setup primary join db_dbcomputer.id = db_dbauthinfo.dbcomputer_id > Info: DbComputer.authinfos setup secondary join None > Info: DbComputer.authinfos synchronize pairs [(db_dbcomputer.id => db_dbauthinfo.dbcomputer_id)] > Info: DbComputer.authinfos secondary synchronize pairs [] > Info: DbComputer.authinfos local/remote pairs [(db_dbcomputer.id / db_dbauthinfo.dbcomputer_id)] > Info: DbComputer.authinfos remote columns [db_dbauthinfo.dbcomputer_id] > Info: DbComputer.authinfos local columns [db_dbcomputer.id] > Info: DbComputer.authinfos relationship direction symbol('ONETOMANY') > Info: DbComputer.authinfos lazy loading clause :param_1 = db_dbauthinfo.dbcomputer_id > Info: DbAuthInfo.dbcomputer lazy loading clause db_dbcomputer.id = :param_1 > Info: DbAuthInfo.dbcomputer will use Session.get() to optimize instance loads > Info: (DbAuthInfo|db_dbauthinfo) initialize prop id > Info: (DbAuthInfo|db_dbauthinfo) initialize prop aiidauser_id > Info: (DbAuthInfo|db_dbauthinfo) initialize prop dbcomputer_id > Info: (DbAuthInfo|db_dbauthinfo) initialize prop auth_params > Info: (DbAuthInfo|db_dbauthinfo) initialize prop enabled > Info: (DbAuthInfo|db_dbauthinfo) _post_configure_properties() complete > Info: (DbComputer|db_dbcomputer) _post_configure_properties() started > Info: (DbComputer|db_dbcomputer) initialize prop _metadata > Info: (DbComputer|db_dbcomputer) initialize prop id > Info: (DbComputer|db_dbcomputer) initialize prop uuid > Info: (DbComputer|db_dbcomputer) initialize prop label > Info: (DbComputer|db_dbcomputer) initialize prop hostname > Info: (DbComputer|db_dbcomputer) initialize prop description > Info: (DbComputer|db_dbcomputer) initialize prop scheduler_type > Info: (DbComputer|db_dbcomputer) initialize prop transport_type > Info: (DbComputer|db_dbcomputer) initialize prop authinfos > Info: (DbComputer|db_dbcomputer) _post_configure_properties() complete > Info: (DbUser|db_dbuser) _post_configure_properties() started > Info: (DbUser|db_dbuser) initialize prop id > Info: (DbUser|db_dbuser) initialize prop email > Info: (DbUser|db_dbuser) initialize prop first_name > Info: (DbUser|db_dbuser) initialize prop last_name > Info: (DbUser|db_dbuser) initialize prop institution > Info: (DbUser|db_dbuser) initialize prop authinfos > Info: (DbUser|db_dbuser) _post_configure_properties() complete > Info: (DbComment|db_dbcomment) _post_configure_properties() started > Info: (DbComment|db_dbcomment) initialize prop dbnode > Info: DbComment.dbnode setup primary join db_dbnode.id = db_dbcomment.dbnode_id > Info: DbComment.dbnode setup secondary join None > Info: DbComment.dbnode synchronize pairs [(db_dbnode.id => db_dbcomment.dbnode_id)] > Info: DbComment.dbnode secondary synchronize pairs [] > Info: DbComment.dbnode local/remote pairs [(db_dbcomment.dbnode_id / db_dbnode.id)] > Info: DbComment.dbnode remote columns [db_dbnode.id] > Info: DbComment.dbnode local columns [db_dbcomment.dbnode_id] > Info: DbComment.dbnode relationship direction symbol('MANYTOONE') > Info: (DbNode|db_dbnode) _configure_property(dbcomments, RelationshipProperty) > Info: DbNode.dbcomments setup primary join db_dbnode.id = db_dbcomment.dbnode_id > Info: DbNode.dbcomments setup secondary join None > Info: DbNode.dbcomments synchronize pairs [(db_dbnode.id => db_dbcomment.dbnode_id)] > Info: DbNode.dbcomments secondary synchronize pairs [] > Info: DbNode.dbcomments local/remote pairs [(db_dbnode.id / db_dbcomment.dbnode_id)] > Info: DbNode.dbcomments remote columns [db_dbcomment.dbnode_id] > Info: DbNode.dbcomments local columns [db_dbnode.id] > Info: DbNode.dbcomments relationship direction symbol('ONETOMANY') > Info: DbNode.dbcomments lazy loading clause :param_1 = db_dbcomment.dbnode_id > Info: DbComment.dbnode lazy loading clause db_dbnode.id = :param_1 > Info: DbComment.dbnode will use Session.get() to optimize instance loads > Info: (DbComment|db_dbcomment) initialize prop user > Info: DbComment.user setup primary join db_dbuser.id = db_dbcomment.user_id > Info: DbComment.user setup secondary join None > Info: DbComment.user synchronize pairs [(db_dbuser.id => db_dbcomment.user_id)] > Info: DbComment.user secondary synchronize pairs [] > Info: DbComment.user local/remote pairs [(db_dbcomment.user_id / db_dbuser.id)] > Info: DbComment.user remote columns [db_dbuser.id] > Info: DbComment.user local columns [db_dbcomment.user_id] > Info: DbComment.user relationship direction symbol('MANYTOONE') > Info: DbComment.user lazy loading clause db_dbuser.id = :param_1 > Info: DbComment.user will use Session.get() to optimize instance loads > Info: (DbComment|db_dbcomment) initialize prop id > Info: (DbComment|db_dbcomment) initialize prop uuid > Info: (DbComment|db_dbcomment) initialize prop dbnode_id > Info: (DbComment|db_dbcomment) initialize prop ctime > Info: (DbComment|db_dbcomment) initialize prop mtime > Info: (DbComment|db_dbcomment) initialize prop user_id > Info: (DbComment|db_dbcomment) initialize prop content > Info: (DbComment|db_dbcomment) _post_configure_properties() complete > Info: (DbGroupNode|db_dbgroup_dbnodes) _post_configure_properties() started > Info: (DbGroupNode|db_dbgroup_dbnodes) initialize prop id > Info: (DbGroupNode|db_dbgroup_dbnodes) initialize prop dbnode_id > Info: (DbGroupNode|db_dbgroup_dbnodes) initialize prop dbgroup_id > Info: (DbGroupNode|db_dbgroup_dbnodes) _post_configure_properties() complete > Info: (DbGroup|db_dbgroup) _post_configure_properties() started > Info: (DbGroup|db_dbgroup) initialize prop user > Info: DbGroup.user setup primary join db_dbuser.id = db_dbgroup.user_id > Info: DbGroup.user setup secondary join None > Info: DbGroup.user synchronize pairs [(db_dbuser.id => db_dbgroup.user_id)] > Info: DbGroup.user secondary synchronize pairs [] > Info: DbGroup.user local/remote pairs [(db_dbgroup.user_id / db_dbuser.id)] > Info: DbGroup.user remote columns [db_dbuser.id] > Info: DbGroup.user local columns [db_dbgroup.user_id] > Info: DbGroup.user relationship direction symbol('MANYTOONE') > Info: (DbUser|db_dbuser) _configure_property(dbgroups, RelationshipProperty) > Info: DbUser.dbgroups setup primary join db_dbuser.id = db_dbgroup.user_id > Info: DbUser.dbgroups setup secondary join None > Info: DbUser.dbgroups synchronize pairs [(db_dbuser.id => db_dbgroup.user_id)] > Info: DbUser.dbgroups secondary synchronize pairs [] > Info: DbUser.dbgroups local/remote pairs [(db_dbuser.id / db_dbgroup.user_id)] > Info: DbUser.dbgroups remote columns [db_dbgroup.user_id] > Info: DbUser.dbgroups local columns [db_dbuser.id] > Info: DbUser.dbgroups relationship direction symbol('ONETOMANY') > Info: DbUser.dbgroups lazy loading clause :param_1 = db_dbgroup.user_id > Info: DbGroup.user lazy loading clause db_dbuser.id = :param_1 > Info: DbGroup.user will use Session.get() to optimize instance loads > Info: (DbGroup|db_dbgroup) initialize prop dbnodes > Info: DbGroup.dbnodes setup primary join db_dbgroup.id = db_dbgroup_dbnodes.dbgroup_id > Info: DbGroup.dbnodes setup secondary join db_dbnode.id = db_dbgroup_dbnodes.dbnode_id > Info: DbGroup.dbnodes synchronize pairs [(db_dbgroup.id => db_dbgroup_dbnodes.dbgroup_id)] > Info: DbGroup.dbnodes secondary synchronize pairs [(db_dbnode.id => db_dbgroup_dbnodes.dbnode_id)] > Info: DbGroup.dbnodes local/remote pairs [(db_dbgroup.id / db_dbgroup_dbnodes.dbgroup_id),(db_dbnode.id / db_dbgroup_dbnodes.dbnode_id)] > Info: DbGroup.dbnodes remote columns [db_dbgroup_dbnodes.dbgroup_id,db_dbgroup_dbnodes.dbnode_id] > Info: DbGroup.dbnodes local columns [db_dbgroup.id] > Info: DbGroup.dbnodes relationship direction symbol('MANYTOMANY') > Info: (DbNode|db_dbnode) _configure_property(dbgroups, RelationshipProperty) > Info: DbNode.dbgroups setup primary join db_dbnode.id = db_dbgroup_dbnodes.dbnode_id > Info: DbNode.dbgroups setup secondary join db_dbgroup.id = db_dbgroup_dbnodes.dbgroup_id > Info: DbNode.dbgroups synchronize pairs [(db_dbnode.id => db_dbgroup_dbnodes.dbnode_id)] > Info: DbNode.dbgroups secondary synchronize pairs [(db_dbgroup.id => db_dbgroup_dbnodes.dbgroup_id)] > Info: DbNode.dbgroups local/remote pairs [(db_dbnode.id / db_dbgroup_dbnodes.dbnode_id),(db_dbgroup.id / db_dbgroup_dbnodes.dbgroup_id)] > Info: DbNode.dbgroups remote columns [db_dbgroup_dbnodes.dbgroup_id,db_dbgroup_dbnodes.dbnode_id] > Info: DbNode.dbgroups local columns [db_dbnode.id] > Info: DbNode.dbgroups relationship direction symbol('MANYTOMANY') > Info: DbNode.dbgroups lazy loading clause :param_1 = db_dbgroup_dbnodes.dbnode_id AND db_dbgroup.id = db_dbgroup_dbnodes.dbgroup_id > Info: DbGroup.dbnodes lazy loading clause :param_1 = db_dbgroup_dbnodes.dbgroup_id AND db_dbnode.id = db_dbgroup_dbnodes.dbnode_id > Info: (DbGroup|db_dbgroup) initialize prop id > Info: (DbGroup|db_dbgroup) initialize prop uuid > Info: (DbGroup|db_dbgroup) initialize prop label > Info: (DbGroup|db_dbgroup) initialize prop type_string > Info: (DbGroup|db_dbgroup) initialize prop time > Info: (DbGroup|db_dbgroup) initialize prop description > Info: (DbGroup|db_dbgroup) initialize prop extras > Info: (DbGroup|db_dbgroup) initialize prop user_id > Info: (DbGroup|db_dbgroup) _post_configure_properties() complete > Info: (DbLog|db_dblog) _post_configure_properties() started > Info: (DbLog|db_dblog) initialize prop _metadata > Info: (DbLog|db_dblog) initialize prop dbnode > Info: DbLog.dbnode setup primary join db_dbnode.id = db_dblog.dbnode_id > Info: DbLog.dbnode setup secondary join None > Info: DbLog.dbnode synchronize pairs [(db_dbnode.id => db_dblog.dbnode_id)] > Info: DbLog.dbnode secondary synchronize pairs [] > Info: DbLog.dbnode local/remote pairs [(db_dblog.dbnode_id / db_dbnode.id)] > Info: DbLog.dbnode remote columns [db_dbnode.id] > Info: DbLog.dbnode local columns [db_dblog.dbnode_id] > Info: DbLog.dbnode relationship direction symbol('MANYTOONE') > Info: (DbNode|db_dbnode) _configure_property(dblogs, RelationshipProperty) > Info: DbNode.dblogs setup primary join db_dbnode.id = db_dblog.dbnode_id > Info: DbNode.dblogs setup secondary join None > Info: DbNode.dblogs synchronize pairs [(db_dbnode.id => db_dblog.dbnode_id)] > Info: DbNode.dblogs secondary synchronize pairs [] > Info: DbNode.dblogs local/remote pairs [(db_dbnode.id / db_dblog.dbnode_id)] > Info: DbNode.dblogs remote columns [db_dblog.dbnode_id] > Info: DbNode.dblogs local columns [db_dbnode.id] > Info: DbNode.dblogs relationship direction symbol('ONETOMANY') > Info: DbNode.dblogs lazy loading clause :param_1 = db_dblog.dbnode_id > Info: DbLog.dbnode lazy loading clause db_dbnode.id = :param_1 > Info: DbLog.dbnode will use Session.get() to optimize instance loads > Info: (DbLog|db_dblog) initialize prop id > Info: (DbLog|db_dblog) initialize prop uuid > Info: (DbLog|db_dblog) initialize prop time > Info: (DbLog|db_dblog) initialize prop loggername > Info: (DbLog|db_dblog) initialize prop levelname > Info: (DbLog|db_dblog) initialize prop dbnode_id > Info: (DbLog|db_dblog) initialize prop message > Info: (DbLog|db_dblog) _post_configure_properties() complete > Info: (DbNode|db_dbnode) _post_configure_properties() started > Info: (DbNode|db_dbnode) initialize prop dbcomputer > Info: DbNode.dbcomputer setup primary join db_dbcomputer.id = db_dbnode.dbcomputer_id > Info: DbNode.dbcomputer setup secondary join None > Info: DbNode.dbcomputer synchronize pairs [(db_dbcomputer.id => db_dbnode.dbcomputer_id)] > Info: DbNode.dbcomputer secondary synchronize pairs [] > Info: DbNode.dbcomputer local/remote pairs [(db_dbnode.dbcomputer_id / db_dbcomputer.id)] > Info: DbNode.dbcomputer remote columns [db_dbcomputer.id] > Info: DbNode.dbcomputer local columns [db_dbnode.dbcomputer_id] > Info: DbNode.dbcomputer relationship direction symbol('MANYTOONE') > Info: (DbComputer|db_dbcomputer) _configure_property(dbnodes, RelationshipProperty) > Info: DbComputer.dbnodes setup primary join db_dbcomputer.id = db_dbnode.dbcomputer_id > Info: DbComputer.dbnodes setup secondary join None > Info: DbComputer.dbnodes synchronize pairs [(db_dbcomputer.id => db_dbnode.dbcomputer_id)] > Info: DbComputer.dbnodes secondary synchronize pairs [] > Info: DbComputer.dbnodes local/remote pairs [(db_dbcomputer.id / db_dbnode.dbcomputer_id)] > Info: DbComputer.dbnodes remote columns [db_dbnode.dbcomputer_id] > Info: DbComputer.dbnodes local columns [db_dbcomputer.id] > Info: DbComputer.dbnodes relationship direction symbol('ONETOMANY') > Info: DbComputer.dbnodes lazy loading clause :param_1 = db_dbnode.dbcomputer_id > Info: DbNode.dbcomputer lazy loading clause db_dbcomputer.id = :param_1 > Info: DbNode.dbcomputer will use Session.get() to optimize instance loads > Info: (DbNode|db_dbnode) initialize prop user > Info: DbNode.user setup primary join db_dbuser.id = db_dbnode.user_id > Info: DbNode.user setup secondary join None > Info: DbNode.user synchronize pairs [(db_dbuser.id => db_dbnode.user_id)] > Info: DbNode.user secondary synchronize pairs [] > Info: DbNode.user local/remote pairs [(db_dbnode.user_id / db_dbuser.id)] > Info: DbNode.user remote columns [db_dbuser.id] > Info: DbNode.user local columns [db_dbnode.user_id] > Info: DbNode.user relationship direction symbol('MANYTOONE') > Info: (DbUser|db_dbuser) _configure_property(dbnodes, RelationshipProperty) > Info: DbUser.dbnodes setup primary join db_dbuser.id = db_dbnode.user_id > Info: DbUser.dbnodes setup secondary join None > Info: DbUser.dbnodes synchronize pairs [(db_dbuser.id => db_dbnode.user_id)] > Info: DbUser.dbnodes secondary synchronize pairs [] > Info: DbUser.dbnodes local/remote pairs [(db_dbuser.id / db_dbnode.user_id)] > Info: DbUser.dbnodes remote columns [db_dbnode.user_id] > Info: DbUser.dbnodes local columns [db_dbuser.id] > Info: DbUser.dbnodes relationship direction symbol('ONETOMANY') > Info: DbUser.dbnodes lazy loading clause :param_1 = db_dbnode.user_id > Info: DbNode.user lazy loading clause db_dbuser.id = :param_1 > Info: DbNode.user will use Session.get() to optimize instance loads > Info: (DbNode|db_dbnode) initialize prop outputs_q > Info: DbNode.outputs_q setup primary join db_dbnode.id = db_dblink.input_id > Info: DbNode.outputs_q setup secondary join db_dbnode.id = db_dblink.output_id > Info: DbNode.outputs_q synchronize pairs [(db_dbnode.id => db_dblink.input_id)] > Info: DbNode.outputs_q secondary synchronize pairs [(db_dbnode.id => db_dblink.output_id)] > Info: DbNode.outputs_q local/remote pairs [(db_dbnode.id / db_dblink.input_id),(db_dbnode.id / db_dblink.output_id)] > Info: DbNode.outputs_q remote columns [db_dblink.input_id,db_dblink.output_id] > Info: DbNode.outputs_q local columns [db_dbnode.id] > Info: DbNode.outputs_q relationship direction symbol('MANYTOMANY') > Info: (DbNode|db_dbnode) _configure_property(inputs_q, RelationshipProperty) > Info: DbNode.inputs_q setup primary join db_dbnode.id = db_dblink.output_id > Info: DbNode.inputs_q setup secondary join db_dbnode.id = db_dblink.input_id > Info: DbNode.inputs_q synchronize pairs [(db_dbnode.id => db_dblink.output_id)] > Info: DbNode.inputs_q secondary synchronize pairs [(db_dbnode.id => db_dblink.input_id)] > Info: DbNode.inputs_q local/remote pairs [(db_dbnode.id / db_dblink.output_id),(db_dbnode.id / db_dblink.input_id)] > Info: DbNode.inputs_q remote columns [db_dblink.input_id,db_dblink.output_id] > Info: DbNode.inputs_q local columns [db_dbnode.id] > Info: DbNode.inputs_q relationship direction symbol('MANYTOMANY') > Info: DbNode.inputs_q lazy loading clause :param_1 = db_dblink.output_id AND db_dbnode.id = db_dblink.input_id > Info: DbNode.outputs_q lazy loading clause :param_1 = db_dblink.input_id AND db_dbnode.id = db_dblink.output_id > Info: (DbNode|db_dbnode) initialize prop id > Info: (DbNode|db_dbnode) initialize prop uuid > Info: (DbNode|db_dbnode) initialize prop node_type > Info: (DbNode|db_dbnode) initialize prop process_type > Info: (DbNode|db_dbnode) initialize prop label > Info: (DbNode|db_dbnode) initialize prop description > Info: (DbNode|db_dbnode) initialize prop ctime > Info: (DbNode|db_dbnode) initialize prop mtime > Info: (DbNode|db_dbnode) initialize prop attributes > Info: (DbNode|db_dbnode) initialize prop extras > Info: (DbNode|db_dbnode) initialize prop repository_metadata > Info: (DbNode|db_dbnode) initialize prop dbcomputer_id > Info: (DbNode|db_dbnode) initialize prop user_id > Info: (DbNode|db_dbnode) initialize prop dbcomments > Info: (DbNode|db_dbnode) initialize prop dbgroups > Info: (DbNode|db_dbnode) initialize prop dblogs > Info: (DbNode|db_dbnode) _post_configure_properties() complete > Info: (DbLink|db_dblink) _post_configure_properties() started > Info: (DbLink|db_dblink) initialize prop input > Info: DbLink.input setup primary join db_dblink.input_id = db_dbnode.id > Info: DbLink.input setup secondary join None > Info: DbLink.input synchronize pairs [(db_dbnode.id => db_dblink.input_id)] > Info: DbLink.input secondary synchronize pairs [] > Info: DbLink.input local/remote pairs [(db_dblink.input_id / db_dbnode.id)] > Info: DbLink.input remote columns [db_dbnode.id] > Info: DbLink.input local columns [db_dblink.input_id] > Info: DbLink.input relationship direction symbol('MANYTOONE') > Info: DbLink.input lazy loading clause :param_1 = db_dbnode.id > Info: DbLink.input will use Session.get() to optimize instance loads > Info: (DbLink|db_dblink) initialize prop output > Info: DbLink.output setup primary join db_dblink.output_id = db_dbnode.id > Info: DbLink.output setup secondary join None > Info: DbLink.output synchronize pairs [(db_dbnode.id => db_dblink.output_id)] > Info: DbLink.output secondary synchronize pairs [] > Info: DbLink.output local/remote pairs [(db_dblink.output_id / db_dbnode.id)] > Info: DbLink.output remote columns [db_dbnode.id] > Info: DbLink.output local columns [db_dblink.output_id] > Info: DbLink.output relationship direction symbol('MANYTOONE') > Info: DbLink.output lazy loading clause :param_1 = db_dbnode.id > Info: DbLink.output will use Session.get() to optimize instance loads > Info: (DbLink|db_dblink) initialize prop id > Info: (DbLink|db_dblink) initialize prop input_id > Info: (DbLink|db_dblink) initialize prop output_id > Info: (DbLink|db_dblink) initialize prop label > Info: (DbLink|db_dblink) initialize prop type > Info: (DbLink|db_dblink) _post_configure_properties() complete > Info: SELECT db_dbsetting.val > FROM db_dbsetting > WHERE db_dbsetting.key = %(key_1)s > Info: [generated in 0.00014s] {'key_1': 'repository|uuid'} > Info: ROLLBACK > Info: Pool disposed. Pool size: 5 Connections in pool: 0 Current Overflow: -5 Current Checked out connections: 0 > Info: Pool recreating > > ``` ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 18, 2024, 10:20am UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/18 "2024-01-18T10:20:41Z")
Is playing and pausing the process going through the daemon worker? When I pause and subsequently play the process, I can see the `Process status` changing in `verdi process list` but I don’t see anything pop up in the daemon worker log. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [January 18, 2024, 10:56am UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/19 "2024-01-18T10:56:09Z")
> Is playing and pausing the process going through the daemon worker? When I pause and subsequently play the process, I can see the `Process status` changing in `verdi process list` but I don’t see anything pop up in the daemon worker log. Yes, if the process is not with a daemon worker, you would get a `UnreachableProcessError`. So the fact that works, and the fact that `verdi devel rabbitmq tasks analyze` says there are no problems, suggests to me that everything is fine on this level and the task is making it to the daemon worker. What I don’t understand is that if there is some problem with continuing the process, we would see some kind of an error. Could you please update this method in your installation: [aiida-core/src/aiida/manage/external/rmq/launcher.py at 06ea130df8854f621e25853af6ac723c37397ed0 · aiidateam/aiida-core · GitHub](https://github.com/aiidateam/aiida-core/blob/06ea130df8854f621e25853af6ac723c37397ed0/src/aiida/manage/external/rmq/launcher.py#L40) Add some print statements just before and after the `_continue` call on line 86. Then repeat the steps of stopping the daemon and starting `verdi daemon worker`. This should cause the print statements to show in your terminal and we can see if it gets to and past the `self._continue` call or not. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 18, 2024, 2:05pm UTC](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220/20 "2024-01-18T14:05:27Z")
> [@sphuber](#): > > What I don’t understand is that if there is some problem with continuing the process, we would see some kind of an error. Could you please update this method in your installation: [aiida-core/src/aiida/manage/external/rmq/launcher.py at 06ea130df8854f621e25853af6ac723c37397ed0 · aiidateam/aiida-core · GitHub](https://github.com/aiidateam/aiida-core/blob/06ea130df8854f621e25853af6ac723c37397ed0/src/aiida/manage/external/rmq/launcher.py#L40) I don’t see any extra output when I do that so looks like we’re not reaching this function? EDIT: I do see some output when I use a print statement and not a LOGGER, very strange. Let me investigate a bit more. EDIT2: I see with printstament that I get to the `super()._continue` call, but I don’t see anything after that so it seems like it’s stuck somewhere in the parent method. [Next page](https://aiida.discourse.group/t/how-to-debug-a-workflow-stuck-in-waiting-state/220.md?page=2)