# How to debug \`engine.submit\` processes? **URL:** \ **Category:** General Usage\ **Created:** [September 26, 2024, 4:48pm UTC](https://aiida.discourse.group/t/how-to-debug-engine-submit-processes/470 "2024-09-26T16:48:18Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![agoscinski](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/agoscinski/32/147_2.png) [@agoscinski](https://aiida.discourse.group/u/agoscinski)\ **Post date:** [September 26, 2024, 4:48pm UTC](https://aiida.discourse.group/t/how-to-debug-engine-submit-processes/470/1 "2024-09-26T16:48:18Z")
Hello all, Sometimes I would like to debug submitted jobs to see what is going wrong, but I am not sure if this is possible. Often I can go back to `engine.run` but some things only fail during `engine.submit` and it would be therefore helpful to be also debug `engine.submit` jobs. On one side I am puzzled how processes are executed with `engine.submit`. When I `engine.run` a job I end up in [the plumpy.processes.Process.execute](https://github.com/aiidateam/plumpy/blob/b3837fc9dbf7dc5aca0785e93b94cf5b89d04a91/src/plumpy/processes.py#L1194) function but it appears that this does not happen in the `engine.submit` case (I write to a file to an absolute path as a check). I tried to see where `engine.submit` goes and besides the initialization I could track it to [send\_task](https://github.com/aiidateam/plumpy/blob/b3837fc9dbf7dc5aca0785e93b94cf5b89d04a91/src/plumpy/process_comms.py#L482) with the message ```auto {'task': 'continue', 'args': {'pid': 354, 'nowait': False, 'tag': None}} ``` where this goes to `kiwipy.rmq.threadcomms.RmqThreadCommunicator`. Is it possible to see where this task is received? I tried the web interface from `rabbitmq_management` but I cannot find anything in this interface. In [this stackexchange](https://stackoverflow.com/questions/10709533/is-it-possible-to-view-rabbitmq-message-contents-directly-from-the-command-line) other alternatives are suggested but maybe there is already a solution that people have established? Another problem I have is that the process is run by a daemon worker, I guess would need to get into its running instance. I tried to use `verdi daemon start --foreground` and `verdi daemon worker` to see if somehow this allows me to receive kill commands for which I know the breakpoints for in the [plumpy code](https://github.com/aiidateam/plumpy/blob/b3837fc9dbf7dc5aca0785e93b94cf5b89d04a91/src/plumpy/processes.py#L903), but it does not really work. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [September 26, 2024, 10:45pm UTC](https://aiida.discourse.group/t/how-to-debug-engine-submit-processes/470/2 "2024-09-26T22:45:26Z")
The `send_task` sends a task to RabbitMQ. That will then send the task to any worker that is subscribed to the task queue, which will then deserialize the process from the database and continue running it. So you are right, it will probably end up in a daemon worker if you are running the daemon. To debug this, I would shut down the daemon, and then run `verdi daemon worker` which runs a single worker in the foreground. Or rather, you probably want to run the actual python function that the CLI command calls, so that you can call it from a debugger. It should be `aiida.engine.daemon.worker.start_daemon_worker`. ---
**Author:** ![agoscinski](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/agoscinski/32/147_2.png) [@agoscinski](https://aiida.discourse.group/u/agoscinski)\ **Post date:** [October 4, 2024, 9:04am UTC](https://aiida.discourse.group/t/how-to-debug-engine-submit-processes/470/3 "2024-10-04T09:04:04Z")
Thanks for the reply! Now it works. I was trying to do this before but I think because I had a daemon running for some other project and used the same ports for rabbitmq my `verdi daemon worker` did not pick up the jobs. Also important to decrease the number of existing daemon workers to 0 so one can be sure only the `verdi daemon worker` pick up the jobs. > Or rather, you probably want to run the actual python function that the CLI command calls, so that you can call it from a debugger. It should be `aiida.engine.daemon.worker.start_daemon_worker` . On can also run the verdi commands in debug mode ```auto python -m pdb $(which verdi) daemon worker ``` One has to go through some CLI layers, but creating a breakpoint in the aiida source code however makes this easy [aiida-core/src/aiida/cmdline/commands/cmd\_daemon.py at c7c289d3892bf76894714f53f58b7ce5b0761178 · aiidateam/aiida-core · GitHub](https://github.com/aiidateam/aiida-core/blob/c7c289d3892bf76894714f53f58b7ce5b0761178/src/aiida/cmdline/commands/cmd_daemon.py#L280) . In the end your suggestion does not require to make temporary changes in the existing code base. ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\ **Post date:** [October 4, 2026, 9:04am UTC](https://aiida.discourse.group/t/how-to-debug-engine-submit-processes/470/4 "2026-10-04T09:04:13Z")
This topic was automatically closed 730 days after the last reply. New replies are no longer allowed.