# CalcJob in \`QUEUED\` status even the actual job on HPC is finished **URL:** \ **Category:** General Usage\ **Created:** [October 15, 2024, 3:32pm UTC](https://aiida.discourse.group/t/calcjob-in-queued-status-even-the-actual-job-on-hpc-is-finished/476 "2024-10-15T15:32:29Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Xing](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/xing/32/10_2.png) [@Xing](https://aiida.discourse.group/u/Xing)\ **Post date:** [October 15, 2024, 3:32pm UTC](https://aiida.discourse.group/t/calcjob-in-queued-status-even-the-actual-job-on-hpc-is-finished/476/1 "2024-10-15T15:32:29Z")
**Problem** : the `CalcJob` processes are in the `QUEUED` status, but the actual jobs on HPC are finished **System** : Ubuntu 20.04 **AiiDA version** : 2.6.2 **Broker** : RabbitMQ v3.8.2 Here are my processes: ```auto $ verdi process list PK Created Process label ♻ Process State Process status ----- --------- ----------------- --- --------------- -------------------------------------- 15840 3D ago Cp2kBaseWorkChain ⏵ Waiting Waiting for child processes: 15846 15845 3D ago Cp2kBaseWorkChain ⏵ Waiting Waiting for child processes: 15852 15846 3D ago Cp2kCalculation ⏵ Waiting Monitoring scheduler: job state QUEUED 15851 3D ago Cp2kBaseWorkChain ⏵ Waiting Waiting for child processes: 15858 15852 3D ago Cp2kCalculation ⏵ Waiting Monitoring scheduler: job state QUEUED ... ``` I checked the remote job using: ```auto verdi calcjob gotocomputer 15846 ``` And the job is finished. Actually, all the jobs on HPC are finished, but the `CalcJob` processes are in the `QUEUED` status. I tried to pause all processes, and found the `CalcJob` processes are unreachable: ```auto $ verdi process pause --all Error: Process<15914> is unreachable. Error: Process<15902> is unreachable. Error: Process<15982> is unreachable. Error: Process<15858> is unreachable. Error: Process<15846> is unreachable. ... Report: request to pause Process<15864> sent Report: request to pause Process<15907> sent Report: request to pause Process<16044> sent Report: request to pause Process<15852> sent Report: request to pause Process<15919> sent Report: request to pause Process<15957> sent ... ``` I tried repair them using: ```auto verdi daemon stop verdi process repair verdi daemon start # then play all verdi process play --all ``` But the process is still in the `QUEUED` status. Any suggestion on how to trigger the process to `RUNNING` and then `FINISHED`? ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [October 15, 2024, 6:17pm UTC](https://aiida.discourse.group/t/calcjob-in-queued-status-even-the-actual-job-on-hpc-is-finished/476/2 "2024-10-15T18:17:52Z")
Instead of trying to force trigger the update, I would first try to figure out what is going wrong. First guess is that there might be a (temporary) problem with the transport (connection problems). If so, this would fail the scheduler update and it would start the exponential backoff mechanism. This won’t be immediately visible in the process list output though. Please check `verdi process report` if you see exception messages of the update task failing. The best way to debug this is to increase the log level, stop the daemon, and then run a worker in a shell to get the output directly: ```auto verdi config set logging.aiida_loglevel INFO verdi daemon stop verdi daemon worker ``` This will then hopefully give useful information on whether the worker actually picks up the jobs and then goes to the update task to refresh the job state. ---
**Author:** ![Xing](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/xing/32/10_2.png) [@Xing](https://aiida.discourse.group/u/Xing)\ **Post date:** [October 15, 2024, 7:22pm UTC](https://aiida.discourse.group/t/calcjob-in-queued-status-even-the-actual-job-on-hpc-is-finished/476/3 "2024-10-15T19:22:23Z")
Hi @sphuber , thanks for the suggestion! The problem is solved. I increased the log level. I then play the process manually by using its pk ```auto verdi process play pk ``` Then the log shows ```auto Info: scheduled request to update CalcJob<15882> Info: updating CalcJob<15882> successful Info: Process<15882>: Broadcasting state change: state_changed.waiting.waiting Info: scheduled request to retrieve CalcJob<15882> Info: retrieving CalcJob<15882> successful ``` And the process finished successfully. Also confirmed in the process report, there is a transport issue, because I have to update multi-factor authentication daily. However, the process status usually shows that the process is paused because the connection failed five times. But this time, it didn’t show this! It’s probably because I restarted the daemon, and the `status message` is somehow lost; I will report an issue if I encounter it. ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\ **Post date:** [October 21, 2024, 3:23pm UTC](https://aiida.discourse.group/t/calcjob-in-queued-status-even-the-actual-job-on-hpc-is-finished/476/4 "2024-10-21T15:23:22Z")
Closed on October 21, 2024, 3:23pm UTC This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.