# Why am I getting "Transport task update was cancelled"?

**URL:** <https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219>\
**Category:** General Usage\
**Tags:** question\
**Created:** [January 16, 2024, 2:35pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219 "2024-01-16T14:35:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [January 16, 2024, 2:35pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219/1 "2024-01-16T14:35:10Z")

</div>

I’m running Gaussian calculations on a remote cluster(aiida-gaussian), and I’ve been seeing the processes get paused as “Transport task update was cancelled”.

I can resume them with `verdi process play my_pk`, but I’m just wondering why this might be occurring and if there is anything I can do in configurations or something to prevent it.

Further, is there a way I could do the process play in Python? So I could say, have a loop going to check for paused processes and play them if it finds any.

Thanks

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 16, 2024, 3:20pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219/2 "2024-01-16T15:20:40Z")

</div>

Hi @kmlefran , to understand the cause behind the cancelled tasks, I would need more information. Could you please share the output of `verdi process report`. That should potentially contain more information.

As for playing paused processes from Python, see the docs on process control API: [Usage — AiiDA 2.5.0.post0 documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/processes/usage.html#the-processes-api)

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [January 16, 2024, 3:45pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219/3 "2024-01-16T15:45:43Z")

</div>

Here is one that is currently running, but was paused at one point. Since it’s so long, I’ve had to remove some of the start

\*\*\* 331: CalcJobState.WITHSCHEDULER, scheduler state: JobState.RUNNING  
\*\*\* Scheduler output: N/A  
\*\*\* Scheduler errors: N/A  
\*\*\* 14 LOG MESSAGES:  
±\> ERROR at 2024-01-14 17:02:26.235864-05:00  
packages/aiida/engine/processes/calcjobs/manager.py", line 98, in \_get\_jobs\_from\_scheduler  
| transport = await request  
| ^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 287, in **await**  
| yield self # This tells Task to wait for completion.  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/tasks.py”, line 339, in \_\_wakeup  
| future.result()  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 203, in result  
| raise self.\_exception.with\_traceback(self.\_exception\_tb)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/transports.py”, line 89, in do\_open  
| transport.open()  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py”, line 516, in open  
| self.\_client.connect(self.\_machine, \*\*connection\_arguments)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 450, in connect  
| self.\_auth(  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 781, in \_auth  
| raise saved\_exception  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 774, in \_auth  
| self.\_transport.auth\_interactive\_dumb(username)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/transport.py”, line 1711, in auth\_interactive\_dumb  
| return self.auth\_interactive(username, handler, submethods)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/transport.py”, line 1688, in auth\_interactive  
| return self.auth\_handler.wait\_for\_response(my\_event)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/auth\_handler.py”, line 249, in wait\_for\_response  
| raise AuthenticationException(“Authentication timeout.”)  
| paramiko.ssh\_exception.AuthenticationException: Authentication timeout.  
±\> ERROR at 2024-01-15 04:03:40.447453-05:00  
| Traceback (most recent call last):  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/utils.py”, line 187, in exponential\_backoff\_retry  
| result = await coro()  
| ^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/tasks.py”, line 193, in do\_update  
| job\_info = await cancellable.with\_interrupt(update\_request)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/utils.py”, line 94, in with\_interrupt  
| result = await next(wait\_iter)  
| ^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/tasks.py”, line 605, in \_wait\_for\_one  
| return f.result() # May raise f.exception().  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 203, in result  
| raise self.\_exception.with\_traceback(self.\_exception\_tb)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/manager.py”, line 132, in \_update\_job\_info  
| self.\_jobs\_cache = await self.\_get\_jobs\_from\_scheduler()  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/manager.py”, line 98, in \_get\_jobs\_from\_scheduler  
| transport = await request  
| ^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 287, in **await**  
| yield self # This tells Task to wait for completion.  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/tasks.py”, line 339, in \_\_wakeup  
| future.result()  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 203, in result  
| raise self.\_exception.with\_traceback(self.\_exception\_tb)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/transports.py”, line 89, in do\_open  
| transport.open()  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py”, line 516, in open  
| self.\_client.connect(self.\_machine, \*\*connection\_arguments)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 358, in connect  
| retry\_on\_signal(lambda: sock.connect(addr))  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/util.py”, line 279, in retry\_on\_signal  
| return function()  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 358, in   
| retry\_on\_signal(lambda: sock.connect(addr))  
| ^^^^^^^^^^^^^^^^^^  
| TimeoutError: [Errno 60] Operation timed out  
±\> ERROR at 2024-01-15 04:07:13.777695-05:00  
| Traceback (most recent call last):  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/utils.py”, line 187, in exponential\_backoff\_retry  
| result = await coro()  
| ^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/tasks.py”, line 193, in do\_update  
| job\_info = await cancellable.with\_interrupt(update\_request)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/utils.py”, line 94, in with\_interrupt  
| result = await next(wait\_iter)  
| ^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/tasks.py”, line 605, in \_wait\_for\_one  
| return f.result() # May raise f.exception().  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 203, in result  
| raise self.\_exception.with\_traceback(self.\_exception\_tb)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/manager.py”, line 132, in \_update\_job\_info  
| self.\_jobs\_cache = await self.\_get\_jobs\_from\_scheduler()  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/manager.py”, line 98, in \_get\_jobs\_from\_scheduler  
| transport = await request  
| ^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 287, in **await**  
| yield self # This tells Task to wait for completion.  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/tasks.py”, line 339, in \_\_wakeup  
| future.result()  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 203, in result  
| raise self.\_exception.with\_traceback(self.\_exception\_tb)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/transports.py”, line 89, in do\_open  
| transport.open()  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py”, line 516, in open  
| self.\_client.connect(self.\_machine, \*\*connection\_arguments)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 358, in connect  
| retry\_on\_signal(lambda: sock.connect(addr))  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/util.py”, line 279, in retry\_on\_signal  
| return function()  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/paramiko/client.py”, line 358, in   
| retry\_on\_signal(lambda: sock.connect(addr))  
| ^^^^^^^^^^^^^^^^^^  
| TimeoutError: [Errno 60] Operation timed out  
±\> WARNING at 2024-01-15 04:07:13.787168-05:00  
| maximum attempts 5 of calling do\_update, exceeded  
±\> ERROR at 2024-01-16 02:31:40.637042-05:00  
| Traceback (most recent call last):  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/utils.py”, line 187, in exponential\_backoff\_retry  
| result = await coro()  
| ^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/tasks.py”, line 193, in do\_update  
| job\_info = await cancellable.with\_interrupt(update\_request)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/utils.py”, line 94, in with\_interrupt  
| result = await next(wait\_iter)  
| ^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/tasks.py”, line 605, in \_wait\_for\_one  
| return f.result() # May raise f.exception().  
| ^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/asyncio/futures.py”, line 203, in result  
| raise self.\_exception.with\_traceback(self.\_exception\_tb)  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/manager.py”, line 132, in \_update\_job\_info  
| self.\_jobs\_cache = await self.\_get\_jobs\_from\_scheduler()  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/engine/processes/calcjobs/manager.py”, line 109, in \_get\_jobs\_from\_scheduler  
| scheduler\_response = scheduler.get\_jobs(\*\*kwargs)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/schedulers/scheduler.py”, line 361, in get\_jobs  
| joblist = self.\_parse\_joblist\_output(retval, stdout, stderr)  
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
| File “/Users/chemlab/anaconda3/envs/aiida3/lib/python3.11/site-packages/aiida/schedulers/plugins/slurm.py”, line 476, in \_parse\_joblist\_output  
| raise SchedulerError(  
| aiida.schedulers.scheduler.SchedulerError: squeue returned exit code 1 (\_parse\_joblist\_output function)  
| stdout=‘’  
| stderr=‘slurm\_load\_jobs error: Unable to contact slurm controller (connect failure)’

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 16, 2024, 3:51pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219/4 "2024-01-16T15:51:21Z")

</div>

The last line contains the hint to the problem

```auto
| raise SchedulerError(
| aiida.schedulers.scheduler.SchedulerError: squeue returned exit code 1 (_parse_joblist_output function)
| stdout=‘’
| stderr=‘slurm_load_jobs error: Unable to contact slurm controller (connect failure)’

```

The engine tries to update the status of the job, and so contacts SLURM (which is the scheduler you configured for your computer). This fails, however, and after a few retries, the engine pauses the job. This is done exactly because probably this is a transient problem, and can be resolved or will resolve itself automatically and is not to do with the job itself. Now the question is _why_ your SLURM scheduler fails to respond. Is the scheduler on shared compute infrastructure? Are there many users running or running a lot of jobs? Perhaps there is a lot of load on the scheduler and so it cannot always respond. This won’t be the first time we have seen this with SLURM. I am afraid that ther is not much that AiiDA can do about this.

---

<div class="post-metadata">

**Author:** ![kmlefran](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/kmlefran/32/48_2.png) [@kmlefran](https://aiida.discourse.group/u/kmlefran)\
**Post date:** [January 16, 2024, 6:00pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219/5 "2024-01-16T18:00:42Z")

</div>

Yeah it’s used by lots of people. Was just thinking maybe there was something that could be done on the AiiDA end configuring the computer. I only had 10 jobs running on it. Ah well, I’ll just add in some code to check for pauses.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\
**Post date:** [January 22, 2024, 2:01pm UTC](https://aiida.discourse.group/t/why-am-i-getting-transport-task-update-was-cancelled/219/6 "2024-01-22T14:01:26Z")

</div>

Closed on January 22, 2024, 2:01pm UTC

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.