# Excepted workchains, due to strange error from kiwipy/plumpy?

**URL:** <https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198>\
**Category:** General Usage\
**Created:** [December 8, 2023, 4:17pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198 "2023-12-08T16:17:47Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [February 26, 2024, 10:43am UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/21 "2024-02-26T10:43:56Z")

</div>

We think we have a solution, but we are still testing it so it hasn’t been released it yet. If you want to try it as well and report whether it solves your problem, that would be great.  
You would have to do the following (check out the branch from my fork and install it):

```auto
git clone https://github.com/sphuber/aiida-core
cd aiida-core
git checkout fix/bump-engine-dependencies
pip install -e .

```

Then restart the daemon and try to run your workchains again.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [February 26, 2024, 10:46am UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/22 "2024-02-26T10:46:41Z")

</div>

And importantly, if not done already by the branch:

```auto
pip install aio-pika~=9.3

```

---

<div class="post-metadata">

**Author:** ![Anooja\_Jayaraj](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/anooja_jayaraj/32/49_2.png) [@Anooja\_Jayaraj](https://aiida.discourse.group/u/Anooja_Jayaraj)\
**Post date:** [February 27, 2024, 1:47pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/23 "2024-02-27T13:47:33Z")

</div>

I tried the above steps, restarted the daemon and ran my workchains again. Now the issue is that `verdi process list` shows the Process status QUEUED and Process state Waiting even though I see that the jobs have finished running on the remote machine and all the outputs have already been written to the expected files and folders.

---

<div class="post-metadata">

**Author:** ![Anooja\_Jayaraj](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/anooja_jayaraj/32/49_2.png) [@Anooja\_Jayaraj](https://aiida.discourse.group/u/Anooja_Jayaraj)\
**Post date:** [February 27, 2024, 1:48pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/24 "2024-02-27T13:48:53Z")

</div>

I double checked and the pip install -e . automatically also installs aio-pika (9.4.0 in my case)

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [February 27, 2024, 2:19pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/25 "2024-02-27T14:19:36Z")

</div>

Are these processes that were launched with the previous versions? I am not sure that this is supported, because there may be some changes in the internal state of the processes. Or are these processes that were started _after_ you installed the new version?

---

<div class="post-metadata">

**Author:** ![Anooja\_Jayaraj](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/anooja_jayaraj/32/49_2.png) [@Anooja\_Jayaraj](https://aiida.discourse.group/u/Anooja_Jayaraj)\
**Post date:** [February 27, 2024, 2:24pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/26 "2024-02-27T14:24:03Z")

</div>

I killed all my processes launched with the previous versions and launched the workchains again after I installed the new version. So these are processes that were started after I installed the new version.

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [February 27, 2024, 2:26pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/27 "2024-02-27T14:26:12Z")

</div>

Did you make sure to run `verdi daemon restart --reset` after installation and before launching the processes?

Can you try to run that now, followed by, after waiting for a minute or so, `verdi process play --all`. The processes might be paused (and not shown correctly by `verdi process list`) and that might spring them back to action.

---

<div class="post-metadata">

**Author:** ![Anooja\_Jayaraj](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/anooja_jayaraj/32/49_2.png) [@Anooja\_Jayaraj](https://aiida.discourse.group/u/Anooja_Jayaraj)\
**Post date:** [February 27, 2024, 2:31pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/28 "2024-02-27T14:31:46Z")

</div>

I ran `verdi daemon restart --reset` after installation and before launching the processes. I tried the same now and did play all but the state remains unchanged from Queued

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [February 27, 2024, 2:38pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/29 "2024-02-27T14:38:18Z")

</div>

Ok, could you please run

```auto
verdi daemon stop
verdi process repair
verdi daemon start

```

and report the output.

How many active processes do you have and how many daemon workers are running (`verdi daemon status`)?

If they remain stuck, can you check the output of `verdi process report` of one of the stuck calcjobs and report the output please.

---

<div class="post-metadata">

**Author:** ![Anooja\_Jayaraj](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/anooja_jayaraj/32/49_2.png) [@Anooja\_Jayaraj](https://aiida.discourse.group/u/Anooja_Jayaraj)\
**Post date:** [February 27, 2024, 2:48pm UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/30 "2024-02-27T14:48:36Z")

</div>

```auto
verdi daemon stop
verdi process repair
verdi daemon start

```

Output is

```auto
Profile: default
Stopping the daemon... OK
Warning: There are active processes without process task: {5120, 5380, 5894, 5639, 4872, 5896, 5642, 5898, 5900, 5134, 5902, 5648, 5904, 5906, 3350, 4886, 5401, 5658, 5148, 5664, 4900, 5671, 5927, 5416, 5162, 5679, 4914, 5430, 5687, 5176, 5690, 4928, 5444, 5190, 4942, 5458, 5715, 5204, 4956, 5472, 5218, 4970, 5232, 5744, 5746, 5748, 4984, 5754, 5246, 5763, 5623, 5765, 4998, 5260, 5772, 5522, 5779, 5012, 5274, 5789, 5534, 5791, 5536, 5027, 5288, 5548, 5556, 4789, 5302, 5048, 5567, 4802, 5570, 5316, 5572, 5574, 5064, 5576, 4816, 5330, 5078, 5849, 4830, 5344, 5092, 5863, 4844, 5874, 5106, 5876, 5618, 5366, 5620, 4858, 5883, 5885, 5630}
Warning: Inconsistencies detected between database and RabbitMQ.
Report: Attempting to fix inconsistencies
Report: Revived process `5120`
Report: Revived process `5639`
Report: Revived process `5642`
Report: Revived process `5134`
Report: Revived process `5648`
Report: Revived process `5658`
Report: Revived process `5148`
Report: Revived process `5664`
Report: Revived process `5671`
Report: Revived process `5162`
Report: Revived process `5679`
Report: Revived process `5687`
Report: Revived process `5176`
Report: Revived process `5690`
Report: Revived process `5190`
Report: Revived process `5715`
Report: Revived process `5204`
Report: Revived process `5218`
Report: Revived process `5232`
Report: Revived process `5744`
Report: Revived process `5746`
Report: Revived process `5748`
Report: Revived process `5754`
Report: Revived process `5246`
Report: Revived process `5763`
Report: Revived process `5765`
Report: Revived process `5260`
Report: Revived process `5772`
Report: Revived process `5779`
Report: Revived process `5274`
Report: Revived process `5789`
Report: Revived process `5791`
Report: Revived process `5288`
Report: Revived process `4789`
Report: Revived process `5302`
Report: Revived process `4802`
Report: Revived process `5316`
Report: Revived process `4816`
Report: Revived process `5330`
Report: Revived process `5849`
Report: Revived process `4830`
Report: Revived process `5344`
Report: Revived process `5863`
Report: Revived process `4844`
Report: Revived process `5874`
Report: Revived process `5876`
Report: Revived process `5366`
Report: Revived process `4858`
Report: Revived process `5883`
Report: Revived process `5885`
Report: Revived process `5380`
Report: Revived process `5894`
Report: Revived process `4872`
Report: Revived process `5896`
Report: Revived process `5898`
Report: Revived process `5900`
Report: Revived process `5902`
Report: Revived process `5904`
Report: Revived process `5906`
Report: Revived process `3350`
Report: Revived process `4886`
Report: Revived process `5401`
Report: Revived process `4900`
Report: Revived process `5927`
Report: Revived process `5416`
Report: Revived process `4914`
Report: Revived process `5430`
Report: Revived process `4928`
Report: Revived process `5444`
Report: Revived process `4942`
Report: Revived process `5458`
Report: Revived process `4956`
Report: Revived process `5472`
Report: Revived process `4970`
Report: Revived process `4984`
Report: Revived process `4998`
Report: Revived process `5522`
Report: Revived process `5012`
Report: Revived process `5534`
Report: Revived process `5536`
Report: Revived process `5027`
Report: Revived process `5548`
Report: Revived process `5556`
Report: Revived process `5048`
Report: Revived process `5567`
Report: Revived process `5570`
Report: Revived process `5572`
Report: Revived process `5574`
Report: Revived process `5064`
Report: Revived process `5576`
Report: Revived process `5078`
Report: Revived process `5092`
Report: Revived process `5106`
Report: Revived process `5618`
Report: Revived process `5620`
Report: Revived process `5623`
Report: Revived process `5630`
Starting the daemon with 1 workers... OK

```

```auto
verdi daemon status
Profile: default
Daemon is running as PID 158564 since 2024-02-27 14:43:12
Active workers [1]:
   PID MEM % CPU % started
------ ------- ------- -------------------
158580 0.815 0 2024-02-27 14:43:12

```

And that now moved the Process status to retrieve 🙂

---

<div class="post-metadata">

**Author:** ![Anooja\_Jayaraj](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/anooja_jayaraj/32/49_2.png) [@Anooja\_Jayaraj](https://aiida.discourse.group/u/Anooja_Jayaraj)\
**Post date:** [February 28, 2024, 9:17am UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/31 "2024-02-28T09:17:41Z")

</div>

Once I did verdi daemon stop, repair, start, all the processes called “HybridizationCalculation” which showed “QUEUED” in verdi but had actually finished running in the remote machine showed  
 ![image](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/ca7e0be7d38d17f4a86e98e7e49ae4cc39225292.png)

The ones with error code [100] shows

```auto
verdi process show 5906
Property Value
----------- ------------------------------------------------------------------------
type HybridizationCalculation
state Finished [100] The process did not have the required `retrieved` output.
pk 5906
uuid 839ed07e-7a6c-40e4-9c9d-613138ed4d38
label
description
ctime 2024-02-26 12:14:53.607658+00:00
mtime 2024-02-27 14:52:44.119861+00:00
computer [5] tigu

Inputs PK Type
-------------------------- ---- -------------
greens_function
    remote_results_folder 5850 RemoteData
los
    remote_results_folder 1019 RemoteData
code 103 InstalledCode
energy_grid_parameters 5409 Dict
greens_function_parameters 5408 Dict
matsubara_grid_size 5411 Int
parameters 5905 Dict
temperature 5410 Float

Outputs PK Type
------------- ---- ----------
remote_folder 5924 RemoteData

Caller PK Type
-------- ---- ------------------------
CALL 5416 CoulombDiamondsWorkChain

```

I double checked and all the HybridizationCalculations that failed have the required outputs just like the ones that finished successfully. I am not sure what causes some of the calculations to pass while some of them don’t.

Could it be due to the fact that I was using 4 daemons and the ‘verdi daemon stop, repair, start procedure’ only starts 1 daemon (that was what I set as my daemon.default\_workers) and the HybridizationCalculation processes that finished successfully were somehow tied to that one daemon?

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [February 28, 2024, 9:41am UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/32 "2024-02-28T09:41:58Z")

</div>

That is really weird. I think this may have been an artifact of the code upgrade while the processes were alive. There may have been subtle changes in the internal state. We cannot support compatibility for these unreleased changes unfortunately. Apologies for the inconvenience.

Could you please run new calculations and report back if the problem remains the same? My guess is that you won’t see this problem for calculations that are launched now, after the update is done. If the problem persists, please open a new thread, because that would be a real bug.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [September 18, 2024, 10:12am UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/33 "2024-09-18T10:12:32Z")

</div>

I write here after a while as i would be keen to understand whether this has been merged into the most recent version of aiida-core. I just installed it via pip but i see that aio-pika v6.8.1 is used, instead of ~9.x.

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [September 18, 2024, 10:15am UTC](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198/34 "2024-09-18T10:15:03Z")

</div>

The new engine fixes have not yet been released. Since they are quite impactful, we wanted to have ample time to test them on `master` before releasing.

[Previous page](https://aiida.discourse.group/t/excepted-workchains-due-to-strange-error-from-kiwipy-plumpy/198.md?page=1)