# SQLAlchemy Error After an Abnormal Restart **URL:** \ **Category:** General Usage\ **Tags:** question, aiida\ **Created:** [May 12, 2025, 8:47am UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613 "2025-05-12T08:47:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![YuhaoJiang](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/yuhaojiang/32/312_2.png) [@YuhaoJiang](https://aiida.discourse.group/u/YuhaoJiang)\ **Post date:** [May 12, 2025, 8:47am UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/1 "2025-05-12T08:47:51Z")
Hi all, After an abnormal computer restart, I found that I cannot properly run any workflows, and the existing node data remains accessible. Submitted workflows have a high probability of getting stuck. I’ve uploaded two error messages in attachment. [daemon\_error2.log.txt](https://aiida.discourse.group/uploads/short-url/hhcWNVFW4uHT8oMxk6qNnj1fbCx.txt) (18.3 KB) [daemon\_error.log.txt](https://aiida.discourse.group/uploads/short-url/mTSKvp4m6yC2Bq4653Mmmh2Xe1N.txt) (41.6 KB) The `verdi daemon stop` and `verdi process repair` has a small chance of restoring workflow operation. But in most cases, the workflow fails to interact with `verdi process kill` commands (It will give Error: Process\<14726\> is unreachable). I have to delete it using `verdi node delete`. I suspect there might be database issues, but cannot pinpoint the exact source. Wondering if others have encountered similar problems or can provide reliable solutions (such as how to rebuild databases). By the way I use the Postgresql database and here is my `verdi status` > ✔ version: AiiDA v2.6.3 > ✔ config: /Users/yuhao/.aiida > ✔ profile: presto > ✔ storage: Storage for ‘presto’ [open] @ postgresql://aiida-presto:\*\*\*@localhost:5432/aiida-presto / DiskObjectStoreRepository: 726fe431f0fa4dee9997b0d835eec29e | /Users/yuhao/.aiida/repository/presto/container > Warning: RabbitMQ v4.0.7 is not supported and will cause unexpected problems! > Warning: It can cause long-running workflows to crash and jobs to be submitted multiple times. > Warning: See [RabbitMQ version to use · aiidateam/aiida-core Wiki · GitHub](https://github.com/aiidateam/aiida-core/wiki/RabbitMQ-version-to-use) for details. > ✔ broker: RabbitMQ v4.0.7 @ amqp://guest:guest@127.0.0.1:5672?heartbeat=600 > /opt/homebrew/Caskroom/miniforge/base/envs/aiida-develop/lib/python3.11/site-packages/paramiko/pkey.py:82: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0. > “cipher”: algorithms.TripleDES, > /opt/homebrew/Caskroom/miniforge/base/envs/aiida-develop/lib/python3.11/site-packages/paramiko/transport.py:253: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0. > “class”: algorithms.TripleDES, > ✔ daemon: Daemon is running with PID 98342 ---
**Author:** ![ali-khosravi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/ali-khosravi/32/58_2.png) [@ali-khosravi](https://aiida.discourse.group/u/ali-khosravi)\ **Post date:** [May 12, 2025, 9:59am UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/2 "2025-05-12T09:59:30Z")
Hi @YuhaoJiang 👋 I think this refers, to a [complicated bug](https://github.com/aiidateam/aiida-core/issues/6524) in `verdi process kill` Fortunately, we fix the bug in our main branch via [this PR](https://github.com/aiidateam/aiida-core/pull/6793); soon to be released with `2.7`. You can try out the pre-release already through pip [aiida-core 2.7.0rc1](https://pypi.org/project/aiida-core/2.7.0rc1/) If after trying out this fix, still `verdi process kill` was unsuccessful, try `verdi process kill -F` to forcefully kill the process. That will detach AiiDA of “being worried” about the job status on your remote HPC, and will just consider the job as failed, without trying to kill it on the remote anymore. ---
**Author:** ![YuhaoJiang](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/yuhaojiang/32/312_2.png) [@YuhaoJiang](https://aiida.discourse.group/u/YuhaoJiang)\ **Post date:** [May 12, 2025, 2:43pm UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/3 "2025-05-12T14:43:40Z")
Thanks Ali, But I’m not sure if these are the same issue, but in my case the `verdi process kill` immediately give the > Error: Process\<14726\> is unreachable. and I can not use `verdi process repair` to fix it, the error messages are as attachments. The key issue is that my workflow will stuck at state Created, and seems that the workflow can not be saved in database. > verdi process list (aiida-develop) > PK Created Process label ♻ Process State Process status > > * * * > > 14726 8h ago PwBandsWorkChain ⏵ Running > 14733 8h ago PwBaseWorkChain ⏹ Created ---
**Author:** ![Xing](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/xing/32/10_2.png) [@Xing](https://aiida.discourse.group/u/Xing)\ **Post date:** [May 12, 2025, 3:04pm UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/4 "2025-05-12T15:04:52Z")
For the process that is stuck in the `Created` state, try to run this: ```auto from aiida.manage import get_manager from aiida import load_profile load_profile() pk = 14733 # process in `Created` state process_controller = get_manager().get_process_controller() process_controller.continue_process(pk) ``` ---
**Author:** ![YuhaoJiang](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/yuhaojiang/32/312_2.png) [@YuhaoJiang](https://aiida.discourse.group/u/YuhaoJiang)\ **Post date:** [May 12, 2025, 4:55pm UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/5 "2025-05-12T16:55:59Z")
Still don’t work 🙁 > In [3]: > …: from aiida.manage import get\_manager > …: from aiida import load\_profile > …: > …: load\_profile() > …: > …: pk = 14733 # process in `Created` state > …: process\_controller = get\_manager().get\_process\_controller() > …: process\_controller.continue\_process(pk) > Out[3]: \ But nothing happened in daemon log, and the process still stuck at Created. After restart the daemon and `verdi process repair`, the daemon log still report the similar error message [daemon\_error3.log.txt](https://aiida.discourse.group/uploads/short-url/yF3LMD3mw1cszstHB6o87QZNp1s.txt) (18.9 KB) For almost every Workflow I created will face the same stuck, I wonder if there is something went wrong with my database? ---
**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\ **Post date:** [May 12, 2025, 6:33pm UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/6 "2025-05-12T18:33:52Z")
From your logs, indeed there are a lot of database errors. Many I think are indirectly related to the main problem, but there are a few InternalErrors, so indeed I think your DB got corrupted. Hopefully, it’s still recoverable (did you run backups until now? AiiDA now has a built-in backup command) In particular from e.g. this error: ```auto (psycopg2.errors.IndexCorrupted) table tid from new index tuple (1135,10) overlaps with invalid duplicate tuple at offset 18 of block 9 in index "ix_db_dbnode_db_dbnode_process_type" ``` It seems the index got corrupted (if it’s just the index, that’s not too bad - let’s hope the data is still here. I recommend stopping any daemon and any connected section (Jupyter notebook loading a profile, verdi shell, …). Then you can try to connect to your database (in ubuntu `sudo su - postgres` and then run the `psql` command (e.g. `psql template1`). Careful about the commands, as you risk to delete everything! You can try e.g. the command below with the correct table name. [https://stackoverflow.com/questions/76251129/how-to-fix-postgresql-errors-telling-tid-from-new-index-tuple-overlaps-with-inva](https://stackoverflow.com/questions/76251129/how-to-fix-postgresql-errors-telling-tid-from-new-index-tuple-overlaps-with-inva) Another option (I would even start from this) is to dump the DB to 1. have a backup and 2. if the error does not get solved by rebuilding all indexes, you can restore it from the dump. Follow e.g. > **[25.1. SQL Dump](https://www.postgresql.org/docs/current/backup-dump.html)** > > 25.1. SQL Dump # 25.1.1. Restoring the Dump 25.1.2. Using pg\_dumpall 25.1.3. Handling Large Databases The idea behind this dump method … ---
**Author:** ![Xing](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/xing/32/10_2.png) [@Xing](https://aiida.discourse.group/u/Xing)\ **Post date:** [May 13, 2025, 7:50pm UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/7 "2025-05-13T19:50:22Z")
While you’re trying the method suggested by @giovannipizzi, it might also be worth a quick attempt to delete the problematic node (PK 14733) and restart the daemon. The logs indicate the daemon encountered an error with that node, and since it’s likely not recoverable anyway, removing it may help. ---
**Author:** ![YuhaoJiang](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/yuhaojiang/32/312_2.png) [@YuhaoJiang](https://aiida.discourse.group/u/YuhaoJiang)\ **Post date:** [May 15, 2025, 3:53am UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/8 "2025-05-15T03:53:53Z")
Thanks Giovanni, Xing and Ali! After reindex the database, I have tried few tests and they can finish normally. 😃 ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\ **Post date:** [May 20, 2025, 11:54pm UTC](https://aiida.discourse.group/t/sqlalchemy-error-after-an-abnormal-restart/613/9 "2025-05-20T23:54:20Z")
This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.