Warning: Inconsistencies detected between database and RabbitMQ

New to AiiDA
1

Hi

I have created a submission task but it is created only.

(aiidaENV) rkarkee@ch-fe2:/lustre/scratch5/rkarkee> verdi process list
  PK  Created    Process label           ♻    Process State    Process status
----  ---------  ----------------------  ---  ---------------  ----------------
8857  4m ago     IRamanSpectraWorkChain       ⏹ Created

Total results: 1

Report: ♻ Processes marked with check-mark were not run but taken from the cache.
Report: Add the option `-P pk cached_from` to the command to display cache source.
Report: Last time an entry changed state: 4m ago (at 08:34:44 on 2024-04-08)
Report: Checking daemon load... OK
Report: Using 0% of the available daemon worker slots.

I tried

verdi daemon stop
verdi process repair
verdi daemon start

Then, in process repair, I saw this warning:

(aiidaENV) rkarkee@ch-fe2:/lustre/scratch5/rkarkee> verdi process repair
Warning: There are active processes without process task: {8857}
Warning: Inconsistencies detected between database and RabbitMQ.
Report: Attempting to fix inconsistencies
Report: Revived process `8857`

The status is still created only. I am not sure how may I run the started task. I believe it may have something to do with the warning.

My verdi status is:

(aiidaENV) rkarkee@ch-fe2:/lustre/scratch5/rkarkee> verdi status
 ✔ version:     AiiDA v2.5.1
 ✔ config:      /users/rkarkee/.aiida
 ✔ profile:     quicksetup
 ✔ storage:     Storage for 'quicksetup' [open] @ postgresql://aiida_qs_rkarkee_20554bcc4bead70a3479c4ef8d5f1f4e:***@localhost:5434/quicksetup_rkarkee_20554bcc4bead70a3479c4ef8d5f1f4e / DiskObjectStoreRepository: 983ccf2a46b74550938fdb7753c1117f | /users/rkarkee/.aiida/repository/quicksetup/container
 ✔ rabbitmq:    Connected to RabbitMQ v3.8.14 as amqp://guest:guest@127.0.0.1:5672?heartbeat=600
 ✔ daemon:      Daemon is running with PID 85831

Also, I could not kill this task even after following the suggestion below.

(aiidaENV) rkarkee@ch-fe2:/lustre/scratch4/turquoise/rkarkee/HfTe5/convectional> verdi process kill 8857
Error: Process<8857> is unreachable.
Report: If one ore more processes are unreachable, you can run the following commands to try and repair them:

    verdi daemon stop
    verdi process repair
    verdi daemon start
2

Thanks @rkarkee. I’m thinking that whatever caused the task to get lost in the first place simply happened again. This would mean that the process is unreachable (again).

The error that caused this should be reported in the daemon logs. Could you try running the following command:

tar -czvf daemon-logs.tgz -C $AIIDA_PATH/.aiida/daemon/log .

And send me the resulting tarball, e.g. via wormhole:

3

Hi @mbercx

There are two shell environment in my HPC under ch-fe1 and ch-fe2, but whenever I login though ssh, it automatically either send me to rkarkee@ch-fe1 or rkarkee@ch-fe2.
When I installed and created everything in aiida, it was on ch-fe1.

This error occurs whenever I am in ch-fe2 and once I switch back to ch-fe1, it is fine.

I dont know the reason why this might be happening but at least that solved my problem.

Thanks

4

Interesting. I don’t have a lot of experience with running AiiDA on the HPC machine, but good to know that multiple login nodes can cause issues.