Killing all processes of my user shut down the database and I cannot access again

General Usage
1

Hi everyone.

This is another post I create regarding messing up the database (sorry again).

This time I realized that the storage of my shared cluster was reaching its limit while I was running a really big workflow. The cluster was stucked, so I had no other choice but deleting the aiida_run folder (that is not a big deal though). That did not solve the problem and I still was unable to use aiida nor any other jupyter notebook. I thought it was a good idea to kill all processes associated with my user, so I did it. That solved the problem partially. Now I can use jupyter notebooks and basically all funcionalities except from aiida.

I think the issue I have is related to the fact that I killed the processes associated with postgresql. Firstly I got this error:

✘ storage: Unable to connect to profile’s storage.
Error: UnreachableStorage: Could not connect to database: (psycopg2.OperationalError) connection to server at “localhost” (127.0.0.1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?

And the logfile was:

2024-02-26 16:57:32.583 UTC [1068074] LOG: starting PostgreSQL 12.15 on x86_64-conda-linux-gnu, compiled by x86_64-conda-linux-gnu-cc (Anaconda gcc) 11.2.0, 64-bit
2024-02-26 16:57:32.583 UTC [1068074] LOG: listening on IPv4 address “127.0.0.1”, port 15451
2024-02-26 16:57:32.584 UTC [1068074] LOG: listening on Unix socket “/tmp/.s.PGSQL.15451”
2024-02-26 16:57:32.601 UTC [1068075] LOG: database system was shut down at 2024-02-26 16:57:26 UTC
2024-02-26 16:57:32.604 UTC [1068074] LOG: database system is ready to accept connections
2024-02-26 16:58:53.143 UTC [1068573] FATAL: database files are incompatible with server
2024-02-26 16:58:53.143 UTC [1068573] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 12.15.

I tried to update postgresql. I do not think I did it properly to be honest. One way or the other, the error I get now is:

✘ storage: Unable to connect to profile’s storage.
Error: UnreachableStorage: Could not connect to database: (psycopg2.OperationalError) connection to server at “localhost” (127.0.0.1), port 5432 failed: FATAL: password authentication failed for user “aiida_qs_jgarridoa_356b0e1fd35b8bba1ab60473c3a74553”
connection to server at “localhost” (127.0.0.1), port 5432 failed: FATAL: password authentication failed for user “aiida_qs_jgarridoa_356b0e1fd35b8bba1ab60473c3a74553”

But I still get the same error in the logfile:

2024-02-28 11:18:26.982 UTC [1499120] FATAL: database files are incompatible with server
2024-02-28 11:18:26.982 UTC [1499120] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 12.15.

Any idea about how to tackle this issue?

Thanks in advance

Jaime

2

Hi @jgarridoa, from the error message “2024-02-26 16:58:53.143 UTC [1068573] DETAIL: The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 12.15.” it seems you used the different PostgreSQL from the one your used for creating the database. You use version 15 to create the DB but now you have version 12.15 in use.

I guess maybe you change the PATH environment parameters and using AiiDA can not find the initial PostgreSQL path? Can you the path of PostgreSQL in your environment, I guess maybe which psql?

3

Hi, sorry for my late reply.

And I find this error weird. I have not downgraded my PostgreSQL version ever, at least not consciously. When I first installed AIIDA in that workstation in October 2023, I simply followed the usual steps and started working. I do not know how could I have downgraded my PostgreSQL version.

The path of psql in my environment is:

(aiida) jgarridoa@fatnode-tacn:~$ which psql
/home/ICN2/jgarridoa/miniconda3Aiida/envs/aiida/bin/psql

4

Just had a zoom chat with @jgarridoa. The problem is caused by somehow the postgresql installed by conda was downgrade by other dependencies.
Update the postgresql version (conda install postgresql=15) to the original one solve the problem.

One valuable feedback is that we don’t have clear conda related troubleshooting section with the commands that user can run to figure out the problem. Would be good to to document it properly. I open an issue for enhancement on this Docs: troubleshooting section for conda installed RMQ and PostgreSQL · Issue #6329 · aiidateam/aiida-core · GitHub