Hi,
While fixing the issue using AiiDaQE in multiple Conda environments, I moved the configuration folder of my environment to a new location and updated the AIIDA_PATH variable as suggested here. I moved the config directory to "$CONDA_PREFIX", but I missunderstood the documentation instructions and initially set:
AIIDA_PATH="$CONDA_PREFIX/.aiida"`
This mistake caused AiiDA to fail to locate the configuration folder, leading to the creation of a new one:
I quickly realized my mistake when my AiiDA profile was missing. I deleted the newly created .aiida folder, corrected the AIIDA_PATH variable , and restarted the environment in a new terminal tab.
Just to be clear, this behavior is exactly as described in the warning in the documentation :
However, after correcting the issue, I discovered that the storage folder was somehow modified, and now the storage is corrupted:
(aiida) gjoalland@ctimac001:~/.conda/envs/aiida/.aiida$ verdi status
✔ version: AiiDA v2.6.2.post0
✔ config: /home/gjoalland/.conda/envs/aiida/.aiida
✔ profile: gjoalland
✘ storage: Storage is corrupted.
Error: CorruptStorage: The database has a repository UUID configured to c228ae00760a4523aa0788862923716d but the disk-objectstore's is b6297ef9a24b4237aa289f4d39bb7490.
✔ broker: RabbitMQ v3.12.1 @ amqp://guest:guest@127.0.0.1:5672?heartbeat=600
✔ daemon: Daemon is running with PID 511294
Any suggestion on how to solve the problem?
On my own, I attempted several solutions and confirmed that the database and nodes are still present, but many operations remain unavailable.
Does anyone have suggestions on how to resolve this issue?
Thanks in advance!
One approach I tried was manually updating the storage configuration:
Locate the storage folder
I found the storage location using the following command:
gjoalland@ctimac001:~/.conda/envs/aiida$ verdi profile show gjoalland | grep repository
repository_uri: file:///home/gjoalland/repository/gjoalland
Modify thecontainer_idkey
I edited the file /home/gjoalland/repository/gjoalland/container/config.json , replacing the value of "container_id":
"b6297ef9a24b4237aa289f4d39bb7490"
with
"c228ae00760a4523aa0788862923716d"
This partially resolved the issue. I regained access to storage nodes, and initially, everything seemed to be working fine:
(aiida) gjoalland@ctimac001:~/repository$ verdi status
✔ version: AiiDA v2.6.2.post0
✔ config: /home/gjoalland/.conda/envs/aiida/.aiida
✔ profile: gjoalland
✔ storage: Storage for 'gjoalland' [open] @ postgresql+psycopg://gjoalland:***@localhost:5432/aiida_database / DiskObjectStoreRepository: c228ae00760a4523aa0788862923716d | /home/gjoalland/repository/gjoalland/container
✔ broker: RabbitMQ v3.12.1 @ amqp://guest:guest@127.0.0.1:5672?heartbeat=600
⏺ daemon: The daemon is not running.
However, after testing further, I found that some features remain broken. For example, I cannot submit calculations, and attempting to retrieve calculation outputs with verdi calcjob outputcat <PK> results in errors like this:
(aiida) gjoalland@ctimac001:~$ verdi calcjob outputcat 20744
Critical: Could not open output path "aiida.out". Exception: object with key `07fd16323f5022864707cba97a86a06911c99c9e61e6d3e491c683d3ffd024f2` does not exist.
Does anyone have suggestions on how to fully resolve this issue?
Hi, I’m not sure I follow 100% what exactly happened, but I give the explanation of what should happen, hopefully you can fix things in this way.
AiiDA stores node information in 2 places:
a. the database (where the node PK, UUID, modification time, links, … are stored, but also all attributes and extras of the nodes).
b. the repository (a folder on disk, managed by the disk-objectstore library for efficiency)
The two have to be always “paired together”: the DB it’s just written something like "This node with PK= 20744 has a file that is stored in the corresponding repository, the filename is aiida.out and the file hash (to find it in the repository) is 07fd16323f5022864707cba97a86a06911c99c9e61e6d3e491c683d3ffd024f2
In order to ensure that you don’t mix by mistake repository and DBs of different repos (what’s happening in your case…), they are assigned an unique UUID (the same for each pair), this is checked and this is the error you get. Indeed, somehow your configuration is trying now to mix a DB and a different repository.
Of course if, as you did, change the UUID of one of the two, AiiDA will not notice immediately, but the files you are looking for will not be there anymore.
So the solution, in your case, (if the DB is correct, which I assume since you know the PK and you get correctly that it’s a CalcJob etc. - but you assigned the wrong repo) is to find where the “correct” repository is, before you moved things around. It should have the correct container_id. You can check in the places where AiiDA would put it by default, that is, as a subfolder of the corresponding .aiida folder.
Thanks for the quick reply and the clear explanations! After further investigation, I can confirm that I do not have another .aiida configuration folder:
(aiida) gjoalland@ctimac001:~/.conda/envs/aiida/.aiida$ sudo find / -type d -name ".aiida"
/home/gjoalland/.conda/envs/aiida/.aiida
This folder doesn’t contain anything particularly relevant except for several config.json files, all of which point to the same storage location that I already know:
This confirms that even two weeks ago, the storage repository was the same, meaning it was likely erased rather than moved.
Just in case, I also searched for any other directories named gjoalland that might contain a storage repository:
(aiida) gjoalland@ctimac001:~/.conda/envs/aiida/.aiida$ sudo find / -type d -name "gjoalland"
/home/gjoalland # the home folder
/home/gjoalland/.conda/envs/aiida/.aiida/access/gjoalland # a subfolder of the config folder. It does not contain any relevent information
/home/gjoalland/.conda/envs/aiida/SAVE_.aiida/access/gjoalland # a copy of the config folder I created
/home/gjoalland/repository/gjoalland # the storage folder of the gjoalland profile
I’m starting to wonder if the “links” you described in 2. still exist. Unfortunately, it doesn’t seem like they do.
Is there any way to restore them? Unfortunately, I don’t have a backup…
If not, would it be possible to delete my current AiiDA profile while keeping the database, then create a new profile with a fresh storage repository but linked to the same database? Would that restore the links?
Mmm that’s strange. I don’t see where AiiDA would delete the container. The only way I see is if you run verdi profile delete (but then you would do it consciously, and also the DB would be deleted, in principle).
There is no other “link” except that the DB has the checksum SHA256 of each file of a node (e.g. 07fd16323f5022864707cba97a86a06911c99c9e61e6d3e491c683d3ffd024f2) - and that the repository container has the same UUID as the DB.
One more thing, just to do a final check, would be: the repository contains folders named loose, duplicates, packs, sandbox (and a file called packs.idx). You could try to do a search for those folders/files…
If not, would it be possible to delete my current AiiDA profile while keeping the database, then create a new profile with a fresh storage repository but linked to the same database? Would that restore the links?
No - I mean, what you did by changing the UUID is essentially this, but the point is: if you have a file (e.g. the raw input or output of a calculation) belonging to a node, that node is always going to complain when you try to access that file, because you don’t have it anymore (the fresh storage repository would also be empty). You should really try to find the repository…
Reading back your earlier message above: is it possible that you didn’t just delete /home/gjoalland/.conda/envs/.aiida/.aiida but /home/gjoalland/.conda/envs/.aiida/ that possibly contained the repository, thus deleting it?
After some further investigations, the folder appears to be the correct one, as it contains the folders named loose, duplicates, packs, and sandbox, as well as the packs.idx file:
(base) gjoalland@ctimac001:~/repository/gjoalland/container$ ls
config.json loose packs.idx packs.idx-wal
duplicates packs packs.idx-shm sandbox
(base) gjoalland@ctimac001:~/repository/gjoalland/container$ ls packs
(base) gjoalland@ctimac001:~/repository/gjoalland/container$ ls sandbox/
(base) gjoalland@ctimac001:~/repository/gjoalland/container$ ls duplicates/
(base) gjoalland@ctimac001:~/repository/gjoalland/container$ ls loose
35 3f 51 9a
However, the loose folder contains only a few files, all created after the incident. This strongly suggests that most of my data has been lost. I used to have over 30,000 nodes, so I would expect roughly 30,000 files spread across the four folders in loose.
Following this message, I was carefull not to do that. I copied the folder elsewhere and then deleted /home/gjoalland/.conda/envs/.aiida/ before moved it back afterward.
After some investigation, I remembered that after moving back the folder, I did run verdi profile delete gjoalland --keep-data, and then reconfigured it using verdi profile setup. I had forgotten this step as in did not change the output of the verdi status command. However, this probably explains the deletion of the files in the loose directory and is consistent with the fact that the database itself was not deleted.
So… I guess I’m kind of screwed, right?
I suppose there’s no other solution than to create a fresh environment and try to recover whatever I can from the corrupted environment. I’ll attempt this by editing the file /home/gjoalland/repository/gjoalland/container/config.json, replacing the value of "container_id" with "c228ae00760a4523aa0788862923716d" in the hope of recovering anything that might still be accessible.
I have to say that yes, from your report, I don’t think there is much more that can be done… Note that AiiDA since 2.6 comes with a backup command. We strongly suggest running backups (in general, but in particular before any kind of migration). I’m sorry…
