I am actively using AiiDA for 2.5 years now. My .aiida repository has grown very large (350 GB, 500 000 nodes) and I am running out of disk space (500 GB hard drive). My repository contains various types of DFT and MD calculations, some of which I need frequently and therefore must be stored locally, and others I only need to keep just in case.
I need an effective way to back-up (archive?) all data, and then manually delete nodes based on certain criteria (e.g. all LAMMPS calculations with more than 1000 atoms older than 6 months). I should also be able to easily import this data again without creating duplicates. Further, it would be a bonus if I could append to an existing archive so that the new local data would merge with the older data that is no longer stored locally. Even better, would it be possible to automate backing up, freeing up space locally and retrieving from cloud when needed in a cloud sync setup?
What is the best way to achieve this in AiiDA? I realize that I probably should have used different profiles for calculations with different purposes - is there a way to do this separation retrospectively? Any advice on good data management practices in AiiDA is appreciated.