I am running AiiDA on my laptop for two years now, and running low on disk space. The size of my container repository is at 215 GB. Today I tried the command “verdi storage maintain” to try to reduce disk space usage, however the command failed shortly with the error “No space left on the device”. It seems that running this command first packs all the 215 GB of loose files, essentially duplicating all data, requiring around 430 GB of space in total. This is more than what I can make available on my laptop. Further, now that I have run this command without completing, the “packs” directory in my container uses an additional 24 GB which I cannot free up again, and I am completely out of disk space. What would be the best approach to either reverse the packing and free up the 24 GB, or to complete the maintenance (which I assume would at completion delete all loose files)? This is my storage info:
This suggests that the packing is less than 10% complete. I could temporarily free up about 100 GB on my laptop, although it would require some effort. Alternatively, could I temporarily use an external hard drive? Any help is appreciated.
to free up space in your current solution, you can free up the loose files that have been already packed. I suggest to anyways free up a few hundreds of MBs in case the SQLite DB needs to be partially updated. Then
run verdi profile show for your profile, check the file path of the storage
go in that folder (there should be a container folder inside)
open a ipython shell in that folder
Then run:
import disk_objectstore as dos
container = dos.Container('container')
container.clean_storage()
Regarding the more general issue: we are aware of this issue and are working to improve it. @geiger_j has been already working to perform the maintainance operations in smaller chunks (typically ~4GB). The PR is WIP here, but not all tests have been implemented. If you have a backup and want to try it, we believe it should work (changes are minimal; it’s mostly some advanced usage and performance that we want to double check, e.g. if you run the command while AiiDA is in use, but in your case I suggest that you stop any daemon and just run the maintenance); and then you could report if it worked in your case (again, do a backup first). @geiger_j can tell you which command to run similar to above, accessing directly the container from a python script.
Thanks a lot for the quick solution. I did the clean storage in python as you suggested, temporarily freed up some space on my disk and ran the storage maintenance in a few batches (repeating this procedure in between). Now it is all fine.