# \`verdi archive\` runs out of space even though it is available. Maybe chunking does not work?

**URL:** <https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409>\
**Category:** General Usage\
**Created:** [June 7, 2024, 9:53am UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409 "2024-06-07T09:53:12Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![agoscinski](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/agoscinski/32/147_2.png) [@agoscinski](https://aiida.discourse.group/u/agoscinski)\
**Post date:** [June 7, 2024, 9:53am UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/1 "2024-06-07T09:53:12Z")

</div>

During the archiving of nodes the process stops very early on. It is a very large database of 2TB, but the space should be there on partition. This is the short error message (whole log at the end as file).

```auto
Archiving database: nodes 0.0%| | 378/16749234
Traceback (most recent call last):
  File "/home/aiida/.virtualenvs/aiida-fermisurf/lib/python3.8/site-packages/archive_path/zip_path.py", line 325, in open
    yield handle
  File "/home/aiida/envs/aiida-fermisurf/code/aiida-core/aiida/tools/archive/implementations/sqlite_zip/writer.py", line 182, in _stream_binary
    shutil.copyfileobj(handle, zip_handle)
  File "/usr/lib/python3.8/shutil.py", line 208, in copyfileobj
    fdst_write(buf)
  File "/usr/lib/python3.8/zipfile.py", line 1141, in write
    self._fileobj.write(data)
OSError: [Errno 28] No space left on device

```

Maybe the caveat is that the space is available on a different partition and not on the partition where the current aiida database is stored. I thought that this might be because the data is not chunked properly and added for a local tests the `-b 1000` option to the archive command, but this still invokes this line

> <https://github.com/aiidateam/aiida-core/blob/d0e1de261881daec87c452369f17c1f2533e3fe5/aiida/tools/archive/implementations/sqlite_zip/writer.py#L182>

during archiving that does not chunk. I am not sure if this fixes the problem. I could hard code it to try out, bu I have doubts that this is the actual solution. On the directory we archive from there is still 1.6TB space free and the database consists is a lot of loose files that should not be that large individually.

### Environment

```auto
py[aiida-fermisurf]  prnmarvelsrv3  ~/e/aiida-fermisurf   main ±  verdi status
 :heavy_check_mark: version: AiiDA v2.3.1
 :heavy_check_mark: config: /home/aiida/envs/aiida-fermisurf/.aiida
 :heavy_check_mark: profile: main
 :heavy_check_mark: storage: Storage for 'main' [open] @ postgresql://aiida:***@localhost:5432/aiida-fermisurf / DiskObjectStoreRepository: 580f68c6eb6347b7a203030615ef1d40 | /hith/aiida-fermisurf/repositories/main/container
 :heavy_check_mark: rabbitmq: Connected to RabbitMQ v3.8.2 as amqp://guest:guest@127.0.0.1:5672?heartbeat=600
 :heavy_check_mark: daemon: Daemon is running with PID 1193148

```

### Full logs

[verdi\_archive-out\_of\_diskspace.txt](https://aiida.discourse.group/uploads/short-url/ys8Ootvq7p96KeWYha1wytqgHRq.txt) (6.0 KB)

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [June 7, 2024, 10:51am UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/2 "2024-06-07T10:51:39Z")

</div>

Just for completeness, what is the exact command that you invoked? And what is the output of `df -h` of the filesystem of the output file.

> I thought that this might be because the data is not chunked properly and added for a local tests the `-b 1000` option to the archive command, but this still invokes this line

The batch size option is not the same as the `buffer_size` argument in the `Writer._stream_binary` method. The batch size dictates the size of the number of nodes that are queried at once from the source database before writing them to the target. The `buffer_size` here is the number of bytes with which the write writes files to the archive.

Anyway, I don’t think the buffering is the problem here. This is mostly to prevent out of memory errors by not reading too much into memory at once. If there is not enough space on the target disk, changing the batch and/or buffer size is not going to help.

> It is a very large database of 2TB,

How did you determine this? When you say database, do you mean the total storage, or just the PostgreSQL database? When you create an archive your are writing the sum of the PSQL database _and_ the file repository to a single file. So your target diskspace should have _at least_ that.

---

<div class="post-metadata">

**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\
**Post date:** [June 7, 2024, 12:13pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/3 "2024-06-07T12:13:49Z")

</div>

I didn’t look into the details of the code, but most probably the code is creating some temporary files on disk (maybe [here](https://github.com/aiidateam/aiida-core/blob/f553f805e86d766da6208eb1682f7cf12c7907ac/src/aiida/tools/archive/implementations/sqlite_zip/writer.py#L77C14-L77C23)? Or maybe somewhere else as well). This will go by default in the [location specified in env vars like `TMPDIR`, `TEMP` or `TMP`](https://docs.python.org/3/library/tempfile.html#tempfile.mkstemp) (see more specific logic [here](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir)). Typically it’s `/tmp` that is in a different partition.

My suggestion is to try, instead of

```auto
verdi archive XXX

```

to run

```auto
TMPDIR=/some/path/to/a/temp/folder/on/the/big/partition verdi archive XXX

```

If this works, we know this is the problem (my bet is that this is the cause). Then you can proceed, but we should then discuss what is the best strategy to prevent this kind of “suprise effect” - maybe changing the TMP folder for this specific operation to the same partition of the destination by default, but still allowing to optionally specify it in the `verdi` command line (e.g. if the space left there is “just enough” and you want to put the temp files in a different large partition).

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [June 7, 2024, 12:32pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/4 "2024-06-07T12:32:54Z")

</div>

Thanks Giovanni. This is indeed very likely the case. The zip writer uses the temp folder as working space. The _file repository contents_ are directly written to the archive (which is the filename specified in the `verdi archive create` command), however, the database contents are written to an sqlite database (i.e. file) inside that working space. When the context manager is exited, that database file is written directly to the archive.

We should document this and for now provide the workaround you highlighted. And then we should discuss what a more permanent solution should look like.

---

<div class="post-metadata">

**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\
**Post date:** [June 7, 2024, 1:03pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/5 "2024-06-07T13:03:56Z")

</div>

Great! I’m curious to see if the workaround I suggest indeed fixes the problem. This is also marginally related to what I was discussing with @geiger_j, @agoscinski, @ali-khosravi and @eimrek, about having a profile accessing directly the sqlite\_zip file. However, in an efficient way to avoid redecompressing at every profile load, but caching the unzipped sqlite db. This will have to be probably written in the profile config.json file, and be configurable for similar reasons to those above. For context, this is to be able to serve the rest api (eg for Materials Cloud) in read only mode directly from an .aiida file, without having to import everything in a new profile, thus making deployment much easier.

---

<div class="post-metadata">

**Author:** ![agoscinski](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/agoscinski/32/147_2.png) [@agoscinski](https://aiida.discourse.group/u/agoscinski)\
**Post date:** [June 7, 2024, 1:28pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/6 "2024-06-07T13:28:47Z")

</div>

> most probably the code is creating some temporary files on disk (maybe [here](https://github.com/aiidateam/aiida-core/blob/f553f805e86d766da6208eb1682f7cf12c7907ac/src/aiida/tools/archive/implementations/sqlite_zip/writer.py#L77C14-L77C23)? Or maybe somewhere else as well). This will go by default in the [location specified in env vars like `TMPDIR`, `TEMP` or `TMP`](https://docs.python.org/3/library/tempfile.html#tempfile.mkstemp) (see more specific logic [here](https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir)). Typically it’s `/tmp` that is in a different partition.

Yes seems more promising. Will try it out

> Just for completeness, what is the exact command that you invoked? And what is the output of df -h of the filesystem of the output file.

### Filesystem on HPC and verdi archive commend

Right. So here again the relevant information of `df -h` and the whole log at the end

```auto
df: /scratch: Stale file handle
Filesystem Size Used Avail Use% Mounted on
...
prnmarvelsrv4:/hith 18T 5.2T 12T 32% /hith_nfs/prnmarvelsrv4 <--- where to archive
...
prnmarvelcommonfs:/home 3.7T 1.9T 1.6T 55% /home <-- repository and $AIIDA_PATH

```

### Command to archive

We used this command

```auto
verdi archive create -a /hith_nfs/prnmarvelsrv4/paulish_qiao-main.aiidaarchive

```

We checked that there is only one profile, so we don’t back up potentially files from other profiles with `-a`

### AiiDA database information

> How did you determine this? When you say database, do you mean the total storage, or just the PostgreSQL database? When you create an archive your are writing the sum of the PSQL database _and_ the file repository to a single file. So your target diskspace should have _at least_ that.

I meant the repository mainly. IT the the same as here [Performance issues with verdi commands `verdi process` and `verdi storage`](https://aiida.discourse.group/t/performance-issues-with-verdi-commands-verdi-process-and-verdi-storage/398) I will copy there things over

```auto
$ du -h --max-depth=1 /home/aiida/repository/main/container
4.0K	./duplicates
4.0K	./sandbox
2.0T	./loose
4.0K	./packs
2.0T	.

```

additonal information. Size of AIIDA path that keeps all the process id’s

```auto
$ du -h --max-depth=1 $AIIDA_PATH/.aiida/access
2.2G	/home/aiida/envs/aiida-fermisurf/.aiida/access/main
2.2G	/home/aiida/envs/aiida-fermisurf/.aiida/access

```

Number of files in this path

```auto
$ find $AIIDA_PATH/.aiida/access -type f | wc -l
557638

```

```auto
$ verdi storage info
entities:
  Users:
    count: 10
  Computers:
    count: 31
  Nodes:
    count: 4734923
  Groups:
    count: 331
  Comments:
    count: 0
  Logs:
    count: 1203422
  Links:
    count: 8936853
repository:
  SHA-hash algorithm: sha256
  Compression algorithm: zlib+1

```

### Appendix

[df\_h.txt](https://aiida.discourse.group/uploads/short-url/2U3rt0QkrumkeaWa1BI59CnxB9U.txt) (1.8 KB)

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [June 7, 2024, 2:36pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/7 "2024-06-07T14:36:14Z")

</div>

> [@giovannipizzi](#):
>
> This is also marginally related to what I was discussing with @geiger_j, @agoscinski, @ali-khosravi and @eimrek, about having a profile accessing directly the sqlite\_zip file. However, in an efficient way to avoid redecompressing at every profile load, but caching the unzipped sqlite db. This will have to be probably written in the profile config.json file, and be configurable for similar reasons to those above.

This is already more or less possible. You can essentially “mount” an archive as a profile. For example:

```auto
verdi profile setup core.sqlite_zip --profile archive-test --filepath archive.aiida

```

You can then just use that profile as normal (read-only of course). The sqlite database is cached (extracted to temporary file on disk) but only as long as the storage is kept alive. So this is not quite yet maintained between profile loads. We would have to look how to graciously implement this.

> [@giovannipizzi](#):
>
> For context, this is to be able to serve the rest api (eg for Materials Cloud) in read only mode directly from an .aiida file, without having to import everything in a new profile, thus making deployment much easier.

Have you benchmarked the performance for the sqlite backend though? I imagine that there may be a non-negligible difference between SQLite and PostgreSQL here. In which case it makes more sense to take the little extra effort to create a `core.psql_dos` profile and import the archive.

I see the `core.sqlite_zip` mounting more as a convenience for users in order to be able to quickly explore an archive without having to import it. Of course, we will have to document this and probably come up with a shortcut command that is easier to use.

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [June 7, 2024, 2:43pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/8 "2024-06-07T14:43:10Z")

</div>

> I meant the repository mainly.

Right, but the database contents itself are also not going to be negligible so you need to take this into account. But if the hypothesis is correct of the `TMP_DIR` running out of space, it is actually the database size that determines the problem as that is copied to the temp dir, and _not_ the file repository, whose contents are streamed directly to the target archive.

> Size of AIIDA path that keeps all the process id’s

😱 2.2 gigs in PID files. Dear lord. I think you should definitely think of running `verdi storage maintain` before creating an archive. This only will this potentially speed up the archiving (and backups) it will also clean these stale PID files.

Finally, to support the hypothesis that the tmp dir size may be the bottleneck, could you please report the output of

```auto
df -h $(python -c "import tempfile;print(tempfile.gettempdir())")

```

---

<div class="post-metadata">

**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\
**Post date:** [June 7, 2024, 4:09pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/9 "2024-06-07T16:09:13Z")

</div>

Hi @sphuber indeed the plan is to benchmark performance. I have the feeling that it will not be much slower, sqlite can in some cases be even faster than psql, but we’ll have to see the results

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\
**Post date:** [June 7, 2026, 4:09pm UTC](https://aiida.discourse.group/t/verdi-archive-runs-out-of-space-even-though-it-is-available-maybe-chunking-does-not-work/409/10 "2026-06-07T16:09:47Z")

</div>

This topic was automatically closed 730 days after the last reply. New replies are no longer allowed.