# Use a common directory for a number of tasks **URL:** \ **Category:** General Usage\ **Created:** [February 10, 2024, 6:40am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255 "2024-02-10T06:40:19Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 10, 2024, 6:40am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/1 "2024-02-10T06:40:19Z")
Aiida uses a new working directory for each task to my understanding. For e.g. reason of I/O efficiency, I want to have multiple tasks in a work chain to put intermediate files in a common directory. Is there an intended way of accomplishing this? ---
**Author:** ![mbercx](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/mbercx/32/345_2.png) [@mbercx](https://aiida.discourse.group/u/mbercx)\ **Post date:** [February 10, 2024, 12:18pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/2 "2024-02-10T12:18:57Z")
Hi @cbehren, 👋 If I/O is your main concern, perhaps you should try to use symlinking. Let me write down some ideas below, let me know if I’m in the ballpark. ### Fast/Hacky approach: the `prepend_text` The fastest way I can think of to achieve a restart from intermediate files from multiple calculations is to add a bunch of `ln` lines to the `metadata.options.prepend_text` of the follow-up `CalcJob` you are trying to run. For example: ```auto prepend_lines = [] for calcjob in (cj1, cj2): prepend_lines.extend( [f"ln -s {Path(calcjob.get_remote_workdir()) / file} {file}" for file in ('file1', 'file2')] ) inputs['metadata']['options']['prepend_text'] = '\n'.join(prepend_lines) ``` In this example `file1` and `file2` will be symlinked from the previously run `cj1` and `cj2`. Of course, this isn’t respecting the provenance very much, i.e. you wouldn’t be able to see that these files came from the previous calculations. ### Using a dynamical input namespace One idea would be for the follow-up calculation to have a dynamical input namespace where you can add multiple remote folders from the previous calculations. These could then be added to the `remote_symlink_list` of the `CalcInfo` instance returned at the end of the `prepare_for_submission` of the final `CalcJob`. This feature unfortunately doesn’t seem to be documented, but it works exactly like the `remote_copy_list`: > **[Usage — AiiDA 2.7.1.post0 documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/calculations/usage.html#remote-copy-list)** In this case the `remote_folder` outputs of the previous calculations would be nicely linked as inputs of the final one, and other AiiDA developers will nod in approval. This one is a bit trickier to implement though, let me know if you need a hand. * * * These two ideas don’t place the intermediate files in a common directory, but I’m not sure that’s necessary for your use case? Also note that using symlinking can have pitfalls in case the final calculation changes the file contents. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [February 10, 2024, 1:41pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/3 "2024-02-10T13:41:21Z")
I am not a 100% sure what you mean with I/O efficiency in this context. There is current no way in the API to control the working directory of calculations. They will always be generated based on the UUID of the node, and so will be unique for each calculation. This is intentional though. If a task needs to operate on the output files of a previous file, then that really needs to be put in as an input to capture the provenance. Files can be stored (roughly) in three ways in AiiDA: - `SinglefileData`: a node that contains a single file - `FolderData`: a node that contains any number of files (optionally nested in directories) - `RemoteData`: A “symlink” to a directory on a remote computer. Here AiiDA _just_ stores the computer and the directory, it doesn’t actually store which files it contains nor their content. These nodes can be used as inputs to `CalcJob`s to make sure the files are available for input to the calculation. AiiDA supports using symlinking files from `RemoteData`, which prevents having to duplicate the files from the working directory of one calculation to the next. `aiida-shell` makes this easy once again. Please have a look at this section in the documentation: [How-to guides — aiida-shell 0.6.0 documentation](https://aiida-shell.readthedocs.io/en/latest/howto.html#running-a-shell-command-with-remote-data) It shows how the working directory of a completed calculation can be passed to the next and you can tell it to use symlinks in order to prevent unnecessary copying. As a quick example, if you have two tasks, of which the second needs the output files of the first, you could do something like the following: ```auto from aiida.engine import workfunction from aiida_shell import launch_shell_job @workfunction def workflow(): results_01, node_01 = launch_shell_job( 'command_1', ) results_02, node_02 = launch-shell_job( 'command_2', nodes={ 'files_command_1': node_01.outputs.remote_folder }, metadata={'options': {'use_symlinks': True}} ) ``` ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 13, 2024, 1:17pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/4 "2024-02-13T13:17:11Z")
Dear @mbercx and @sphuber, Thanks for the input! The `RemoteData` approach or the `remote_symlink_list` sound like what I need, I will try that first. By I/O efficiency, I really just meant that I want to avoid to copy potentially large files or directories from job to job. Originally, I thought it would be a good idea to really have a shared directory, but I do see that I can arrive at the same thing by symlinking the previous jobs. ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 13, 2024, 5:30pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/5 "2024-02-13T17:30:57Z")
Dear @sphuber, I tried to implement your variant, but I get the problem that apparently, the sym link options tries to link the dir of task 1 directly into the dir of task 2: `FileExistsError: [Errno 17] File exists: '/home/xyz/.aiida_run/30/8f/eb42-4341-4b3d-9e13-752f1c9b1eae/_aiidasubmit.sh' -> '/home/xyz/.aiida_run/f5/99/00a2-3deb-4a02-87b4-bb3e3f53dd2f/./_aiidasubmit.sh'` In the latter directory, all the files from the previous jobs are directly symlinked, when it tries to also copy the aiidasubmit, it obviously crashes - I would have expected the files to be linked into a subfolder of the second job’s output dir. Am I missing something? ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [February 13, 2024, 8:00pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/6 "2024-02-13T20:00:18Z")
No, your expectation is correct, this is just a bug in `aiida-shell` that was recently uncovered. I only added the symlink functionality recently and didn’t think of this. Here is the issue: [Bug: Default submit script is overridden by `RemoteData` · Issue #58 · sphuber/aiida-shell · GitHub](https://github.com/sphuber/aiida-shell/issues/58) I have a fix for this is in this PR: [`ShellJob`: Detect and prevent filename clashes by sphuber · Pull Request #70 · sphuber/aiida-shell · GitHub](https://github.com/sphuber/aiida-shell/pull/70) But that requires a change in `aiida-core` first, see this PR: [`CalcJob`: Allow to define order of copying of input files by sphuber · Pull Request #6285 · aiidateam/aiida-core · GitHub](https://github.com/aiidateam/aiida-core/pull/6285) So if you check out the two branches of those repositories and install them, it should resolve the problem: ```bash git clone https://github.com/sphuber/aiida-shell cd aiida-shell git checkout fix/068/input-output-filename-overlap pip install -e . cd ../ git clone https://github.com/sphuber/aiida-core cd aiida-core git checkout feature/6012/calcjob-file-copy-order pip install -e . ``` It would be great if you could give that a try and let us know if it works. With the feedback, I could hopefully accelerate the review process and get the fixes merged and released. Thanks! ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 13, 2024, 9:52pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/7 "2024-02-13T21:52:02Z")
I applied both fixes, but I still get the same error message. Below the expanded message: Error: Exception whilst using transport: Traceback (most recent call last): File “/home/xyz/test\_aiida/aiida-core/src/aiida/engine/transports.py”, line 105, in request\_transport yield transport\_request.future File “/home/xyz/test\_aiida/aiida-core/src/aiida/engine/processes/calcjobs/tasks.py”, line 94, in do\_upload remote\_folder = execmanager.upload\_calculation(node, transport, calc\_info, folder) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/home/xyz/test\_aiida/aiida-core/src/aiida/engine/daemon/execmanager.py”, line 217, in upload\_calculation \_copy\_remote\_files(logger, node, computer, transport, remote\_copy\_list, remote\_symlink\_list) File “/home/xyz/test\_aiida/aiida-core/src/aiida/engine/daemon/execmanager.py”, line 316, in \_copy\_remote\_files transport.symlink(remote\_abs\_path, dest\_rel\_path) File “/home/xyz/test\_aiida/aiida-core/src/aiida/transports/plugins/local.py”, line 835, in symlink os.symlink(os.path.join(this\_file), os.path.join(self.curdir, remotedestination, this\_remote\_dest)) FileExistsError: [Errno 17] File exists: ‘/home/xyz/.aiida\_run/e6/6e/dd03-c81d-4897-bcef-edd79ee0a54f/\_aiidasubmit.sh’ → ‘/home/xyz/.aiida\_run/87/96/e341-8453-441b-8939-398e6958e5bc/./\_aiidasubmit.sh’ This is `ls -ahl` on the relevant dir. Note the symbolic links in place. It copied over e.g. the status from the previous job: drwxr-xr-x 3 xyz xyz 4,0K 13. Feb 22:40 . drwxr-xr-x 3 xyz xyz 4,0K 13. Feb 22:40 … drwxr-x— 2 xyz xyz 4,0K 13. Feb 22:40 .aiida -rw-r–r-- 1 xyz xyz 446 13. Feb 22:40 \_aiidasubmit.sh lrwxrwxrwx 1 xyz xyz 86 13. Feb 22:40 \_scheduler-stderr.txt → /home/xyz/.aiida\_run/e6/6e/dd03-c81d-4897-bcef-edd79ee0a54f/\_scheduler-stderr.txt lrwxrwxrwx 1 xyz xyz 71 13. Feb 22:40 status → /home/xyz/.aiida\_run/e6/6e/dd03-c81d-4897-bcef-edd79ee0a54f/status lrwxrwxrwx 1 xyz xyz 71 13. Feb 22:40 stderr → /home/xyz/.aiida\_run/e6/6e/dd03-c81d-4897-bcef-edd79ee0a54f/stderr lrwxrwxrwx 1 xyz xyz 71 13. Feb 22:40 stdout → /home/xyz/.aiida\_run/e6/6e/dd03-c81d-4897-bcef-edd79ee0a54f/stdout ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [February 13, 2024, 10:32pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/8 "2024-02-13T22:32:01Z")
Thanks for trying. I accidentally gave you the wrong branch. That only contains part of the fix. The branch `feature/058/remote-folder-symlink` should fix this problem. However, I just realized that this is still not an ideal solution. Although it should now properly symlink all files from the first calculation, it essentially results in the original output files being overwritten. This might not _immediately_ be a problem, since this files are just in the scratch files and the originals were saved permanently in AiiDA’s repository and they won’t be affected. But maybe this could be a problem further down the line, if the original `RemoteFolder` is reused once again for a follow-up command. That is the nature of symlinks though. Maybe I have to rethink the interface of the symlink functionality. Instead of just having a boolean flag that symlinks everything, the user would have to specify manually which files exactly should be symlinked. Do you have any thoughts on what a better interface could be? ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 14, 2024, 5:38am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/9 "2024-02-14T05:38:30Z")
Hey, the fix did something: I did not get an error message, but the result looks different from what I expected. The directory of the second task looks like this: drwxr-x— 2 xyz xyz 4.0K Feb 14 06:00 .aiida lrwxrwxrwx 1 xyz xyz 80 Feb 14 06:00 \_aiidasubmit.sh → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/\_aiidasubmit.sh lrwxrwxrwx 1 xyz xyz 86 Feb 14 06:00 \_scheduler-stderr.txt → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/\_scheduler-stderr.txt lrwxrwxrwx 1 xyz xyz 86 Feb 14 06:00 \_scheduler-stdout.txt → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/\_scheduler-stdout.txt lrwxrwxrwx 1 xyz xyz 71 Feb 14 06:00 status → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/status lrwxrwxrwx 1 xyz xyz 71 Feb 14 06:00 stderr → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/stderr lrwxrwxrwx 1 xyz xyz 71 Feb 14 06:00 stdout → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/stdout lrwxrwxrwx 1 xyz xyz 76 Feb 14 06:00 myfiles → /home/xyz/.aiida\_run/4e/85/72a7-065f-4486-8632-8f1c72f2e0f0/myfiles So it now has linked all files to the previous job, including the \_aiidasubmit.sh file, and it has overwritten the \_aiidasubmit.sh of the previous job. What it has not linked is the .venv folder that is a result of task 1, but this might be intentional (note the dot in the beginng of the name - I guess you ignore those, which is fine. It shows up when I replace .venv by venv, removing the dot). The dir myfiles was generated from the second job. What I intended was something like \/ myfiles/ job1/ job1/venv/ where job1/ is a symbolic link to the working dir of job1. For me, the ideal way of handling this would be to be able to specify the directory I want to like to, and a name for the link. Maybe it is helpful to explain what I want to do. The pipeline I am working on contains about 10 steps, only one of them is computationally heavy. In the process, ~6 git repos have to cloned, some of them contain data, other contain scripts that are used. I need to install them in a clean virtual environment, but for performance reasons, I would like to install all of them in the same virtual environment - so, I want to basically carry the virtual env from task to task without copying. The same goes for the cloned git repos. I should add that I also would not store the venv or the git repos in the database - the git repos should be characterized by their commit hashes anyway, and I want to store those in the database only. However, going through your fix I just realized that aiida-shell will automatically replace the shell command with its absolute path in the startup environment of the computer - which means that I will not be able to use aiida-shell to utilize said virtual environment. I would need to tweak this here to get around this issue, right? I could add an option to launch\_shell\_job like `defer_resolution=True` to jump over that `which {command}`. Then, I could put something like `python` as command, and the \_aiida\_submit.sh would contain just that command with no absolute path. > ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [February 14, 2024, 7:38am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/10 "2024-02-14T07:38:34Z")
> [@cbehren](#): > > Maybe it is helpful to explain what I want to do. The pipeline I am working on contains about 10 steps, only one of them is computationally heavy. In the process, ~6 git repos have to cloned, some of them contain data, other contain scripts that are used. I need to install them in a clean virtual environment, but for performance reasons, I would like to install all of them in the same virtual environment - so, I want to basically carry the virtual env from task to task without copying. The same goes for the cloned git repos. I should add that I also would not store the venv or the git repos in the database - the git repos should be characterized by their commit hashes anyway, and I want to store those in the database only. That is indeed useful. I am wondering if you actually shouldn’t store the virtual environment if reproducibility is crucial. Even if you install from some repo, and the commit hash is stored as an input, if it has any dependencies that are not all pinned to a particular version as well, running the same install in the future might actually produce a different env. What I am thinking off is to first run a step that creates the env and then stores that in AiiDA using a `FolderData`. You can then use this as an input to each next step and it will be copied to the working directory. This will guarantee perfect reproducibility, as you can always recover the exact virtual env in the future in which a step was run. The downside is that you would be storing quite a bit of data and copying it for each step to the scratch space. Defining manually the working directory for a `CalcJob` is currently not supported in `aiida-core`. The working directory is generated automatically based on the work dir of the computer and the UUID of the `CalcJobNode`, see here: > The reason for not allowing the same working directory is based at least on the same observation we made here, where certain default input/output files can overwrite one another. > I would need to tweak this here to get around this issue, right? I could add an option to launch\_shell\_job like `defer_resolution=True` to jump over that `which {command}`. Then, I could put something like `python` as command, and the \_aiida\_submit.sh would contain just that command with no absolute path. This is something we can definitely add to `aiida-shell`. I think that this would be necessary to support running commands inside a virtual environment. I will work on a PR today ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 14, 2024, 7:49am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/11 "2024-02-14T07:49:33Z")
I will come back to this later, just a quick remark on this part: > [@sphuber](#): > > This is something we can definitely add to `aiida-shell`. I think that this would be necessary to support running commands inside a virtual environment. I will work on a PR today This should work - I just commented out the part in prepare\_code right now, and passed on `command` instead of `executable`. The only other thing is that somewhere deeper down, e.g. `python` is still going to replaced by whatever `which` gives (I did not look up where exactly!). To make sure `python` is run from the virtual env, one has to provide the command as `venv/bin/python`. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [February 14, 2024, 8:01am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/12 "2024-02-14T08:01:27Z")
> [@cbehren](#): > > To make sure `python` is run from the virtual env, one has to provide the command as `venv/bin/python`. An alternative is just to specify `python` and then add the command ```auto source venv/bin/activate ``` to the `metadata.options.prepend_text` input. This command will be executed before the main executable is called, so it first loads the env, and then `python` will point to the Python version installed in the virtual env. [https://aiida-shell.readthedocs.io/en/latest/howto.html#customizing-run-environment](https://aiida-shell.readthedocs.io/en/latest/howto.html#customizing-run-environment) Edit: I created a PR with an option to skip resolving the command: [`launch_shell_job`: Add option to keep skip resolving of `command` by sphuber · Pull Request #73 · sphuber/aiida-shell · GitHub](https://github.com/sphuber/aiida-shell/pull/73) ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 14, 2024, 1:58pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/13 "2024-02-14T13:58:28Z")
> [@sphuber](#): > > That is indeed useful. I am wondering if you actually shouldn’t store the virtual environment if reproducibility is crucial. Even if you install from some repo, and the commit hash is stored as an input, if it has any dependencies that are not all pinned to a particular version as well, running the same install in the future might actually produce a different env. You are right - this would be better, although it would require to store quote a large amount of data. I understand that the working directories are not changeable right now. My naive assumption was that there is some way to point to a shared folder within the framework of Aiida (outside of the working directories) but I see this might be difficult. ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 14, 2024, 2:00pm UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/14 "2024-02-14T14:00:21Z")
> [@sphuber](#): > > An alternative is just to specify `python` and then add the command > > ```auto > source venv/bin/activate > > ``` > > to the `metadata.options.prepend_text` input. This command will be executed before the main executable is called, so it first loads the env, and then `python` will point to the Python version installed in the virtual env. > > [How-to guides — aiida-shell 0.6.0 documentation](https://aiida-shell.readthedocs.io/en/latest/howto.html#customizing-run-environment) Sorry, `source` does not work in my shell (but as it turned out, “.” instead of it works), so I missed that until now. Thanks. > [@sphuber](#): > > Edit: I created a PR with an option to skip resolving the command: [`launch_shell_job`: Add option to keep skip resolving of `command` by sphuber · Pull Request #73 · sphuber/aiida-shell · GitHub](https://github.com/sphuber/aiida-shell/pull/73) Looks great, I will take a look at it. ---
**Author:** ![cbehren](https://avatars.discourse-cdn.com/v4/letter/c/e9a140/32.png) [@cbehren](https://aiida.discourse.group/u/cbehren)\ **Post date:** [February 19, 2024, 9:06am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/15 "2024-02-19T09:06:31Z")
> [@sphuber](#): > > Edit: I created a PR with an option to skip resolving the command: [`launch_shell_job`: Add option to keep skip resolving of `command` by sphuber · Pull Request #73 · sphuber/aiida-shell · GitHub](https://github.com/sphuber/aiida-shell/pull/73) This works fine for me! ---
**Author:** ![system](https://global.discourse-cdn.com/free1/uploads/aiida/original/1X/c417197d3750949351575d03ed03f5e2ad32e649.png) [@system](https://aiida.discourse.group/u/system)\ **Post date:** [February 18, 2026, 9:07am UTC](https://aiida.discourse.group/t/use-a-common-directory-for-a-number-of-tasks/255/16 "2026-02-18T09:07:05Z")
This topic was automatically closed 730 days after the last reply. New replies are no longer allowed.