I’ve been working on creating a Docker image that has an AiiDA computer pre-set up and configured. My current base is aiidateam/aiida-core-with-services:sha-56f9f6c, which, at the time of writing, is the latest (or almost the latest) build available.
From what I’ve gathered, many user setup operations in AiiDA seem to occur during the CMD stage (at runtime) rather than during the RUN stage (which, a bit confusingly, happens at build time). I’ve been trying to get the computer and code configured during the image build itself, so it’s ready to use right out of the box.
I believe that having such configurations and installations done during the build phase would be beneficial for many, especially when there’s a need to distribute customized AiiDA setups tailored for specific use cases.
Questions:
Is there a recommended way to set up and configure an AiiDA computer within a Docker image during its build?
Has anyone successfully achieved this and can share their approach?
Looking into the future, do you think there’s potential value in providing mechanisms (or official images) to ship pre-configured AiiDA installations for certain workflows or use-cases?
Any insights or feedback would be highly appreciated!
Briefly, things that touch AiiDA DB need to happen at runtime. I am not familiar with this particular image, but generally you can prepare a bash script that automatically sets up the computers / code upon initial container startup. The path where you should copy the script will depend on the specific container setup. Note that the script should be idempotent, i.e. when you stop the container and start it again, the script should recognise that the computers already exist and exit early.
For inspiration, you can look how I did it in my AiiDAlab based image.
My current base is aiidateam/aiida-core-with-services:sha-56f9f6c , which, at the time of writing, is the latest (or almost the latest) build available.
This image will have a tag after we release aiida-core==2.5.0, it will then have the tag latest which always points to the stable released version of aiida-core. Now there is aiidateam/aiida-core-with-services:edge which points to the code base of main branch of the aiida-core. It is okay if you use sha-xxx tag which is associated with a commit in the main branch and, therefore, is more static than edge. But please note that you can update your Dockerfile and use the latest or version number when we make the release.
Is there a recommended way to set up and configure an AiiDA computer within a Docker image during its build?
It is not recommended to configure the computer during the build phase. The reason is mainly that it makes more sense the computer/code is set after the aiida profile is ready, which happened during the “runtime”.
Here is the lifecycle of the container and the order of services running when the first time the container is running (aiida-core-with-services as an example):
When the container just started no services are running. But the image has aiida-core package and PostgreSQL and RMQ installed. This is everything that happened in the Dockerfile, after every other thing is started/running as “services” by using the s6-overlay which is a services management dedicated to the container.
PostgreSQL DB will be initialized and start the service.
RabbitMQ services will start.
After these two services are ready if the AiiDA profile is not detected in the user’s home the profile will be created.
The decision was made from the experience we got from AiiDAlab image that the user’s home is mounted to the persistent resource so the data would be lost if the container is wiped. It thus requires checking that the mounted storage has the AiiDA profile and will create if not.
Looking into the future, do you think there’s potential value in providing mechanisms (or official images) to ship pre-configured AiiDA installations for certain workflows or use cases?
Thanks for bringing this up. Sure it is.
Although the computer/code setup has to happen after the profile is ready, the installation of plugins from pip is definitely feasible in the build phase. If you use the aiidateam/aiida-core-with-services as the base image in your Dockerfile and install your workflow/plugins from pip after the container starts the plugin will be ready.
We also have a plan to ship for example the aiida-quantumespresso image that has a plugin that can be used out-of-box, and also consider to have it in the plugin cookie-cutter.
Has anyone successfully achieved this and can share their approach?
I only had a very limited number of hours to allocate to the particular project this was associated with. In our use case we just resorted to having a bash script inside the docker that sets up the code/computer once manually. It might be nice to have something (perhaps it is impossible) like a first_time_start.sh script that only activates the first time the docker is initialized. This would allow, for example, users creating a custom plugin to easily distribute a docker instance that contains their plugin.
Thanks @Daniel_Marchand for posting the conversation. I updated my answer with a new feature to make the support easier, it is now merged and released with aiida-core-with-services:2.4.1 (this release has a bug for verdi shell autocomplete caused by jedi dependencies, please downgrade to jedi<0.19 if using verdi shell).
Not sure I understand what you mean, the method I provide is to run the first time the docker is initialized, it will add a lock-file in the user home and the second time container starts with persistent volume mounted if it is detected it will skip.
# If lock file exists, then we have already run this script
if [ -f ${HOME}/.lock_initial_code_setup ]; then
exit 0
else
touch ${HOME}/.lock_initial_code_setup
fi
