# Questions about \`with\_mpi\` **URL:** \ **Category:** General Usage\ **Created:** [October 18, 2023, 3:05pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128 "2023-10-18T15:05:04Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![t-reents](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/t-reents/32/82_2.png) [@t-reents](https://aiida.discourse.group/u/t-reents)\ **Post date:** [October 18, 2023, 3:05pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/1 "2023-10-18T15:05:04Z")
Hi, I have a couple of questions concerning the usage of `with_mpi` in `CalcJobs`. More precisely, I am referring to the Wannier90 and QE plugins, but it seems that this behavior is fundamental and I am not 100% sure whether it is intended like this. In case it is, I would be really interested in the line of reasoning. There are 3 different levels of specifying `withmpi`. In the QE plugin, `withmpi` is set to True per default but submitting a `PwCalculation` shows that it is submitted without the usage of MPI. A small example to reproduce the behavior: ```auto b = PwCalculation.get_builder() b.code = code b.structure = structure kpoints = KpointsData() kpoints.set_kpoints_mesh([4, 4, 4]) b.kpoints = kpoints b.metadata.options.resources = {"num_machines": 1, "num_mpiprocs_per_machine": 4} b.parameters = pw.parameters b.pseudos = pseudos submit(b) ``` In case one adds the line `b.metadata.options.withmpi = True`, MPI is used correctly. `withmpi` isn’t specified on the code level, so this isn’t an issue of overwriting the different levels. On the other hand, one doesn’t observe this behavior when using the `PwBaseWorkChain`, in that case, the builder correctly recognizes the default (True) value for `withmpi`. I followed the different parts of the source code, ending up in the plumpy package and my naive understanding is the following: The CalchJob looks for `withmpi` in the `raw_inputs` ([https://github.com/aiidateam/aiida-core/blob/ec64780c206cdb040eee740b17865e6f0ff81cd8/aiida/engine/processes/calcjobs/calcjob.py#L925C31-L925C42](https://github.com/aiidateam/aiida-core/blob/ec64780c206cdb040eee740b17865e6f0ff81cd8/aiida/engine/processes/calcjobs/calcjob.py#L925C31-L925C42)) and I would expect that this is populated by the default value which is specified in the QE plugin. If this would not be the case, and since `withmpi` is neither specified in the `CodeInfo` nor on the `code` level, all three levels would be None and the default of the aiida-core `CalcJob` implementation, which is False, would be used (this seems to be what is happening in the example). When one checks the `_inputs` method of the builder, one observes that `withmpi` isn’t returned, in case it isn’t explicitly specified. It seems that these inputs would be included in the `raw_inputs` and therefore it makes somehow sense why it works when a WorkChain is used. Please correct me, if I’m wrong. Although it seems understandable, why this happening wrt. the code, I’m not 100% sure whether this is the intended behavior. As a user, one would expect that the default value of the CalcJob is used. Thanks a lot in advance! ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [October 18, 2023, 3:37pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/2 "2023-10-18T15:37:22Z")
Can you please specify which versions of `aiida-core` and `aiida-quantumespresso` you are using? [The behavior of `with_mpi` changed in AiiDA v2.3](https://github.com/aiidateam/aiida-core/blob/main/CHANGELOG.md#controlling-usage-of-mpi). Its current behavior and design is described [here in the documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/calculations/usage.html#controlling-mpi). ---
**Author:** ![t-reents](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/t-reents/32/82_2.png) [@t-reents](https://aiida.discourse.group/u/t-reents)\ **Post date:** [October 18, 2023, 3:52pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/3 "2023-10-18T15:52:22Z")
Yes, we also checked the wiki. I’m using `aiida-core` v2.4.0 and `aiida-quantumespresso` v4.4.0. My colleague who initially experienced this behavior of `Wannier90` is also using `aiida-core` v2.4.0 and `aiida-wannier90` v2.1.0. So basically the latest versions. ---
**Author:** ![yakutovicha](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/yakutovicha/32/18_2.png) [@yakutovicha](https://aiida.discourse.group/u/yakutovicha)\ **Post date:** [December 8, 2023, 1:38pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/4 "2023-12-08T13:38:59Z")
It looks like there might be a bug in the current behaviour of the MPI setting. According to the [documentation](https://aiida.readthedocs.io/projects/aiida-core/en/latest/topics/calculations/usage.html#conflict-resolution) if the feature is not set in any of those places: ```auto The Code input The CalcJob implementation The metadata.options.withmpi input ``` Then the value is taken from the default of `metadata.options.withmpi`. After migrating AiiDA from 2.2 to 2.4 our simulations started to fail cause MPI was not enabled. One particular example is the [`Cp2kPdosWorkChain`](https://github.com/nanotech-empa/aiida-nanotech-empa/blob/master/aiida_nanotech_empa/workflows/cp2k/pdos_workchain.py). Internally, it uses an overlap plugin where the default of [`metadata.options.withmpi` is set to `True`](https://github.com/nanotech-empa/aiida-nanotech-empa/blob/master/aiida_nanotech_empa/plugins/overlap.py#L18). For your information, this is the output of `verdi code show`: ```auto PK 18904 UUID cd30b62f-9df2-412f-879b-414d591a984f Type core.code.installed Computer daint-hybrid-s1141 (daint.cscs.ch), pk: 2 Filepath executable /users/cpi/aiida-soft/cp2k-spm-tools/overlap_from_wfns.py Label k-overlap Description Overlap code from https://github.com/nanotech-empa/cp2k-spm-tools.git. Default calc job plugin nanotech_empa.overlap Use double quotes False With mpi Prepend text module load cray-python export PYTHONPATH=$PYTHONPATH:/users/keimre/soft/ase Append text ``` As you can see, the `With mpi` attribute is not set. We are a bit at a loss currently. Any help would be aprreciated. ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [December 8, 2023, 9:52pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/5 "2023-12-08T21:52:22Z")
I think I see where the bug is. It seems that the implementation does not actually respect the documentation and ignores the default of `metadata.options.withmpi` port. I have opened a PR that should fix the problem: [`CalcJob`: Fix MPI behavior if `withmpi` option default is True by sphuber · Pull Request #6218 · aiidateam/aiida-core · GitHub](https://github.com/aiidateam/aiida-core/pull/6218) Could you please give that branch a go? Thanks! ---
**Author:** ![t-reents](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/t-reents/32/82_2.png) [@t-reents](https://aiida.discourse.group/u/t-reents)\ **Post date:** [December 11, 2023, 9:02am UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/6 "2023-12-11T09:02:20Z")
Hi @sphuber, I just gave that branch a try and it solved the issue. Thanks a lot! ---
**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\ **Post date:** [December 11, 2023, 9:20am UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/7 "2023-12-11T09:20:15Z")
Great, thanks for testing and letting me know. I will make sure it gets merged and released with the upcoming v2.5 which should be out soon. ---
**Author:** ![yakutovicha](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/yakutovicha/32/18_2.png) [@yakutovicha](https://aiida.discourse.group/u/yakutovicha)\ **Post date:** [December 12, 2023, 6:20pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/8 "2023-12-12T18:20:00Z")
Thanks for taking care of that @sphuber. ---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 5:51pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/10 "2024-01-17T17:51:35Z")
---
**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\ **Post date:** [January 17, 2024, 5:53pm UTC](https://aiida.discourse.group/t/questions-about-with-mpi/128/11 "2024-01-17T17:53:38Z")
Just a note that this was also backported and released in v2.4.3 >