Importing QE files from my folders

New to AiiDA
1

Dear community. I am really new using Aiida. I already installed the VM Quantum Mobile+Aiida.

I was using Quantum Espresso (v.6.7 and v.7.2) with Burai, NanoLabo, and Exabyte. I did many QE runs checking dft functionals, now I would like to use all my outputs with aiida. How can I upload on aiida all my files for post-processing? I don’t know if it is possible to upload the data to my Nomada account where I store some calculations then I can obtain the link to this website and use it on aiida. Thanks in advance.

2

Welcome to the community! :wave:

I’m not very familiar with the tools you mentioned, I’m afraid. However, we have already been working on making it possible to import already run calculations, see this how-to in the documentation.

Starting from there is going to be challenging, however. Specifically for Quantum ESPRESSO, there is a open PR from @sphuber to add the importer for pw.x calculations:

Maybe this is a good time to wrap up this work and then you can test if it works for your use case.

1 Like
3

Dear Marnik, thank you so much.

I followed these lines in “verdi shell”:

In [1]: import io
…: node = Node()
…: node.put_object_from_tree(‘/home/max/Documents/MY_FOLDER’)
…: node.list_object_names()
Out[1]:
[‘file1.dat’,
‘file1.out’,
‘file2.out’]

In [2]: node.get_object_content(‘file1.dat’)
Out[2]: HERE IS SHOWN ALL THE DATA CONTAINED IN “file1.dat”

I hope I will try to work from this with my data after checking your recommendations.

Regards

4

Hi Deivi,

What you showed won’t really work, because to persist the changes to the database, you need to call .store() on the node. Otherwise, when you close the shell, the changes are lost. However, a Node instance cannot be stored. You could fix this by changing it to a Data node, which is storable.

Stil, I am not sure if this is the best approach because, sure your data will be stored in AiiDA’s database, it will not be stored at all as a pw.x calculation would be stored if run through AiiDA directly.

This use-case is exactly why we came up with the importer system. As Marnik mentioned, I have already a working version in a branch on aiida-quantumespresso. I have just updated the branch. Here is how you can install it:

git clone https://github.com/aiidateam/aiida-quantumespresso
cd aiida-quantumespresso
git checkout feature/809/calc-job-importer-pw
pip install -e .

Then you need to configure your local machine as a Computer in AiiDA if you haven’t done already:

verdi computer setup -L localhost -H localhost -T core.local -S core.direct -n
verdi computer configure core.local localhost --safe-interval 0 -n

Then you can open a verdi shell and do the following:

import pathlib
from aiida_quantumespresso.calculations.importers.pw import PwCalculationImporter
from aiida_quantumespresso.calculations.pw import PwCalculation

localhost = orm.load_computer('localhost')
filepath_remote = pathlib.Path('/home/max/Documents/MY_FOLDER')
remote_data = RemoteData(str(filepath_remote), computer=localhost)
input_file_name = 'aiida.in'
pseudo_folder_path = filepath_remote / 'pseudo'
inputs = PwCalculationImporter().parse_remote_data(
    remote_data, input_file_name, pseudo_folder_path=str(pseudo_folder_path)
)
results, node = run.get_node(PwCalculation, **inputs)

You should now see the calculation in the output of verdi process list -a.

Please give this a go and let us know if you run into trouble. If there are bugs, I am happy to update the PR and we can use your feedback to finalize it and release it with an official version of aiida-quantumespresso.

2 Likes
6

Dear Dr Hubert,

Thank you so much for supporting me last week. I updated my VM and everything was wrong (I don’t recommended it), I installed a new VM. Now, I followed the procedure until the localhost configuration. I open the verdi shell and I am stuck in this part:

(base) max@qmobile:~/aiida-quantumespresso$ verdi shell
Python 3.10.9 | packaged by conda-forge | (main, Feb 2 2023, 20:20:04) [GCC 11.3.0]
Type ‘copyright’, ‘credits’ or ‘license’ for more information
IPython 8.15.0 – An enhanced Interactive Python. Type ‘?’ for help.

In [1]: import pathlib
…: from aiida_quantumespresso.calculations.importers.pw import PwCalculationImporter
…: from aiida_quantumespresso.calculations.pw import PwCalculation

ImportError Traceback (most recent call last)
Cell In[1], line 2
1 import pathlib
----> 2 from aiida_quantumespresso.calculations.importers.pw import PwCalculationImporter
3 from aiida_quantumespresso.calculations.pw import PwCalculation

ImportError: cannot import name ‘PwCalculationImporter’ from ‘aiida_quantumespresso.calculations.importers.pw’ (/home/max/aiida-quantumespresso/src/aiida_quantumespresso/calculations/importers/pw.py)

7

Did you clone the aiida-quantumespresso repository and checkout the branch with the importer?

git clone https://github.com/aiidateam/aiida-quantumespresso
cd aiida-quantumespresso
git checkout feature/809/calc-job-importer-pw
pip install -e .
8

When I tried to clone the aiida-quantumespresso repository this was the message:

(base) max@qmobile:~ workon aiida (aiida) max@qmobile:~ git clone GitHub - aiidateam/aiida-quantumespresso: The official AiiDA plugin for Quantum ESPRESSO
fatal: destination path ‘aiida-quantumespresso’ already exists and is not an empty directory.
(aiida) max@qmobile:~ cd aiida-quantumespresso (aiida) max@qmobile:~/aiida-quantumespresso git checkout feature/809/calc-job-importer-pw
Already on ‘feature/809/calc-job-importer-pw’
Your branch is up to date with ‘origin/feature/809/calc-job-importer-pw’.
(aiida) max@qmobile:~/aiida-quantumespresso$ pip install -e .
Obtaining file:///home/max/aiida-quantumespresso
Installing build dependencies … done

From here all Requirements are already satisfied…

9

Using checked this:

(aiida) max@qmobile:~/aiida-quantumespresso/src/aiida_quantumespresso/calculations/importers$ cat pw.py

-- coding: utf-8 --

“”“:class:aiida.engine.processes.calcjobs.importer.CalcJobImporter implementation for PwCalculation.”“”
from future import annotations

import pathlib
import tempfile

from aiida.engine import CalcJobImporter
from aiida.orm import Code, Node, RemoteData

class PwCalculationImmigrator(CalcJobImporter):
“”":class:aiida.engine.processes.calcjobs.importer.CalcJobImporter implementation for PwCalculation.

This class allows to import a completed ``pw.x`` calculation that was executed without AiiDA, into an AiiDA profile.
"""

I guess ( I am not sure) this line:

from aiida_quantumespresso.calculations.importers.pw import PwCalculationImporter

Could be…

from aiida_quantumespresso.calculations.importers.pw import PwCalculationImmigrator

10

It seems you have an old version of the branch. Please do the following in the aiida-quantumespresso directory

git fetch --all -p
git checkout feature/809/calc-job-importer-pw
git reset --hard origin/feature/809/calc-job-importer-pw

Then you should have the correct latest version of the branch and you can try again.

11

Thanks Dr. Huber,

Now, I have the latest version of the branch but I am stuck with this issue:

In [1]: import pathlib
…: from aiida_quantumespresso.calculations.importers.pw import PwCalculatio
…: nImporter
…: from aiida_quantumespresso.calculations.pw import PwCalculation

In [2]: localhost = orm.load_computer(‘localhost’)

NameError Traceback (most recent call last)
in <cell line: 1>()
----> 1 localhost = orm.load_computer(‘localhost’)

NameError: name ‘orm’ is not defined
.
.

12

a simple

from aiida import orm

will fix that

13

I guess that I almost did it!, It is still an error. I would like to ask if it is mandatory to import the files one by one, or is it enough to introduce the inputs that are connected that produce the respective outputs?

In [1]: import pathlib
…: from aiida_quantumespresso.calculations.importers.pw import PwCalculationImporter
…: from aiida_quantumespresso.calculations.pw import PwCalculation
…: from aiida import orm
…: from aiida.orm import RemoteData
In [2]: localhost = orm.load_computer(‘localhost’)
…: filepath_remote = pathlib.Path(‘/home/max/Documents/CC_mGGA1’)
…: remote_data = RemoteData(str(filepath_remote), computer=localhost)
…: input_file_name = ‘pw_scf.in’
…: pseudo_folder_path = filepath_remote / ‘/home/max/Documents/CC_mGGA1/pseudo’
In [3]: inputs = PwCalculationImporter().parse_remote_data(
…: remote_data, input_file_name, pseudo_folder_path=str(pseudo_folder_path))

In [4]: results, node = run.get_node(PwCalculation, **inputs)

NameError Traceback (most recent call last)
in <cell line: 1>()
----> 1 results, node = run.get_node(PwCalculation, **inputs)

NameError: name ‘run’ is not defined

14

That you can import as follows

from aiida.engine import run
15

I tried setting this:

from aiida.engine import run, run_get_node, runner

In [12]: results, node = run.get_node(PwCalculation, **inputs)

AttributeError Traceback (most recent call last)
Cell In[12], line 1
----> 1 results, node = run.get_node(PwCalculation, **inputs)

File ~/.conda/lib/python3.10/site-packages/aiida/engine/launch.py:60, in run_get_node(process, *args, **inputs)
57 else:
58 runner = manager.get_manager().get_runner()
—> 60 return runner.run_get_node(process, *args, **inputs)

File ~/.conda/lib/python3.10/site-packages/aiida/engine/runners.py:277, in Runner.run_get_node(self, process, *args, **inputs)
268 def run_get_node(self, process: TYPE_RUN_PROCESS, *args: Any, **inputs: Any) → ResultAndNode:
269 “”"
270 Run the process with the supplied inputs in this runner that will block until the process is completed.
271 The return value will be the results of the completed process
(…)
275 :return: tuple of the outputs of the process and the calculation node
276 “”"
→ 277 result, node = self._run(process, *args, **inputs)
278 return ResultAndNode(result, node)

…continue…

but is still a problem, I also checked: verdi computer show localhost and everything is ok

16

Could you please show the end of the stack trace of the exception? That is where the useful information is at. And maybe, instead of going step by step, an easier approach is if you could send me one example of your output directories. I can then test it locally and make it working and give you complete instructions. Think this will take a while otherwise. Maybe you could make a zip archive (or tar-ball) of one of your example output directories and upload them somewhere so I can download it.

1 Like
17

Thank you so much in advance Dr. Huber.
Attached is the link to open the zip file containing a txt file with all the code run on my virtual machine, and the folder with the inputs, pseudo folder, and outputs created with QE v.7.2.
Kind of regards

https://drive.google.com/file/d/1mu4ekZpD0yg00irPmpJUbOWLWGcxytfF/view?usp=drive_link

18

Thanks. I had a look and made it work but it required a couple of fixes. Also, I need to note that currently the import is only implemented for pw.x calculations. This means your bands.x and projwfc.x calculations cannot yet be imported in this manner. This would first have to be implemented, but I don’t have the bandwidth for this in the near future. You could open a feature request on aiida-quantumespresso or try to implement it yourself.

To import the pw.x calculations, do the following:

  • Install qe-tools v2.1.0 by running pip install qe-tools==2.1.0
  • Update the aiida-quantumespresso branch and pull it. Go to the aiida-quantumespresso repo and do git pull
  • In the folder with your output files run:
    mkdir scf
mv pw_scf.in scf/aiida.in
mv pw_scf.out scf/aiida.out
    You do the same for the pw_nscf and pw_bands files, putting them in a nscf and bands folder, respectively
  • Copy the following script to the file run_importer.py in the folder with all your output files
    #!/usr/bin/env python
import pathlib
import click
from aiida.cmdline.utils import decorators


@click.command()
@click.argument('directory', type=click.Path(file_okay=False, dir_okay=True, resolve_path=True, path_type=pathlib.Path))
@decorators.with_dbenv()
def main(directory):
    from aiida import engine, orm
    from aiida_quantumespresso.calculations.importers.pw import PwCalculationImporter
    from aiida_quantumespresso.calculations.pw import PwCalculation

    localhost = orm.load_computer('localhost')
    inputs = PwCalculationImporter().parse_remote_data(
        remote_data=orm.RemoteData(str(directory), computer=localhost),
        input_file_name='aiida.in',
        pseudo_folder_path=str(directory.parent / 'pseudos'),
        metadata={'computer': localhost}
    )
    results, node = engine.run.get_node(PwCalculation, **inputs)
    print(node)

if __name__ == '__main__':
    main()
  • Now you run the script for each of the output folders, e.g. python run_importer.py scf

This should print some output that looks something like

(aiida-py311) sph@invader:CC_mGGA1$ ./run_importer.py scf
09/26/2023 09:52:25 PM <16162> aiida.orm.nodes.process.calculation.calcjob.CalcJobNode: [WARNING] could not parse scheduler output: the `_scheduler-stderr.txt` file is missing
09/26/2023 09:52:25 PM <16162> aiida.orm.nodes.process.calculation.calcjob.CalcJobNode: [WARNING] could not parse scheduler output: the `_scheduler-stdout.txt` file is missing
09/26/2023 09:52:25 PM <16162> aiida.parser.PwParser: [WARNING] key 'symmetries' is not present in raw output dictionary
09/26/2023 09:52:25 PM <16162> aiida.parser.PwParser: [WARNING] Warning: card &CELL ignored
09/26/2023 09:52:25 PM <16162> aiida.parser.PwParser: [WARNING] Warning: card / ignored
09/26/2023 09:52:25 PM <16162> aiida.parser.PwParser: [ERROR] The retrieved folder did not contain the required XML file.
09/26/2023 09:52:25 PM <16162> aiida.orm.nodes.process.calculation.calcjob.CalcJobNode: [WARNING] output parser returned exit code<303>: The retrieved folder did not contain the required XML file.
uuid: bb56501a-4f8b-4c64-8657-0a364e35567a (pk: 115789) (aiida.calculations:quantumespresso.pw)

The last line says that a new calculation has been created to represent the imported calculation. You can now inspect it like any other AiiDA calculation, for example with verdi process show 115789.

1 Like
19

Thanks a lot Dr. Huber, everything works perfectly now, I even did a vc-relax folder to migrate the optimization run and it was okay. My best wishes! Deivi

2 Likes