Convergence strategy in DielectricWorkChain

Hi @bastonero

My calculation is taking forever to converge (1e-10 threshold) in DielectricWorkChain part. It converges pretty fast in normal calculation without E field (normal scf calculation takes a minute to complete). Is there anything I can do to make it converge faster while E field is applied, in original overrides.yaml file? Below is my overrides.yaml file

  │   ├── PwBaseWorkChain<3429> Finished [0] [3:results]
        │   │   └── PwCalculation<3441> Finished [0]
        │   ├── PwBaseWorkChain<3431> Finished [0] [3:results]
        │   │   └── PwCalculation<3451> Finished [0]
        │   └── generate_phonopy_data<3526> Finished [0]
        └── DielectricWorkChain<3341> Waiting [8:while_(should_run_electric_field_scfs)]
            ├── PwBaseWorkChain<3348> Finished [0] [3:results]
            │   └── PwCalculation<3353> Finished [0]
            ├── PwBaseWorkChain<3368> Finished [0] [3:results]
            │   └── PwCalculation<3372> Finished [0]
            ├── compute_critical_electric_field<3464> Finished [0]
            ├── get_accuracy_from_critical_field<3466> Finished [0]
            ├── get_electric_field_step<3468> Finished [0]
            ├── PwBaseWorkChain<3472> Finished [0] [3:results]
            │   └── PwCalculation<3475> Finished [0]
            ├── PwBaseWorkChain<3534> Finished [0] [3:results]
            │   └── PwCalculation<3537> Finished [0]
            ├── PwBaseWorkChain<3540> Waiting [2:while_(should_run_process)(1:run_process)]
            │   └── PwCalculation<3543> Waiting
            ├── PwBaseWorkChain<3546> Waiting [2:while_(should_run_process)(1:run_process)]
            │   └── PwCalculation<3549> Waiting
            ├── PwBaseWorkChain<3552> Waiting [2:while_(should_run_process)(1:run_process)]
            │   └── PwCalculation<3555> Waiting
            ├── PwBaseWorkChain<3558> Waiting [2:while_(should_run_process)(1:run_process)]
            │   └── PwCalculation<3561> Waiting
            └── PwBaseWorkChain<3564> Waiting [2:while_(should_run_process)(1:run_process)]
                └── PwCalculation<3567> Waiting
lean_workdir: false # whether to clean the working directiories
dielectric:
  clean_workdir: false
  kpoints_parallel_distance: 0.2 # kpoints distance in Angstrom^-1 to sample the BZ parallel to the electric field. If used, it should help in converging faster the final results
  property: raman
  # central_difference: # if you know what you are doing, custom numerical derivatives with respect to electric field
  #   accuracy: 2
  #   electric_field_step: 0.0005
  scf:
    pseudo_family: PBE_PSP_SR
    kpoints_distance: 0.4 # kpoints distance in Angstrom^-1 to sample the BZ
    kpoints_force_parity: false
    max_iterations: 10
    pw:
      metadata:
        options:
          max_wallclock_seconds: 57600
          resources:
            num_machines: 1
            num_mpiprocs_per_machine: 256
              #queue_name: standard # for SLURM
          # account: account_name # for SLURM, also for project etc
          withmpi: true
      settings:
        cmdline: ['-pd', '.true.']
      parameters:
        ELECTRONS:
          conv_thr: 2.0e-10
          electron_maxstep: 200
          mixing_beta: 0.3
        SYSTEM:
          ecutrho: 280.0
          ecutwfc: 70.0
          vdw_corr: Grimme-D2




  settings:
    sleep_submission_time: 1.0
phonon:
  clean_workdir: false
  displacement_generator:
    distance: 0.05 # atomic displacements for phonon calculation, in Angstrom
  scf:
    pseudo_family: PBE_PSP_SR
    kpoints_distance: 0.15 # kpoints distance in Angstrom^-1 to sample the BZ
    kpoints_force_parity: false
    max_iterations: 5
    pw:
      metadata:
        options:
          max_wallclock_seconds: 57600
          resources:
            num_machines: 1
            num_mpiprocs_per_machine: 256
          # queue_name: partition_name # for SLURM
          # account: account_name # for SLURM, also for project etc
          withmpi: true
      settings:
        cmdline: ['-pd', '.true.']
        # gamma_only: True # to use only if KpointsData has only a mesh 1 1 1 0 0 0 (i.e. Gamma not shifted)
      parameters:
        ELECTRONS:
          conv_thr: 2.0e-10
          electron_maxstep: 200
          mixing_beta: 0.3
        SYSTEM:

          ecutwfc: 70.0
          ecutrho: 280

          vdw_corr: Grimme-D2



  settings:
    sleep_submission_time: 1.0 # waiting time in seconds between different submission of SCF calculation. Recommended to be at least 1 second, to not overload.
settings:
  run_parallel: true
  use_primitive_cell: false
symmetry:
  distinguish_kinds: false
  is_symmetry: true
  symprec: 1.0e-05

Hi, is it the problem that it is slow or that I does not converge at all? If the scf accuracy oscillates, it might be a problem related to zener tunneling, hence it should be better to lower either the kpoints or the electric field.

If instead it is just slow in the computation, unfortunately there’s no much to do. Applying an electric field is basically like using an hybrid functional, as a non-local operator is computed (the electric enthalpy). Usually converged spectra can be also obtained with much less k-points (when speaking about relative intensities, which is what one is usually interested in).

Hi @bastonero
In some cases, it oscillates about 1e-6- 1e7.
So what do I need to do for reducing electric field in overrides.yaml?

I would suggest you to first look what automated electric field step has been found. Then you can uncomment in the overrides “central_difference” and “electric_field_step”. For the latter, choose a smaller value than the one automatically chosen, and it should be enough. Also, it could be wiser to decrease the conv_thr in ELECTRONS, since the forces produced by a smaller field will also be smaller. If this is not enough, try reducing the kpoints.

Just I understood correctly,

I went to aiida.in

&ELECTRONS
  conv_thr =   2.0000000000d-10
  efield_cart(1) =   6.6538910844d-05
  efield_cart(2) = 0
  efield_cart(3) =   6.6538910844d-05
  efield_phase = 'read'
  electron_maxstep = 200
  mixing_beta =   3.0000000000d-01
  startingpot = 'file'
  startingwfc = 'file'
/

So I choose electric_field_step lower than 6.6538910844d-05, right?

Ok, actually, it is already a very small electric field. Is your system provided with a band gap? Can you verify if this is the case? The method does not work for metals, and also its underlying approximation. You cannot have non-resonant Raman in metals.

Hi Lorenzo,

It has a small gap of ~0.4 eV at DFT level.
In the workchain it tried doing something but still not helping to converge.

2024-03-28 16:16:24 [1747 | REPORT]:       [3546|PwBaseWorkChain|report_error_handled]: PwCalculation<3549> failed with exit status 410: The electronic minimization cycle did not reach self-consistency.
2024-03-28 16:16:24 [1748 | REPORT]:       [3546|PwBaseWorkChain|report_error_handled]: Action taken: reduced beta mixing from 0.3 to 0.24 and restarting from the last calculation
2024-03-28 16:16:24 [1749 | REPORT]:       [3546|PwBaseWorkChain|inspect_process]: PwCalculation<3549> failed but a handler dealt with the problem, restarting

Also, @bastonero
I separately run scf calculation with efield 6e-6, which seem to converge and and was faster.
When I changed overrides.yaml and submit the job, I got following error.

Traceback (most recent call last):
  File "/lustre/scratch5/.mdt0/rkarkee/submit_with_overrides.py", line 86, in <module>
    main()
  File "/lustre/scratch5/.mdt0/rkarkee/submit_with_overrides.py", line 49, in main
    builder = IRamanSpectraWorkChain.get_builder_from_protocol(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/aiida_vibroscopy/workflows/spectra/iraman.py", line 110, in get_builder_from_protocol
    inputs = cls.get_protocol_inputs(protocol, overrides)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/aiida_quantumespresso/workflows/protocols/utils.py", line 67, in get_protocol_inputs
    overrides = yaml.safe_load(file)
                ^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
                         ^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
           ^^^^^^^^^^^^^^^^^^
  File "/users/rkarkee/conda/envs/aiidaENV/lib/python3.11/site-packages/yaml/scanner.py", line 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "overrides_hfte5_pbe.yaml", line 6, column 22

My overrides.yaml is:

clean_workdir: false # whether to clean the working directiories
dielectric:
  clean_workdir: false
  kpoints_parallel_distance: 0.2 # kpoints distance in Angstrom^-1 to sample the BZ parallel to the electric field. If used, it should help in converging faster the final results
  property: raman
   central_difference: # if you know what you are doing, custom numerical derivatives with respect to electric field
    accuracy: 2
   electric_field_step: 6.65e-06
  scf:
    pseudo_family: PBE_PSP_SR
    kpoints_distance: 0.4 # kpoints distance in Angstrom^-1 to sample the BZ
    kpoints_force_parity: false
    max_iterations: 10
    pw:
      metadata:
        options:
          max_wallclock_seconds: 57600
          resources:
            num_machines: 1
            num_mpiprocs_per_machine: 256
              #queue_name: standard # for SLURM
          # account: account_name # for SLURM, also for project etc
          withmpi: true
      settings:
        cmdline: ['-pd', '.true.']
      parameters:
        ELECTRONS:
          conv_thr: 2.0e-9
          electron_maxstep: 200
          mixing_beta: 0.3
        SYSTEM:
          ecutrho: 280.0
          ecutwfc: 70.0
          vdw_corr: Grimme-D2

  settings:
    sleep_submission_time: 1.0
phonon:
  clean_workdir: false
  displacement_generator:
    distance: 0.05 # atomic displacements for phonon calculation, in Angstrom
  scf:
    pseudo_family: PBE_PSP_SR
    kpoints_distance: 0.15 # kpoints distance in Angstrom^-1 to sample the BZ
    kpoints_force_parity: false
    max_iterations: 5
    pw:
      metadata:
        options:
          max_wallclock_seconds: 57600
          resources:
            num_machines: 1
            num_mpiprocs_per_machine: 256
          # queue_name: partition_name # for SLURM
          # account: account_name # for SLURM, also for project etc
          withmpi: true
      settings:
        cmdline: ['-pd', '.true.']
        # gamma_only: True # to use only if KpointsData has only a mesh 1 1 1 0 0 0 (i.e. Gamma not shifted)
      parameters:
        ELECTRONS:
          conv_thr: 2.0e-10
          electron_maxstep: 200
          mixing_beta: 0.3
        SYSTEM:

          ecutwfc: 70.0
          ecutrho: 280

          vdw_corr: Grimme-D2

  settings:
    sleep_submission_time: 1.0 # waiting time in seconds between different submission of SCF calculation. Recommended to be at least 1 second, to not overload.
settings:
  run_parallel: true
  use_primitive_cell: false
symmetry:
  distinguish_kinds: false
  is_symmetry: true
  symprec: 1.0e-05

The problem is how you indented the variables in the yaml file. It should be

dielectric:
  clean_workdir: false
  kpoints_parallel_distance: 0.2
  property: raman
  central_difference:
    accuracy: 2
    electric_field_step: 6.65e-06
  scf:
    ...

Hi @bastonero ,

Something I realized for convergence,
Any combination that involves efield_cart(3) is having hard time to converge. But When efield_cart(3) is not involved or 0, it is converged quite fast, even for larger value of efield. Any idea why this might be happening only when it involves efield_cart(3) and not for others? The crystal structure is bulk.

Hi, probably the small band gap is in that direction, so there not much to do if not decreasing the number of kpoints or the electric field step value.

Moreover, consider the fact that for small band gaps, non-resonant Raman might not be describing well anymore the experiments (if the laser frequency is ~ band gap).

But surprisingly such didn’t happen when I was doing with PBE functional. Unfortunately I don’t have DFPT from PBE. I am surprised it happened in LDA.

Different band gaps?

It is different bad gaps but it’s showing gap of 0.6 eV.

Also experiment is using 532nm which is non resonant case.

Have you tried to either lower the kpoints or the electric field step?

Yes,

I went upto 2 2 1 kgrid and 4e-6 efield step. (My original calculation was 4 4 2 kgrid)

It might also be that now that you have a tighter threshold for electronic convergence, your system wouldn’t even converge for PBE (good to check this).

Good to check the Kohn-Sham band gap (most likely lower than experiments). Then, try to set just a single kpoint. Do you have any sort of magnetism?

Moreover, certain compilation flags (-O3 in particular) used to somehow make the code to not converge properly. So, it would be good to also try a very simple Silicon structure to make sure the code is working properly. Silicon is very fast to simulate, it’s basically a benchmark.