Do glob patterns work for remote copy lists?

bastonero · October 9, 2024, 5:34pm

Hi everyone,
I would implemented something on the line:

calc_info.remote_copy_list[(self.inputs.parent_folder.computer.uuid, 'restart_*', None)]

As I might not know the name of the file. Would this example work?
Otherwise i guess i might dynamically look into the remote tree and pick the names, although less convenient.
Thanks a lot for any pointer!

sphuber · October 9, 2024, 6:15pm

I think that should be possible but not a 100% sure. Looking at the source code here

github.com

aiidateam/aiida-core/blob/98ffc331d154cc7717861fde6c908ec510003926/src/aiida/engine/daemon/execmanager.py#L293


      
          
          def _copy_remote_files(logger, node, computer, transport, remote_copy_list, remote_symlink_list):
              """Perform the copy instructions of the ``remote_copy_list`` and ``remote_symlink_list``."""
              for remote_computer_uuid, remote_abs_path, dest_rel_path in remote_copy_list:
                  if remote_computer_uuid == computer.uuid:
                      logger.debug(
                          f'[submission of calculation {node.pk}] copying {dest_rel_path} '
                          f'remotely, directly on the machine {computer.label}'
                      )
                      try:
                          transport.copy(remote_abs_path, dest_rel_path)
                      except FileNotFoundError:
                          logger.warning(
                              f'[submission of calculation {node.pk}] Unable to copy remote '
                              f'resource from {remote_abs_path} to {dest_rel_path}! NOT Stopping but just ignoring!.'
                          )
                      except OSError:
                          logger.warning(
                              f'[submission of calculation {node.pk}] Unable to copy remote '
                              f'resource from {remote_abs_path} to {dest_rel_path}! Stopping.'
                          )

we can see the remote_copy_list items are passed to Transport.copy which in turn seems to support globs in the source:

github.com

aiidateam/aiida-core/blob/98ffc331d154cc7717861fde6c908ec510003926/src/aiida/transports/plugins/ssh.py#L1161


      
          
          if not remotedestination:
              raise ValueError(
                  'Input to copy() must be a non empty string. '
                  + f'Found instead {remotedestination} as remotedestination'
              )
          
          if self.has_magic(remotedestination):
              raise ValueError('Pathname patterns are not allowed in the destination')
          
          if self.has_magic(remotesource):
              to_copy_list = self.glob(remotesource)
          
              if len(to_copy_list) > 1:
                  if not self.path_exists(remotedestination) or self.isfile(remotedestination):
                      raise OSError("Can't copy more than one file in the same destination file")
          
              for file in to_copy_list:
                  self._exec_cp(cp_exe, cp_flags, file, remotedestination)
          
          else:

Why don’t you give it a try?

bastonero · October 10, 2024, 8:21am

Thanks Sebastiaan! I tried it out but unfortunately it doesn’t work, as it requires to specify 3 strings (so None is not accepted), and at the same time it doesn’t accept glob patterns for the last string (as it stop throwing out a related error saying that). I am working with aiida-core v2.3.1, so don’t know if there is any new functionality that allows to do so - otherwise, it seems that one cannot use wildcards for the remote copy list right now.

giovannipizzi · October 10, 2024, 9:40am

Did you try to specify a folder for the last (destination) path, as you would do with cp? Try with a subfolder of the scratch that you should create beforehand (this should work, I think?) and then also with . if you want to copy in to the main scratch folder of the job (there is a small probability that the latter does not work, in which case it’s a bug and we should fix it).

bastonero · October 10, 2024, 10:35am

Thanks @giovannipizzi ! It did work using .

system · October 16, 2024, 6:35am

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
AiiDAlab transport plugins Ideas discussion	3	57	July 18, 2023
How crucial is `dereference` -support for symlink- in ssh transport plugin Developer	1	24	June 11, 2024
Use a common directory for a number of tasks General Usage	14	105	February 19, 2024
AiiDA Submission Controller Help New to AiiDA question	7	124	January 19, 2024
List(node.res) does not work New to AiiDA	1	55	January 29, 2024

Do glob patterns work for remote copy lists?

Related topics