Paramiko error with portable code

General Usage
1

Dear all,
I’m trying to use portable codes in my project and I’m getting an error from paramiko while uploading the code file (test.py):
Expected Unicode or bytes, got PurePosixPath('test.py')
I temporarily solved the issue by modifying the aiida/transports/plugins/ssh.py file giving as argument to sftp.normalize, sftp.stat, sftp.chmod, sftp.put methods, str(path) instead of path alone.

Do you have an idea how I can solve it without modifying aiida-core scripts?

I’ll send you below the scripts I’m using for code definition, plugin and the full error.

Thank you.
Davide

Code definition:

from pathlib import Path
from aiida.orm import PortableCode
from aiida import load_profile
load_profile()
  
code = PortableCode(
    label='test',
    filepath_files=Path('/home/bidoggia/onedrive/aiida/tests/test_portable/'),
    filepath_executable='test.py'
)
code. Store()

Part of plugin:

codeinfo = datastructures.CodeInfo()
codeinfo.code_uuid = self.inputs.code.uuid
calcinfo = datastructures.CalcInfo()
calcinfo.local_copy_list = []
calcinfo.codes_info = [codeinfo]

Full error:

 | Traceback (most recent call last):
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/aiida/engine/utils.py", line 202, in exponential_backoff_retry
 |     result = await coro()
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/aiida/engine/processes/calcjobs/tasks.py", line 94, in do_upload
 |     execmanager.upload_calculation(node, transport, calc_info, folder)
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/aiida/engine/daemon/execmanager.py", line 193, in upload_calculation
 |     transport.put(handle.name, (root / filename))
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/aiida/transports/plugins/ssh.py", line 857, in put
 |     if self.isdir(remotepath):
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/aiida/transports/plugins/ssh.py", line 766, in isdir
 |     return S_ISDIR(self.stat(path).st_mode)
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/aiida/transports/plugins/ssh.py", line 633, in stat
 |     return self.sftp.stat(path)
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/paramiko/sftp_client.py", line 491, in stat
 |     path = self._adjust_cwd(path)
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/paramiko/sftp_client.py", line 914, in _adjust_cwd
 |     path = b(path)
 |   File "/home/bidoggia/py_envs/aiida/lib/python3.10/site-packages/paramiko/py3compat.py", line 156, in b
 |     raise TypeError("Expected unicode or bytes, got {!r}".format(s))
 | TypeError: Expected unicode or bytes, got PurePosixPath('test.py')
1 Like
2

Hi @Davide_Bidoggia, thanks for posting it again. Could you please provide us some more information about your environment, most importantly the AiiDA version? I just tried to reproduce your issue, but don’t get the exception with the latest version 2.6.1. Your setup as you posted it here seems correct to me (apart from the capitalization of .Store(), but I guess this is an error of copying the snippet here).

Instead, if I wrap ‘test.py’ in Path, I’m indeed getting a similar traceback:

Click to expand traceback 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /home/geiger_j/test/test_davide.py:8
      4 from aiida import load_profile
      6 load_profile()
----> 8 code = PortableCode(
      9     label='test',
     10     filepath_files=Path('/home/geiger_j/test'),
     11     filepath_executable=Path('test.py'),
     12 )
     13 code.store()

File ~/dev/aiida-dev/aiida-core/src/aiida/orm/nodes/data/code/portable.py:92, in PortableCode.__init__(self, filepath_executable, filepath_files, **kwargs)
     90 super().__init__(**kwargs)
     91 type_check(filepath_files, pathlib.Path)
---> 92 self.filepath_executable = filepath_executable  # type: ignore[assignment]
     93 self.base.repository.put_object_from_tree(str(filepath_files))

File ~/dev/aiida-dev/aiida-core/src/aiida/orm/nodes/data/code/portable.py:174, in PortableCode.filepath_executable(self, value)
    168 @filepath_executable.setter
    169 def filepath_executable(self, value: str) -> None:
    170     """Set the relative filepath of the executable that this code represents.
    171 
    172     :param value: The relative filepath of the executable within the directory of uploaded files.
    173     """
--> 174     type_check(value, str)
    176     if pathlib.PurePosixPath(value).is_absolute():
    177         raise ValueError('The `filepath_executable` should not be absolute.')

File ~/dev/aiida-dev/aiida-core/src/aiida/common/lang.py:42, in type_check(what, of_type, msg, allow_none)
     40     if msg is None:
     41         msg = f"Got object of type '{type(what)}', expecting '{of_type}'"
---> 42     raise TypeError(msg)
     44 return what

TypeError: Got object of type '<class 'pathlib.PosixPath'>', expecting '<class 'str'>'

while filepath_files has to be wrapped in Path. If not:

Click to expand traceback 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File /home/geiger_j/test/test_davide.py:8
      4 from aiida import load_profile
      6 load_profile()
----> 8 code = PortableCode(
      9     label='test',
     10     filepath_files='/home/geiger_j/test',
     11     filepath_executable='test.py',
     12 )
     13 code.store()

File ~/dev/aiida-dev/aiida-core/src/aiida/orm/nodes/data/code/portable.py:91, in PortableCode.__init__(self, filepath_executable, filepath_files, **kwargs)
     75 """Construct a new instance.
     76 
     77 .. note:: If the files necessary for this code are not all located in a single directory or the directory
   (...)
     88 :param filepath_files: The filepath to the directory containing all the files of the code.
     89 """
     90 super().__init__(**kwargs)
---> 91 type_check(filepath_files, pathlib.Path)
     92 self.filepath_executable = filepath_executable  # type: ignore[assignment]
     93 self.base.repository.put_object_from_tree(str(filepath_files))

File ~/dev/aiida-dev/aiida-core/src/aiida/common/lang.py:42, in type_check(what, of_type, msg, allow_none)
     40     if msg is None:
     41         msg = f"Got object of type '{type(what)}', expecting '{of_type}'"
---> 42     raise TypeError(msg)
     44 return what

TypeError: Got object of type '<class 'str'>', expecting '<class 'pathlib.Path'>'

Thanks!

3

Hi Davide, I think this may be a bug in aiida-core. The relevant line is this:

The engine is using the transport to copy over the files from the PortableCode to the remote working directory. It is specifying the remote path as (root / filename) which is an instance of pathlib.Path but unfortunately the Transport interface doesn’t support pathlib.Path objects yet and requires normal str strings.

I am a bit surprised this isn’t tested or somehow was missed in the tests. Have to check how this slipped past. The PortableCode concept is not used often in real use cases by users, which may be why I went unnoticed for so long.

I have opened an issue to fix it: `PortableCode` breaks calculation jobs as `upload_calculation` is using `pathlib.Path` objects with the `Transport` interface. · Issue #6518 · aiidateam/aiida-core · GitHub

I just tried to reproduce your issue, but don’t get the exception with the latest version 2.6.1.

@geiger_j note that the reported exception does not occur upon instantiating the PortableCode instance, but when it is actually used for a CalcJob. It happens during the upload calculation step.

1 Like
4

Thank you @sphuber @geiger_j,
yes, exactly, the problem arises when executing the CalcJob.
I’m using version 2.5.1

5

I confirmed the bug and opened a PR with a fix: Engine: Fix bug in upload calculation for `PortableCode` with SSH by sphuber · Pull Request #6519 · aiidateam/aiida-core · GitHub

It also contains a regression test and an explanation why this was missed in the test suite. Daniele, for now you will have to just use your local fix until this is merged and I can release a patch version. Thanks for reporting the bug!

6

Ah, sorry, my bad. Thanks a lot for taking care of this so quickly, @sphuber!