How to use scp/rsync instead of sftp for transport

General Usage
1

Dear all,

I am setting up the computer within AiiDA on my personal laptop for a remote HPC (bridges2 from PSC), with the configuration as following:

allow_agent: true
compress: true
gss_auth: false
gss_deleg_creds: false
gss_host: bridges2.psc.edu
gss_kex: false
key_filename: /Users/PATH/.ssh/id_rsa_github
key_policy: AutoAddPolicy
load_system_host_keys: false
look_for_keys: true
port: 22
proxy_command: ''
proxy_jump: ''
safe_interval: 30.0
timeout: 60
use_login_shell: true
username: MyUserName

The configuration is about the login node of PSC HPC.
When I did the test, I got the following error:

* Opening connection... [FAILED]: Error while trying to connect to the computer
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 511, in open
      self._sftp = self._client.open_sftp()
                   ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/client.py", line 608, in open_sftp
      return self._transport.open_sftp_client()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/transport.py", line 1216, in open_sftp_client
      return SFTPClient.from_transport(self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/sftp_client.py", line 169, in from_transport
      chan.invoke_subsystem("sftp")
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 70, in _check
      return func(self, *args, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 281, in invoke_subsystem
      self._wait_for_event()
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 1224, in _wait_for_event
      raise e
  paramiko.ssh_exception.SSHException: Channel closed.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 553, in computer_test
      with transport:
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 178, in __enter__
      self.open()
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 514, in open
      raise InvalidOperation(
  aiida.common.exceptions.InvalidOperation: Error in ssh transport plugin. This may be due to the remote computer not supporting SFTP. Try setting it up with the aiida.transports:ssh_only transport from the aiida-sshonly plugin instead.
Warning: 1 out of 1 tests failed

It is saying that SFTP is not supported on the login node, while based on the tutorial of the PSC HPC, the files can be transferred via SFTP on the data node (data.bridges2.psc.edu).
So I regenerate the setup of computer regarding that data node, with configuration:

allow_agent: true
compress: true
gss_auth: false
gss_deleg_creds: false
gss_host: data.bridges2.psc.edu
gss_kex: false
key_filename: /Users/PATH/.ssh/id_rsa_github
key_policy: AutoAddPolicy
load_system_host_keys: false
look_for_keys: true
port: 22
proxy_command: ''
proxy_jump: ''
safe_interval: 30.0
timeout: 60
use_login_shell: true
username: MyUserName

And I got the following error:

* Opening connection... [OK]
* Checking for spurious output... /Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py:674: AiidaDeprecationWarning: `chdir()` is deprecated and will be removed in the next major version. Use absolute paths instead. (this will be removed in v3)
  warn_deprecation(
[Failed]: TimeoutError:
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 697, in recv
      out = self.in_buffer.read(nbytes, self.timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 154, in read
      raise PipeTimeout()
  paramiko.buffered_pipe.PipeTimeout

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 564, in computer_test
      success, message = test(
                         ^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 78, in _computer_test_no_unexpected_output
      retval, stdout, stderr = transport.exec_command_wait('echo -n')
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 500, in exec_command_wait
      retval, stdout_bytes, stderr_bytes = self.exec_command_wait_bytes(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 1552, in exec_command_wait_bytes
      stdout_bytes.append(stdout.read())
                          ^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/file.py", line 184, in read
      new_data = self._read(self._DEFAULT_BUFSIZE)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 1359, in _read
      return self.channel.recv(size)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
      raise socket.timeout()
  TimeoutError
* Getting number of jobs from scheduler... [Failed]: TimeoutError:
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 697, in recv
      out = self.in_buffer.read(nbytes, self.timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 154, in read
      raise PipeTimeout()
  paramiko.buffered_pipe.PipeTimeout

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 564, in computer_test
      success, message = test(
                         ^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 58, in _computer_test_get_jobs
      found_jobs = scheduler.get_jobs(as_dict=True)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/schedulers/plugins/bash.py", line 52, in get_jobs
      retval, stdout, stderr = self.transport.exec_command_wait(self._get_joblist_command(jobs=jobs, user=user))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 500, in exec_command_wait
      retval, stdout_bytes, stderr_bytes = self.exec_command_wait_bytes(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 1552, in exec_command_wait_bytes
      stdout_bytes.append(stdout.read())
                          ^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/file.py", line 184, in read
      new_data = self._read(self._DEFAULT_BUFSIZE)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 1359, in _read
      return self.channel.recv(size)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
      raise socket.timeout()
  TimeoutError
* Determining remote user name... [Failed]: TimeoutError:
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 697, in recv
      out = self.in_buffer.read(nbytes, self.timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 154, in read
      raise PipeTimeout()
  paramiko.buffered_pipe.PipeTimeout

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 564, in computer_test
      success, message = test(
                         ^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 115, in _computer_get_remote_username
      remote_user = transport.whoami()
                    ^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 858, in whoami
      retval, username, stderr = self.exec_command_wait(command)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 500, in exec_command_wait
      retval, stdout_bytes, stderr_bytes = self.exec_command_wait_bytes(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 1552, in exec_command_wait_bytes
      stdout_bytes.append(stdout.read())
                          ^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/file.py", line 184, in read
      new_data = self._read(self._DEFAULT_BUFSIZE)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 1359, in _read
      return self.channel.recv(size)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
      raise socket.timeout()
  TimeoutError
* Creating and deleting temporary file... [Failed]: TimeoutError:
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 697, in recv
      out = self.in_buffer.read(nbytes, self.timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 154, in read
      raise PipeTimeout()
  paramiko.buffered_pipe.PipeTimeout

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 564, in computer_test
      success, message = test(
                         ^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 135, in _computer_create_temp_file
      workdir = authinfo.get_workdir().format(username=transport.whoami())
                                                       ^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 858, in whoami
      retval, username, stderr = self.exec_command_wait(command)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 500, in exec_command_wait
      retval, stdout_bytes, stderr_bytes = self.exec_command_wait_bytes(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 1552, in exec_command_wait_bytes
      stdout_bytes.append(stdout.read())
                          ^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/file.py", line 184, in read
      new_data = self._read(self._DEFAULT_BUFSIZE)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 1359, in _read
      return self.channel.recv(size)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
      raise socket.timeout()
  TimeoutError
* Checking for possible delay from using login shell... [Failed]: TimeoutError:
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 697, in recv
      out = self.in_buffer.read(nbytes, self.timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/buffered_pipe.py", line 154, in read
      raise PipeTimeout()
  paramiko.buffered_pipe.PipeTimeout

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 564, in computer_test
      success, message = test(
                         ^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 221, in _computer_use_login_shell_performance
      timing_false = time_use_login_shell(authinfo, auth_params_clone, False, iterations)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 192, in time_use_login_shell
      transport.whoami()
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 858, in whoami
      retval, username, stderr = self.exec_command_wait(command)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 500, in exec_command_wait
      retval, stdout_bytes, stderr_bytes = self.exec_command_wait_bytes(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh.py", line 1552, in exec_command_wait_bytes
      stdout_bytes.append(stdout.read())
                          ^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/file.py", line 184, in read
      new_data = self._read(self._DEFAULT_BUFSIZE)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 1359, in _read
      return self.channel.recv(size)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/paramiko/channel.py", line 699, in recv
      raise socket.timeout()
  TimeoutError
Warning: 5 out of 6 tests failed

It seems that the sftp connect was established successfully, but the data node does not allow any operations.

I also checked that, there is one way to use only scp/rsync for file transfer, with the module “aiida_onlyssh”. However, I got a severe error about the version mismatch between the libraries in my Conda environment and the aiida_onlyssh software, as it was developed 5 years ago.

I am wondering if it is possible to disable the sftp file transfer when setting up the computer within AiiDA?
As I have checked, the file transfer with rsync is ok.
I truly appreciate it if anyone could help me out.

I am using AiiDA version 2.7.1, and the no-password ssh connect has been setup between my local laptop and PSC HPC, and the AiiDA was well configured on my local laptop:

 ✔ version:     AiiDA v2.7.1
 ✔ config:      /Users/PATH/.aiida
 ✔ profile:     PSAL_dos
 ✔ storage:     Storage for ...
Warning: RabbitMQ v4.1.4 is not supported and will cause unexpected problems!
Warning: It can cause long-running workflows to crash and jobs to be submitted multiple times.
Warning: See https://github.com/aiidateam/aiida-core/wiki/RabbitMQ-version-to-use for details.
 ✔ broker:      RabbitMQ v4.1.4 ... 
 ✔ daemon:      Daemon

Best
Zhishuo

2

Hi @Zhishuo! I’m not sure if this is possible at the moment to be honest. I’m going to ping @ali-khosravi, our transport expert, but he’s on holiday at the moment.

I wasn’t even aware of aiida-sshonly. Seems the developer was running into a similar issue as you, and decided to develop their own transport plugin. But indeed, the work was done some time ago and I see the package hasn’t been updated for aiida-core v2. It might be worthwhile to do so, however.

3

I think the new async plugin with the openssh backend should be able to to the job, after configuring ssh properly for your host in your .ssh/config

Riccardo Bertossa

4

Thank you Marnik for your response.

5

Hello Riccardo,
Thank you very much for your suggestion.
I tried to setup it up using core.ssh_acrync, for which my config file is:

label: HPCACCESSPSC_verdi_async
description: PSC in ACCESS
hostname: bridges2.psc.edu
transport: core.ssh_async
scheduler: core.slurm
work_dir: "/ocean/projects/PROJECT/USERNAME"
mpirun_command: "srun -n {tot_num_mpiprocs}"
mpiprocs_per_machine: 128

And I tried “verdi computer setup --config HPCACCESSPSC_verdi_async.yaml”, and the interaction is as following:

Report: enter ? for help.
Report: enter ! to ignore the default and set no value.
Shebang line (first line of each script, starting with #!) [#!/bin/bash]:
Default amount of memory per machine (kB).: 265000000
Escape CLI arguments in double quotes [y/N]: y
Success: Computer<19> HPCACCESSPSC_verdi_async created
Report: Note: before the computer can be used, it has to be configured with the command:
Report:   verdi -p PSAL_dos computer configure core.ssh_async HPCACCESSPSC_verdi_async

Then I configured it, “verdi -p PSAL_dos computer configure core.ssh_async HPCACCESSPSC_verdi_async”, with the procedure:

Report: enter ? for help.
Report: enter ! to ignore the default and set no value.
Host as in 'ssh <HOST>' (needs to be a password-less setup in your ssh config) [bridges2.psc.edu]:
Maximum number of concurrent I/O operations [8]:
Local script to run *before* opening connection (path) [None]:
Type of async backend to use, `asyncssh` or `openssh` [asyncssh]: openssh
Use login shell when executing command [Y/n]: Y
Connection cooldown time (s) [30.0]:
Report: Configuring computer HPCACCESSPSC_verdi_async for user mypersonalemail@gmail.com.
Success: HPCACCESSPSC_verdi_async successfully configured for mypersonalemail@gmail.com

The wired thing is that the configuration is for my personal gmail, which is basic the user information of my local laptop, and I can not specify the user name for the HPCs.
I tried to do verdi computer test,

Report: Testing computer<HPCACCESSPSC_verdi_async> for user<mypersonalemail@gmail.com>...
* Opening connection... Warning: RabbitMQ v4.1.4 is not supported and will cause unexpected problems!
Warning: It can cause long-running workflows to crash and jobs to be submitted multiple times.
Warning: See https://github.com/aiidateam/aiida-core/wiki/RabbitMQ-version-to-use for details.
[OK]
Zhishuo@bridges2.psc.edu's password:

It seems that the username is the username on my local laptop.
I tried to re-configure the username “verdi computer configure core.ssh_async -u “HPCusername” HPCACCESSPSC_verdi_async”, and I got:

Usage: verdi computer configure core.ssh_async [OPTIONS] COMPUTER
Try 'verdi computer configure core.ssh_async --help' for help.

Error: Invalid value for '-u' / '--user': User 'HPCusername' not found

Could you please give me a hand on how to provide the login information (username and path of the private ssh-key) of HPCs?

Best
Zhishuo

6

Hi, asyncssh+openssh should use the configuration in your ~/.ssh./config file. So, if you are able to do ssh bridges2.psc.edu without specifying username and password, also AiiDA should be able to do it.

Do you also have a prompt for a password if you do ssh bridges2.psc.edu? (when running this command without specifying a username). I would guess so. Then, configure accordingly your ~/.ssh/config until the password is not required. I.e. add the following lines to it (possibly create the file if it does not exist):

Host bridges2.psc.edu
  Hostname bridges2.psc.edu
  User <YOUR_USERNAME_ON_PSC>
  IdentityFile <PATH_TO_SSH_KEY>

IdentityFile is optional, but ideally you should put the absolute path to your SSH key that allows you to login without a password.

7

Hi @Zhishuo , Seems like the email is taken from your profile setup.
I’d follow @giovannipizzi suggestion. In case your HPC strictly requires you to login with manually entering password every time, let us know so that we’ll suggest further solutions on that.

8

Hello Giovanni,

Thank you very much for your suggestion.
It worked for the connection with ssh to the PSC.
The previous problem is that I set a name different from the hostname in the config file:

Host HPCACCESSPSC
        HostName bridges2.psc.edu
        User USERNAME
        IdentityFile ~/.ssh/id_rsa_github

I changed it into the hostname:

Host bridges2.psc.edu
        HostName bridges2.psc.edu
        User USERNAME
        IdentityFile ~/.ssh/id_rsa_github

However, when I tested the computer with “verdi computer test HPCACCESSPSC_verdi_async”, I got the error as following:

* Opening connection... Warning: RabbitMQ v4.1.4 is not supported and will cause unexpected problems!
Warning: It can cause long-running workflows to crash and jobs to be submitted multiple times.
Warning: See https://github.com/aiidateam/aiida-core/wiki/RabbitMQ-version-to-use for details.
[OK]
* Checking for spurious output... [OK]
* Getting number of jobs from scheduler... [OK]: 1854 jobs found in the queue
* Determining remote user name... [OK]: Username
* Creating and deleting temporary file... [Failed]: OSError: Error while uploading file /var/folders/cz/hcgvrw9502sg1lqf51hzw6pc0000gq/T/tmpmt_dtvs2: {'subsystem request failed on channel 0\r\nscp: Connection closed\r\n'}
  Full traceback:
  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh_async.py", line 529, in putfile_async
      await self.async_backend.put(
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/async_backend.py", line 652, in put
      raise OSError({stderr})
  OSError: {'subsystem request failed on channel 0\r\nscp: Connection closed\r\n'}

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 564, in computer_test
      success, message = test(
                         ^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/cmdline/commands/cmd_computer.py", line 145, in _computer_create_temp_file
      transport.putfile(tempf.name, remote_file_path)
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 1828, in putfile
      return self.run_command_blocking(self.putfile_async, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/transport.py", line 1804, in run_command_blocking
      return loop.run_until_complete(func(*args, **kwargs))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/engine/runners.py", line 160, in run_until_complete
      return self._loop.run_until_complete(future)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete
      return f.result()
             ^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/asyncio/futures.py", line 203, in result
      raise self._exception.with_traceback(self._exception_tb)
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/asyncio/tasks.py", line 277, in __step
      result = coro.send(None)
               ^^^^^^^^^^^^^^^
    File "/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/ssh_async.py", line 534, in putfile_async
      raise OSError(f'Error while uploading file {localpath}: {exc}')
  OSError: Error while uploading file /var/folders/cz/hcgvrw9502sg1lqf51hzw6pc0000gq/T/tmpmt_dtvs2: {'subsystem request failed on channel 0\r\nscp: Connection closed\r\n'}
* Checking for possible delay from using login shell... [OK]
Warning: 1 out of 6 tests failed

It seems that the “scp” could not work well.
I tried to use “scp” in command line:

(aiida) Zhishuo@Mac Phonon_splits % scp Check_log bridges2.psc.edu:
subsystem request failed on channel 0
scp: Connection closed

But if I add “-O”, it works fine:

(aiida) Zhishuo@Mac Phonon_splits % scp -O Check_log bridges2.psc.edu:
Check_log                                                                                                                                                             100%   68KB 215.6KB/s   00:00

Do you have any idea to solve it?

Best
Zhishuo

9

Hi Zhishuo,

Thanks for trying this out and reporting your solution, I wasn’t aware of flag -O. From scp manual I read:

-O Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a ‘~’ prefix for older SFTP servers.

Since -O solves the error, then the fastest way is to adopt the source code of aiida installation in your computer. (Unfortunately at the moment we the configuration doesn’t accept custom flags)

Solution #1, fast and dirty

Take the path from in your terminal run, just to be sure:

$ python -c "import aiida; print(aiida.__file__)"
/Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/__init__.py

Then open this file:

$ open /Users/Zhishuo/anaconda3/envs/aiida/lib/python3.11/site-packages/aiida/transports/plugins/async_backend.py

Replace all options = [] with options = ['-O'] , save the file and exit.

$ verdi daemon restart
$ verdi computer test HPCACCESSPSC_verdi_async

Solution #2, clean, but no guarantee that it works:

add an alias to your .bashrc, .zshrc, etc.

alias scp=‘scp -O’
10

Hello Ali,

Thank you very much for the instruction.

I tried Solution #1, and it works well with all the 6 tests past.

Actually, my colleague tried Solution #2, which resulted in other issues (Sorry I do not have the details from him).

Best
Zhishuo

11

Glad to hear that worked! let us know if you had any other issues.

Cheers!

12

Hi @Zhishuo
We are implementing this as an optional flag.
Before we merge it would be great if you guys also want to try it out, and give us your opinion:

13

Hi @ali-khosravi and everyone else who worked on this,

Thank you for taking the time to implement this, we appreciate the effort. I am working together with @Zhishuo in the same group, and we ran into the issue on this HPC.

Unfortunately the HPC now blocks both SFTP and legacy SCP completely on the login node, so we can’t test the implementation on this HPC anymore.

Originally our problem was that the HPC we are using has two different types of nodes: one for data transfer and one for login.
On the data transfer node, file transfers work, but you cannot log in or run any commands.
On the login node, you can log in and run commands, but file transfers are not allowed.
A few months ago scp -O still worked on the login node, which is how we got AiiDA working in the first place, but that has since been blocked as well.

Our main problem now is that no single hostname on our cluster supports both command execution and file transfer.
For this HPC we would need a split approach, where file transfer and command execution can be done on two different nodes.
I saw that there is a related topic listed on the v3 parent issue, “Decouple upload/download from scheduler comms”, currently under “non-breaking, lower priority, defer to v3.1+”.
Is there already a workaround for this, or is something like that possible in some other way?

In the meantime, I can test the flag on another HPC, which in general doesn’t need it, but I could still enable it and test whether everything works as expected, and report back if I run into any errors, if that’s useful to you.

Thanks again for all the work on this!

14

Hi @Ruben

Thanks a lot for your detailed response.
Ok, I see the problem. We also have heard of this new approach from another user.
I think we should implement this, as it’s a reasonable request.

Just to understand, before implementing anything:

  1. How do you login to the data transfer node? with the same login credentials, ssh key, etc. or are they different? Do either of them require 2fa for login?
  2. Do they publicly announce set of commands that you can run on either of these nodes? for example on login: only slurm commands: squeue, sbatch, etc.. and on data transfer everything else, only apart from the scheduler commands. For instance: mkdir, chdir, test, ln, rm, etc.. is my understanding correct?

In the meantime, I can test the flag on another HPC,

I think it’s ok, would be unnecessarily, don’t bother! But thanks :slight_smile:

15

Hi @ali-khosravi,

Thanks for getting back to us and for looking into implementing this.

  1. Both nodes use the same SSH keys and login credentials, and no 2FA is required for either.
  2. I couldn’t find any publicly documented list of allowed commands for either node, this is based on my own testing rather than something PSC states explicitly.
    On the login node, I can log in via SSH and run any shell
    commands, including Slurm commands (e.g., squeue, sbatch).
    On the data transfer node, I can only log in via SFTP, where operations like mkdir, chdir, ln, rm, get and put do work, as you mentioned.
    The test command, for example, does not work there, my understanding is that the DTN only exposes SFTP’s built-in file operations rather than a general shell, so anything outside that fixed set (like `test`) isn’t available.
16

Thanks a lot @Ruben for following up,

Both nodes use the same SSH keys and login credentials, and no 2FA is required for either.

That simplifies life! very good.
Ok, I implemented your use case, can you please verify if it works for you?

Since you are not bonded to rcp anymore, I suggest to choose asyncssh as backend, it’s a lot faster. While configuring the computer you’ll see a question like this:

Login host as in 'ssh <HOST>' (needs to be a password-less setup in your ssh config) [myhpc]: 
Data transfer host as in 'ssh <HOST>' ('None' to use the login host) [None]:

Where you can configure two nodes for your aiida computer.

Note: I assumed all not sftp commands, including some file operations like mkdir, mv, test etc still functions on your login node.

Let me know if it all works for you!

17

Hi @ali-khosravi,

Thank you for implementing the approach so quickly!
I’ve tested it with several of my workflows now, using a separate login and data transfer host as you set it up, and everything is working smoothly so far.

If I run into any issues or bugs down the line, I’ll make sure to report them.

18

Great @Ruben! thanks for trying out.
Just a summary for anyone who comes across this issue again:

  • If your HPC center:
    • does not permit SFTP operations on the login node, but instead provides a dedicated data transfer node for such operations, follow this.
    • does not support SFTP and does not provide a dedicated data transfer node at all, follow this. This solution switches to legacy rcp protocol.
    • provides a dedicated data transfer node, but does not support SFTP for security reasons --rare scenario, you can combine the two configurations above.