Accessing clusters with 2-factor authentication (Sigma2) using AiiDA

[adamg]

Hello. I have been using AiiDA very extensively in my PhD project to run DFT and MD simulations on the Norwegian HPC clusters (Sigma2) for more than two years now. Basically all my simulations and steps in setting them up are implemented as AiiDA jobs / calcfunctions, and all my post-analysis is implemented as AiiDA queries. Recently Sigma2 enabled 2-factor authentication, and I no longer can access the clusters using AiiDA which is currently completely blocking any progress in my project.

The general solution provided by Sigma2 to prevent having to type in a password each time is adding this to my ssh config file:

Host fram
User myusername
HostName login.fram.sigma2.no
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p

and starting an initial connection manually which will be used by all subsequent connections:

ssh -CX -o ServerAliveInterval=30 -fN fram

Reference: One-time-pad (OTP) / Two-factor authentication — Sigma2 documentation

This works for manual ssh connections, however it seems to me that this does not work in combination with AiiDA. I see this solution has been discussed in the past, and it is mentioned that paramiko does not support multiplexing: Access to cluster with two-factor authentication - #7 by giovannipizzi

Is there any known solution or workaround to get this working? I am a fairly advanced user of AiiDA and I am willing to try potentially less robust workarounds to get this working, since currently my only other alternative is to apply for HPC resources elsewhere.

[giovannipizzi]

Unfortunately, I think that’s technically not possible, I’m not aware of python implementations that allow to use the multiplexing ControlMaster approach (except workarounds that I don’t know how supported are, see e.g. this - if this works for you, one would need to wrap it in a transport plugin, supporting the key methods to copy/get files, execute commands etc).
I was also checking asyncssh that @ali-khosravi implemented a few weeks ago and will end up in the upcoming 2.7 release, but it seems that also that library does not support it, again for the design of it.

A few other options:

• try out this branch (never merged though, so I don’t know if it works) of paramiko that might support it Mux transport for client (rebased, the second) by ltalirz · Pull Request #1931 · paramiko/paramiko · GitHub
• check this approach to use OTP login (DISCLAIMER: we do not recommend it unless you check that your center is OK with this - if this works, you would still need to integrate it in the SSH transport plugin): SSH connection with 2FA (two-factor authentication) using Google Authenticator · Issue #5020 · aiidateam/aiida-core · GitHub
• check with the supercomputer center if they would allow e.g. connections with MFA from a specific IP address you manage, or if they allow a jump host without MFA
• check if they would allow to have a small VM inside their network, where you could install AiiDA and run from there - so you need to use 2FA to connect to the AiiDA machine, but then connections to submit are without 2FA
• You could also check if they support tunneling and, when tunneling, you are not asked the password (e.g. ssh -L <SOMELOCALPORT>:localhost:22 <SUPERCOMPUTER> and then ssh to your localhost on port <SOMELOCALPORT>

I hope you manage to make one of these options work! Keep us updated!

[ali-khosravi]

@adamg

I no longer can access the clusters using AiiDA which is currently completely blocking any progress in my project.

@adamg, maybe a naive suggestion. But just to be clear, I think still it’s possible to use aiida. You just need to do the manual part yourself in a separate shell, and make ssh farm work passwordless. (is that what you do by ssh -CX -o ServerAliveInterval=30 -fN fram ?) – Even if that means to buy the pain and manually enter otp to start the master connection.

Then you can just use core.ssh_auto as transport plugin. Maybe I’m wrong, but at least I don’t immediately see why that wouldn’t work.

[adamg]

Yes, the point of running ssh -CX -o ServerAliveInterval=30 -fN fram is to start a manual connection, for which I of course manually type in the 2FA code and password. It is no problem for me to do this step manually. Once this is done, ssh fram works passwordless as intended. The problem is that even though ssh fram from the shell works, AiiDA is not able to connect to the cluster. From what I understand this is due to the way paramiko is implemented. So for now I haven’t found any solution, not even a tedious one where I type in a code for each job which is submitted, on how to let AiiDA connect to the Sigma2 clusters.

[giovannipizzi]

Hi @adamg, I still think the first thing you should try is to install this and see if this works, as some people reported it was working, and this is the simplest thing to try without having to code new things in python. Let us know if you try it and it works!

[ali-khosravi]

Ok, I see now
apparently there’s no python libraries for SSH that supports multiplexing as a client.

[adamg]

For anyone out there who might have the same problem, I did get this working in the end with the help of people from both this forum and Sigma2 support. Here is a step-by-step guide of what worked for my setup:

Step 1 - Add the following to .ssh/config

Host fram.sigma2.no
    User myusername
    HostName login.fram.sigma2.no
    ControlMaster auto
    ControlPath ~/.ssh/%r@%h:%p
    ForwardAgent yes

Step 2 - Initialize a master connection by running

ssh -CX -o ServerAliveInterval=30 -fN fram.sigma2.no

Step 3 - Install the mux enabled branch of paramiko:

Mux transport for client (rebased, the second)

Step 4 - Apply the patch that is pasted at the end of this post to aiida/transports/plugins/ssh.py

Step 5 - If modifying an existing installation of AiiDA, run the following commands in the aiida source directory to delete cache files. These will be regenerated during runtime will enable the changes made in the patch.

find /path/to/aiida -name "*.pyc" -delete
find /path/to/aiida -name "__pycache__" -type d -exec rm -r {} +

Step 6 - Reconfigure the aiida computer by running

verdi -p myprofile computer configure core.ssh mycomputer

and typing

/home/user/.ssh/myusername@login.fram.sigma2.no:22

when prompted for “Control path”

Step 7 - Test by running

verdi computer test mycomputer

This is the patch to be applied in step 4 (provided by Sigma2 support):

diff --git a/pyproject.toml b/pyproject.toml
index 0370771ff..7afac9e08 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -34,7 +34,6 @@ dependencies = [
   'kiwipy[rmq]~=0.8.4',
   'importlib-metadata~=6.0',
   'numpy~=1.21',
-  'paramiko~=3.0',
   'plumpy~=0.24.0',
   'pgsu~=0.3.0',
   'psutil~=5.6',
diff --git a/src/aiida/transports/plugins/ssh.py b/src/aiida/transports/plugins/ssh.py
index fb5d7dec7..f2ed33682 100644
--- a/src/aiida/transports/plugins/ssh.py
+++ b/src/aiida/transports/plugins/ssh.py
@@ -188,6 +188,14 @@ class SshTransport(BlockingTransport):
                 'non_interactive_default': True,
             },
         ),
+        (
+            'controlpath',
+            {
+                'prompt': 'Control path',
+                'help': 'Path to ControlMaster socket, usually in ~/.ssh',
+                'non_interactive_default': True,
+            },
+        ),
         # for Kerberos support through python-gssapi
     ]
 
@@ -327,6 +335,11 @@ class SshTransport(BlockingTransport):
         """Return a suggestion for the specific field."""
         return 'True'
 
+    @classmethod
+    def _get_controlpath(cls, computer):
+        """Return a suggestion for the control path."""
+        return '/home/<user>/.ssh/<user>@<host>:<port>'
+
     @classmethod
     def _get_load_system_host_keys_suggestion_string(cls, computer):
         """Return a suggestion for the specific field."""
@@ -495,16 +508,21 @@ class SshTransport(BlockingTransport):
             self._proxy = _DetachedProxyCommand(proxycmdstring)
             connection_arguments['sock'] = self._proxy
 
-        try:
-            self._client.connect(self._machine, **connection_arguments)
-        except Exception as exc:
-            self.logger.error(
-                f"Error connecting to '{self._machine}' through SSH: "
-                + f'[{self.__class__.__name__}] {exc}, '
-                + f'connect_args were: {self._connect_args}'
-            )
-            self._close_proxies()
-            raise
+        if connection_arguments.get('controlpath', None):
+            transport = paramiko.Transport((self._machine, connection_arguments.get('port', 22)), controlpath=connection_arguments.get('controlpath', None))
+            transport.start_client(timeout=connection_arguments.get('timeout', 60))
+            self._client._transport = transport
+        else:
+            try:
+                self._client.connect(self._machine, **connection_arguments)
+            except Exception as exc:
+                self.logger.error(
+                    f"Error connecting to '{self._machine}' through SSH: "
+                    + f'[{self.__class__.__name__}] {exc}, '
+                    + f'connect_args were: {self._connect_args}'
+                )
+                self._close_proxies()
+                raise
 
         # Open the SFTP channel, and handle error by directing customer to try another transport
         try:

[giovannipizzi]

Thanks for the report! Very useful to know that this solution works! It would be great if you could put a thumbs up or a comment in the corresponding PR of paramiko, saying that you are using it successfully and that it’s useful for you!