Hello. I have been using AiiDA very extensively in my PhD project to run DFT and MD simulations on the Norwegian HPC clusters (Sigma2) for more than two years now. Basically all my simulations and steps in setting them up are implemented as AiiDA jobs / calcfunctions, and all my post-analysis is implemented as AiiDA queries. Recently Sigma2 enabled 2-factor authentication, and I no longer can access the clusters using AiiDA which is currently completely blocking any progress in my project.
The general solution provided by Sigma2 to prevent having to type in a password each time is adding this to my ssh config file:
Host fram
User myusername
HostName login.fram.sigma2.no
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
and starting an initial connection manually which will be used by all subsequent connections:
This works for manual ssh connections, however it seems to me that this does not work in combination with AiiDA. I see this solution has been discussed in the past, and it is mentioned that paramiko does not support multiplexing: Access to cluster with two-factor authentication - #7 by giovannipizzi
Is there any known solution or workaround to get this working? I am a fairly advanced user of AiiDA and I am willing to try potentially less robust workarounds to get this working, since currently my only other alternative is to apply for HPC resources elsewhere.
Unfortunately, I think thatās technically not possible, Iām not aware of python implementations that allow to use the multiplexing ControlMaster approach (except workarounds that I donāt know how supported are, see e.g. this - if this works for you, one would need to wrap it in a transport plugin, supporting the key methods to copy/get files, execute commands etc).
I was also checking asyncssh that @ali-khosravi implemented a few weeks ago and will end up in the upcoming 2.7 release, but it seems that also that library does not support it, again for the design of it.
check with the supercomputer center if they would allow e.g. connections with MFA from a specific IP address you manage, or if they allow a jump host without MFA
check if they would allow to have a small VM inside their network, where you could install AiiDA and run from there - so you need to use 2FA to connect to the AiiDA machine, but then connections to submit are without 2FA
You could also check if they support tunneling and, when tunneling, you are not asked the password (e.g. ssh -L <SOMELOCALPORT>:localhost:22 <SUPERCOMPUTER> and then ssh to your localhost on port <SOMELOCALPORT>
I hope you manage to make one of these options work! Keep us updated!
@adamg, maybe a naive suggestion. But just to be clear, I think still itās possible to use aiida. You just need to do the manual part yourself in a separate shell, and make ssh farm work passwordless. (is that what you do by ssh -CX -o ServerAliveInterval=30 -fN fram ?) ā Even if that means to buy the pain and manually enter otp to start the master connection.
Then you can just use core.ssh_auto as transport plugin. Maybe Iām wrong, but at least I donāt immediately see why that wouldnāt work.
Yes, the point of running ssh -CX -o ServerAliveInterval=30 -fN fram is to start a manual connection, for which I of course manually type in the 2FA code and password. It is no problem for me to do this step manually. Once this is done, ssh fram works passwordless as intended. The problem is that even though ssh fram from the shell works, AiiDA is not able to connect to the cluster. From what I understand this is due to the way paramiko is implemented. So for now I havenāt found any solution, not even a tedious one where I type in a code for each job which is submitted, on how to let AiiDA connect to the Sigma2 clusters.
Hi @adamg, I still think the first thing you should try is to install this and see if this works, as some people reported it was working, and this is the simplest thing to try without having to code new things in python. Let us know if you try it and it works!
For anyone out there who might have the same problem, I did get this working in the end with the help of people from both this forum and Sigma2 support. Here is a step-by-step guide of what worked for my setup:
Step 1 - Add the following to .ssh/config
Host fram.sigma2.no
User myusername
HostName login.fram.sigma2.no
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
ForwardAgent yes
Step 2 - Initialize a master connection by running
Step 3 - Install the mux enabled branch of paramiko:
Step 4 - Apply the patch that is pasted at the end of this post to aiida/transports/plugins/ssh.py
Step 5 - If modifying an existing installation of AiiDA, run the following commands in the aiida source directory to delete cache files. These will be regenerated during runtime will enable the changes made in the patch.
Thanks for the report! Very useful to know that this solution works! It would be great if you could put a thumbs up or a comment in the corresponding PR of paramiko, saying that you are using it successfully and that itās useful for you!