Logging into clusters

The most common method to access a cluster is through a ssh tunnel. However, there are ways to access the HPC facilities through other mediums, such as Jupyter Notebooks, or web-based desktop environments.

SSH Access

An ssh connection provides a user with a terminal-only environment in which they can interface with the compute cluster. The terminal usually runs a Unix shell or the Bash interpreter, and text files can be edited via vim or nano. The user typically lands on a login node, which doesn’t necessarily have the same hardware as the compute nodes and often restricts activity to short compilation to avoid impacting other logged in users.

Some clusters provide X11 forwarding on their login nodes, in which case one can log in via ssh -X or ssh -Y and start GUI applications. Some clusters provide either port forwarding or reverse dynamic forwarding to communicate data between the remote cluster and the client, e.g. to run Jupyter Notebooks, in which case one can log in via ssh -L resp. ssh -R to exchange data. For Microsoft Windows and macOS operating systems, users need to find an app that supports the ssh protocol, for example putty. Refer to the cluster user guide to find which app is recommended.

An ssh connection is encrypted and requires authentication. Each cluster has its own security policy, and these policies are periodically updated to face new cybersecurity threats. For example, the Bee cluster require users to authenticate first on the ICP internal network, and then to authenticate on the cluster with a password. The HPC Vega supercomputer requires users to send their SSH public key.

Using ssh

In order to use ssh on Linux, open a new terminal with the Ctrl+Alt+Tab shortcut. Inside the shell, type the exact ssh command from the cluster user guide. For example, to connect to the Ant cluster:

ssh ant

and press Enter. If this is the first time you logged into the cluster on your workstation, ssh will ask to verify the legitimacy of the server. Follow the instructions in Verifying the host key. Then, ssh will try to authenticate you as a legitimate user. Several methods exist, for example large clusters rely on multi-factor authentication; ssh may ask you to type in a number communicated to you by SMS, or to type in a one-time password (OTP) from a smartphone app that expires every 30 seconds. Small university clusters like Ant prefer to opt for a password, which is easier to set up. In the case of a leak, the login/password pair is useless if the attacker cannot first log into the university network.

Verifying the host key

The first time a user logs into any remote machine, ssh will ask to verify the host key with a message similar to this:

The authenticity of host 'ant (129.69.120.30)' can't be established.
ED25519 key fingerprint is SHA256:luxPt965f5utw+7WkTvs9fJwMu93+vAktFFQA0WJHI8.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

The goal of this warning is to make sure users are not accidentally logging into a malicious server that spoofed the cluster IP address (man-in-the-middle attack) to steal credentials. The correct way to go past this warning is to copy-paste the fingerprint from a trusted source, such as a HTTPS web page from the cluster IT team. For example, the Ant cluster page shows a list of fingerprints from various encryption algorithms; since the message above asks for the ED25519 key fingerprint, one has to type “SHA256:luxPt965f5utw+7WkTvs9fJwMu93+vAktFFQA0WJHI8”. Do not visually compare the fingerprint and then type yes! It is relatively easy to forge the MD5 [Stevens et al., 2007, Stevens et al., 2009] or SHA1 [Leurent and Peyrin, 2020] of key fingerprints by leveraging weaknesses in these hash functions.

If the trusted source hash doesn’t match with the ssh hash, do not proceed with authentication! Instead, get in touch with the IT team and ask them about the discrepancy between the server and the web page. In exceptional cases, a host key will change, for example when hardware is replaced or when the cluster admin team rotates the SSH key (see e.g. GitHub’s RSA SSH host key rotation); in any case the trusted source should always be up-to-date, and a notification should be sent to the cluster mailing list. To remove the deprecated key fingerprint associated with a cluster, e.g. Ant, run the following command:

ssh-keygen -R ant

If you’re a cluster admin and need to rotate keys, here is how to update the trusted source with the new key fingerprints:

$ ssh user@cluster
$ ssh-keyscan localhost | ssh-keygen -E sha256 -lf -
3072 SHA256:JOg7saslfaqZdPVy8sTv2qoWy/cCFgTIvADhzj6cHfw localhost (RSA)
256 SHA256:/fY0bZIwZ6O6+5CWAvgL79+AoxMlOelhdb71ecskKfE localhost (ECDSA)
256 SHA256:luxPt965f5utw+7WkTvs9fJwMu93+vAktFFQA0WJHI8 localhost (ED25519)

2FA

Many clusters nowadays enforce multi-factor authentication, most often two-factor authentication (2FA).

Clusters are regular targets in cyber attack campaigns, because they provide a lot of computer power to mine cryptocurrencies and contain valuable intellectual property from industry users. Many German HPC clusters were affected by a cyber attack in 2020 [Krause, 2020] that left them offline for several weeks. These facilities now use 2FA. The ICP CI/CD fleet was the target of a cyber attack in 2021 [Codecov team, 2021] that attempted to exfiltrate credentials, tokens and keys of the institute IT infrastructure. While the firewall blocked the exfiltration attempt, all keys were rotated out of caution as soon as the attack was made public, and the entire continuous delivery pipeline was migrated to a cloud-based service.

SMS

SMS codes are the most widespread technique for 2FA.

TOTP

The time-based one-time password (TOTP) algorithm generates unique one-time passwords (OTPs) every 30 seconds, using a private key and the current time as input. TOTP can be managed by smartphone apps, token generators, or KeePassXC. The first two options have extensive documentation online.

KeePassXC can be used on laptops and desktop workstations. In the software, create a new entry, set the cluster name in the “Title” field, set the login in the “Username” field, the SSH address in the “URL” field, leave the “Password” field emtpy (very important! it’s pointless to store the password together with the OTP, because it completely defeats the purpose of having an OTP). Then right-click on the entry, hover over “TOTP”, click “Set up TOTP”, leave the default settings and enter the private key. The private key is obtained from the 2FA server. It may be communicated as a string, as a parameter in a URL (never open that url in a web browser!), and/or as a QR-code that can be decoded with either a smartphone, or with the zbarimg software (Ubuntu package zbar-tools).

Kerberos

A Kerberos token can be created by contacting a server and providing the user login and password. This token is stored on the workstation as clear text and is valid for 24 hours.