Munge operations

The slurmctld charm contains an internal etcd3 server that, among other uses, stores the Munge key.

The etcd server has authentication enabled by default, to protect the Munge key. In order to be able to retrieve the Munge key, an etcd account is required. OSD provides the Juju action etcd-create-munge-account on the slurmctld charm to automate the creation of such an account.

By default, etcd is started using plain HTTP. It is possible to use HTTPS by supplying the necessary certificates to slurmctld.

Creating an etcd account

To create an etcd account to query the Munge key requires a username and password to be specified:

$ juju run-action slurmctld/leader etcd-create-munge-account user="theusername" password="aVerySafePassword!" --wait
unit-slurmctld-7:
  UnitId: slurmctld/7
  id: "654"
  results:
    Stdout: |
      User theusername created
      Role munge-readers is granted to user theusername
    created-new-user: theusername
  status: completed
  timing:
    completed: 2022-05-12 17:57:44 +0000 UTC
    enqueued: 2022-05-12 17:57:41 +0000 UTC
    started: 2022-05-12 17:57:44 +0000 UTC

Querying the munge key

The etcd server is running on the slurmctld node and listening on the port 2379. The Munge key is stored at munge/key entry. The Munge key is binary data and is stored encoded in a base 64 representation. The examples below demonstrate how to query the secret key and decode them to be used.

Using Bash

It is possible to get the Munge key using Bash, with curl and jq. First, you need to get an access token for your username and then query the key:

user=theusername
pass=aVerySafePassword!
host=10.107.185.126
port=2379

token=$(curl -L -s -X POST "$host:$port/v3/auth/authenticate" -d '{"name":"'"$user"'", "password":"'"$pass"'"}' | jq .token | tr -d '"')
key=$(printf munge/key | base64)

munge_key=$(curl -L -s -X POST "$host:$port/v3/kv/range" -H "Authorization: ${token}" -d '{"key":"'$key'"}' | jq .kvs[0].value | tr -d '"' | base64 -d)

As an example, running the above commands will result in:

$ echo $munge_key # it is base64 encoded
+CzBbB5c+WduA0bSNb5x6+U3Mj/QuladAFH5vqyyQyvyA5YUcUikz64YEakqB852D7Ml5jmCOuxpa0VXoYMeiaRFRw2WFyR+6H66buQeCBClgfk4t0QQKKRZsaPjVojkmhoNTgQz/8mseHDnrOpxyTEsFeAw9kMzhraK1fvHOgMbDDDnkmN1oOscNhBmnreQmcr+gy2NliGiRjCjyMh6TgwJqQCBU/4LCSs+cetbvKLMdj+7X8PbYpSALx8x4JXUjNj9sv1jLLu3f684P/G3unB709hhNvj3NoSPTZ68FhgjxqUFtXF91/Dce0hJForv9EAwgyTHmCJ+WNUI5ag+1P454nWFYqJkXTSX5gvPYBCwTGbOAjgLdyPa+SGOXeHutpCJtPjbpOtL9PkP+T5+mQrgDE1KjkDYLdlIyobVNUXfziPH82uSZft8ZRuke+vP+GpExSWsdj7sYuVAhUu9Utkfiyy1hVdmKoXeppJopjnj2J+Zmaa8/qvxYXNhmrbPdp8hadSdu2XRrNGu0AbYAGUsLrvSOSKc9J8+tkjFa5Qzcn8xR8HSWK12/mjF941NyCxDLqkfD6066KmL3ARqxdU7qVjZSMlIF8UzCiKIH2WDHkZa5LIV7Cj7ZxUTaAIjdAxmwr4ajnRqK1GFI736F+BqTWLRAuXM18S4r7KDjBYSUMNiXdL7qqY3Ao/LkniraGr4CxIBApicV0SuWfMafwoCb59oMNzyaCAiuI/JjFPqxD0Tn36ngtRHgdQnESuZeTSV68b7tOBc+MkzbUhBzmWy9TGSF2gsq6hoxIRj1oYdrTpwPQrfut4VLOkiWnJuJtaKX55z3c/o3bDZ1Ykd/unUL1vdDjlyo6VrNRu+/ijttiWAGm0PJOoMMe3PEqBjDBJAHdudlyez+bsUvH+Mm3Vnuu1gIlQ4vzxNcX1kUQC/8Mef7LjaRhD1DwUhPFw0E/AEhW0wKRuPD5vEGU1nlJLZbGcEy5x6I0qhyV/pr/SbLj5W6fTuQ5XVoHbKe5fh4U7iX7JDb1UlRZWoqZ9jbHqoWwwwkTe1VEr+tXtDFU4PJ8aEcyO4qRrSibKfXd+EYWJQ0Y+BRt+iRohXjy5CV9PMctwuciZhWMP0DGoYPfsObqoCLgLF56Gu6uVOhyC0AFrEm2LAOWzVlsxENCK4jYMtznwrRV4mwPVsFLZqCVIaAwW/KDfse7Bs2Y8RcA6Jl/Rv/kk2byyFhy230GqMcXiNEtCCViJvC/H6+V58IygJkpftrncFXafQ0u+ti/4U3ZRDD4Hfy3tepKxo9Sj9p8BBHZT7jVFYsQh3TVJqZow5lPRuumM0rz2k4izC9nW/TxGe/bmvPGzQxfQDYiEgQg==
# decode the key and save it to a file:
$ echo $munge_key | base64 -d > munge.key

Note

The etcd key must be base64 encoded when using curl.

Note

If the server is using HTTPS with a self-signed certificate, or a non-public Certificate Authority (CA), you need to supply the public certificate in the curl commands by adding --cacert path/to/cert.crt.

Using Python

To get the Munge key in Python, we suggest using the etcd3gw as a starting point:

from etcd3gw.client import Etcd3Client
from etcd3gw.exceptions import Etcd3Exception


class Etcd3AuthClient(Etcd3Client):
    """Handle etcd3 requests with auth."""
    def __init__(self, host='localhost', port=2379, protocol="http",
                 ca_cert=None, cert_key=None, cert_cert=None, timeout=None,
                 username=None, password=None, api_path="/v3/"):
        """Initialize class."""
        super(Etcd3AuthClient, self).__init__(host=host, port=port,
                                              protocol=protocol,
                                              ca_cert=ca_cert,
                                              cert_key=cert_key,
                                              cert_cert=cert_cert,
                                              timeout=timeout,
                                              api_path=api_path)
        self.username = username
        self.password = password

    def authenticate(self):
        """Authenticate the client."""
        if 'Authorization' in self.session.headers:
            del self.session.headers['Authorization']

        response = super(Etcd3AuthClient, self).post(
            self.get_url('/auth/authenticate'),
            json={"name": self.username, "password": self.password}
        )

        self.session.headers['Authorization'] = response['token']

    def post(self, *args, **kwargs):
        """Wrap the internal post function with authentication."""
        try:
            return super(Etcd3AuthClient, self).post(*args, **kwargs)
        except Etcd3Exception as e:
            if self.username and self.password:
                print("# etcd: Might need to (re)authenticate: %r:\n%s",
                      e, e.detail_text)
                self.authenticate()
                return super(Etcd3AuthClient, self).post(*args, **kwargs)
            raise


client = Etcd3AuthClient(host="10.107.185.126",
                         username="theusername",
                         password="aVerySafePassword!")
client.authenticate()
munge_key_encoded = client.get(key="munge/key")[0]
munge_key = b64decode(munge_key_encoded)

Note

The Python3 code does not automatically identify if the connection uses HTTP or HTTPS. By default, the library assumes HTTP. If the server is using HTTPS you need to specify protocol="https" when instantiating the client:

client = Etcd3AuthClient(host="10.107.185.126", protocol="https",
                         username="theusername",
                         password="aVerySafePassword!")

Configuring HTTPS

To use HTTPS for the connection, you need to supply the TLS certificates to slurmctld-charm via Juju Configuration options. You need to supply the public certificate (a .crt file) and the private key (a .key file). Additionally, you must provide the Certificate Authority's (CA) public certificate (another .crt file) if using self-signed certificates or a non-public CA.

To configure these certificates:

# the public and private TLS files
$ juju config slurmctld tls-cert="$(cat tls.crt)"
$ juju config slurmctld tls-key="$(cat tls.key)"

# the optional CA public certificate
$ juju config slurmctld tls-ca-cert="$(cat CA.crt)"