Logging#

OSD can forward internal logs to a centralized place using the built-in integration (a Juju relation) to Fluentbit.

Fluentbit#

Fluentbit is a lightweight and high-performance log processor and forwarder, and is available as a Juju charm: charm-fluentbit

OSD configures Fluentbit to automatically forward the following logs:

  • NHC logs (/var/log/nhc.log), only on the compute nodes (running slurmd charm)

  • internal Slurm logs:

    • /var/log/slurm/slurmctld.log on the nodes running slurmctld charm

    • /var/log/slurm/slurmd.log on the nodes running slurmd charm

    • /var/log/slurm/slurmdbd.log on the nodes running slurmdbd charm

Each log entry is enriched with:

  • cluster-name

  • hostname

  • service (one of nhc, slurmd, slurmdbd, slurmctld)

  • partition-name (only for nodes running the slurmd charm)

Deploy#

To deploy the fluentbit charm and relate it to the other components:

$ juju deploy fluentbit
$ juju relate fluentbit:fluentbit slurmd
$ juju relate fluentbit:fluentbit slurmdbd
$ juju relate fluentbit:fluentbit slurmctld

This configures everything needed to read, parse, and filter the log files.

Configuration#

Output#

The Juju relation between Fluentbit and the Slurm charms setup the source logs to be forwarded, but does not define the output. This has to be done either manually (via a Fluentbit charm configuration) or via a centralized logging system that can be related to it.

Fluentbit support many and multiple outputs. Please consult the online documentation for details: Fluentbit Outputs.

Graylog#

To configure Fluentbit to forward logs to a Graylog cluster running, first setup a Graylog GELF Input, then create a graylog.fluentbit text file with the following:

[{"output": [["name", "gelf"],
             ["match", "*"],
             ["host", "10.220.130.119"],
             ["port", "12201"],
             ["mode", "udp"],
             ["Gelf_Short_Message_Key", "log"]]}]

Modify the host, port, and mode accordingly.

Then, configure Fluentbit to use this:

$ juju config fluentbit custom-config="$(cat graylog.fluentbit)"