chenxiaolong / ipmi-fan-control

ipmi-fan-control

ipmi-fan-control is a program written in Rust to control the fans on Supermicro motherboards based on the readings of temperature sensors.

Note: This has primarily been tested on a 6028U-TR4T+, which uses the X10DRU-i+ motherboard. Also, only Linux and other Unix-like operating systems are currently supported.

2022-11-04 Update: This project is no longer in active development because I no longer have a Supermicro motherboard that requires more granular fan control. However, the project is feature complete and I will continue to do maintenance work, like updating dependencies and creating binary repos for new Linux distro releases. The underlying fan control IPMI commands have not changed across 5 generations of motherboards. It's unlikely that ipmi-fan-control will stop working on any currently supported motherboards.

2024-04-14 Update: I am archiving this repo and no longer plan to keep the binary repos up to date for new distro releases. Folks who want to continue using this project should build it from source. As mentioned in the previous update, this project is feature complete and is unlikely to suddenly stop working.

Installation

Prebuilt packages for Arch Linux, CentOS, Fedora, openSUSE, Debian, and Ubuntu are available from OBS. Please follow the instructions at the repo's landing page.

For other Unix-like systems, follow the instructions in the next section to build from source. Windows is currently not supported.

Building

This project depends on:

• the freeipmi suite of libraries (specifically, libfreeipmi and libipmimonitoring)
• pkg-config
• the Clang compiler (for generating Rust FFI bindings to the freeipmi libraries)
• the Rust compiler
• [optional] smartmontools (for querying HDD/SSD drive temperatures)
• [optional] hdparm (for querying Hitachi/HGST/WD drive temperatures while spun down)

These packages can be installed from the system package manager:

# Fedora
sudo dnf install freeipmi-devel pkgconf-pkg-config clang-devel cargo
# OpenSUSE
sudo zypper in freeipmi-devel pkgconf-pkg-config clang-devel cargo
# Arch Linux
sudo pacman -S freeipmi pkgconf clang cargo
# Debian-based distros
sudo apt install libfreeipmi-dev libipmimonitoring-dev pkg-config libclang-dev cargo

Then, to make a debug build, run:

cargo build

or to make a release build, run:

cargo build --release

To build Linux distro-specific packages, first build the corresponding source package:

# SRPM for RPM-based distros
./dist/build_source_package.py -t srpm
# PKGBUILD for Arch Linux
./dist/build_source_package.py -t pkgbuild
# dsc for Debian-based distros
./dist/build_source_package.py -t debian

and then use the distro's standard utilities for building the binary packages. The source packages will be placed in dist/output/.

Running

If ipmi-fan-control was installed from a package, update /etc/ipmi-fan-control.toml to match the desired configuration and then enable and start the ipmi-fan-control systemd service.

If built from source, make a copy of config.sample.toml and update the values to match your server's configuration. Then, run ipmi-fan-control with:

# Debug
sudo ./target/debug/ipmi-fan-control --config config.toml
# Release
sudo ./target/release/ipmi-fan-control --config config.toml

ipmi-fan-control's Issues

Use libfreeipmi directly instead of parsing ipmitool shell output

ipmi-fan-control currently invokes a long-lived ipmitool shell session for sending commands and parsing information out of the human-readable output. While it works decently well, it's quite fragile. It depends on ipmitool's output format never changing and IPMI errors currently aren't reliably detected.

With libfreeipmi, we'll have a proper library to interact with. I don't intend to create a full Rust binding for libfreeipmi though--it's too much work. We'll generate FFI wrappers via bindgen + pkg-config (for finding libfreeipmi) and only use the few functions we need.

The configuration will remain exactly the same for local connections (the default setup). The remote session configuration will change though. Instead of providing command line arguments, like:

[sessions]
"foobar" = ["-I", "lanplus", "-H", "<host>", "-U", "<username>", "-P", "<password>"]

it'll be more like:

[sessions]
"foobar" = { type = "remote", host = "<host>", username = "<username>", password = "<password>" }

I'll consider adding backwards compatibility for the old syntax with ipmitool's CLI args, but at most, it'll only support -I lanplus, -H, -U, and -P.

Version 0.4.0 doesn't read HDD temperature via SMART

It just fails, no actual attempts are visible in the log:

[INFO  ipmi_fan_control] [default] Original fan mode: Full
[INFO  ipmi_fan_control] [default] Starting loop for IPMI zones [0]
[INFO  ipmi_fan_control] [default] Starting loop for IPMI zones [1]
[DEBUG ipmi_fan_control] [default] Zone 0: zone_temp=33C, dcycle_cur=100%, dcycle_new=21%
[INFO  ipmi_fan_control] [default] Setting zone 0 duty cycle to 100%
[INFO  ipmi_fan_control] [default] Setting zone 1 duty cycle to 100%
[ERROR ipmi_fan_control] Failed all 3 attempt(s); last attempt error: Block device has no temperature reading: "/dev/sda"

Improvment ideas

I've been using https://github.com/khorton/nas_fan_control for a while, and have always disliked that it was Perl. I had been thinking of writing my own in Rust to learn. Today I was poking around github and to my surprise found this, which is in the direction I wanted to go anyway.

I have some ideas for future improvements and wanted to see what you think

• Support for reading the HDD temperatures and averaging them as a sensor (either using smartctl or some other method of reading the data, along with a way to exclude drives)
• per zone interval. HDDs don't need to react as fast (say 10 seconds) as a CPU zone (say 1 second)
• curve support instead of steps. One thing I miss about a consumer motherboard is the ability to set a fan curve. Steps are pretty rigid, and a curve would allow for finer control and better ability to reduce fan noise. I imagine this would have the same data structure as the steps but instead of a upper and lower bound check the current temperature would control the duty cycle depending on where it fell between the curve steps

This project looks promising, once I find time to learn a little rust I may try to contribute my ideas if you want them in your project

interval=1820 gives an error

I want to give a large interval (more than 30 minutes) to the HDD zone so it can play nice with hd-idle. Unfortunately, it bombs out

ERROR ipmi_fan_control] Failed to parse config "/etc/ipmi-fan-control.toml": invalid value: integer `1820`, expected u8 for key `zones.interval` at line 73 column 12

tune log level/verbosity

Hey, thanks for great software.
works wonderfully with my H12DSi-NT6
I added package to gentoo linux repo https://packages.gentoo.org/packages/sys-apps/ipmi-fan-control

my question is:

currently default loglevel is INFO

Sep 14 14:37:19 chimera ipmi-fan-control[1482167]: [2022-09-14T21:37:19Z INFO  ipmi_fan_control] [default] Zone 1: zone_temp=53C, dcycle_cur=25%, dcycle_new=26%
Sep 14 14:37:20 chimera ipmi-fan-control[1482167]: [2022-09-14T21:37:20Z INFO  ipmi_fan_control] [default] Zone 0: zone_temp=54C, dcycle_cur=35%, dcycle_new=38%

it prints status every interval.

but if I launch it with RUST_LOG=warn envvar - it's completely silent.

what I propose it change startup/shutdown messages to warn, so users who run it with warn loglevel still see startup message and shutdown message when it restores BMC settings, but not every loop interval report.
WDYT about such change?

2 zone example for Supermicro X9/X10/X11 boards

I would appreciate an example for a two zone system. This is common for Supermicro X9/X10/X11 as described here.

In my case I have Zone 0 has the CPU/Rear case fans; and Zone 1 has the front rack case fans for HDD. I am note sure what to put in the file. For instance, whether i need two step definitions or not. I was previously using IPMIFanControl.pl and could see what it is doing in the its log. This program does not output much of a log.

The config i used is listed below. Is this correct?

[[zones]]
ipmi_zones = [0]
interval = 5
sources = [
    { type = "ipmi", sensor = "CPU1 Temp" },
    { type = "ipmi", sensor = "CPU2 Temp" },
    { type = "file", path = "/sys/class/thermal/thermal_zone0/temp" },
]
steps = [
    { temp = 30, dcycle = 30 },
    { temp = 70, dcycle = 70 },
]
[[zones]]
ipmi_zones = [1]
interval = 5
sources = [
    { type = "smart", block_dev = "/dev/disk/by-id/disk1" },
    { type = "smart", block_dev = "/dev/disk/by-id/disk2" },
]
aggregation = { type = "average" }
steps = [
    { temp = 30, dcycle = 30 },
    { temp = 70, dcycle = 70 },
]
[sessions]

unable to send RAW command

sudo ipmi-fan-control --config /etc/ipmi-fan-control.toml
[2022-05-07T00:23:10Z ERROR ipmi_fan_control] IPMI error: Failed to parse line 'Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x45 rsp=0xc1): Invalid command
': invalid digit found in string

ipmi-fan-control leaves thousands of hdparm zombies

[root@home]/home/stigger# ps x|grep hdparm|wc -l
55870

Create Debian dpkg build scripts

This morning I was researching deb packages and found cargo deb. I was able to get it set up very quickly.

When coming here to talk about how you feel about using something like cargo deb in conjunction with something like Alien to convert the deb into an rpm I noticed you had committed a build script intended to be used to create system packages.

I presume you would prefer to use that build script to create the deb package over using a tool like cargo deb correct?

› cargo deb --target=x86_64-unknown-linux-gnu
# build stuff
› dpkg -c target/x86_64-unknown-linux-gnu/debian/ipmi-fan-control_0.3.0_amd64.deb     
drwxr-xr-x 0/0               0 2020-12-05 10:12 usr/
drwxr-xr-x 0/0               0 2020-12-05 10:12 usr/share/
drwxr-xr-x 0/0               0 2020-12-05 10:12 usr/share/doc/
drwxr-xr-x 0/0               0 2020-12-05 10:12 usr/share/doc/ipmi-fan-control/
-rw-r--r-- 0/0             171 2020-12-05 10:12 usr/share/doc/ipmi-fan-control/copyright
drwxr-xr-x 0/0               0 2020-12-05 10:12 lib/
drwxr-xr-x 0/0               0 2020-12-05 10:12 lib/systemd/
drwxr-xr-x 0/0               0 2020-12-05 10:12 lib/systemd/system/
-rw-r--r-- 0/0             205 2020-12-05 10:12 lib/systemd/system/ipmi-fan-control.service
drwxr-xr-x 0/0               0 2020-12-05 10:12 usr/bin/
-rwxr-xr-x 0/0         3312056 2020-12-05 10:12 usr/bin/ipmi-fan-control
drwxr-xr-x 0/0               0 2020-12-05 10:12 etc/
-rw-rw-r-- 0/0            3686 2020-12-05 10:12 etc/ipmi-fan-control.toml

diff --git a/Cargo.toml b/Cargo.toml
index 910ad44..4fb8d78 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -8,6 +8,17 @@ version = "0.3.0"
 authors = ["Andrew Gunnerson <[email protected]>"]
 edition = "2018"
 
+
+[package.metadata.deb]
+assets = [
+    ["target/release/ipmi-fan-control", "usr/bin/", "755"],
+    ["config.sample.toml", "etc/ipmi-fan-control.toml", "664"],
+]
+conf-files = ["etc/ipmi-fan-control.toml"]
+maintainer-scripts = "dist/debian/"
+systemd-units = { enable = true }
+
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
 
 [dependencies]
diff --git a/dist/debian/ipmi-fan-control.service b/dist/debian/ipmi-fan-control.service
new file mode 100644
index 0000000..f0868c5
--- /dev/null
+++ b/dist/debian/ipmi-fan-control.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=SuperMicro IPMI fan control daemon
+
+[Service]
+ExecStart=/usr/bin/ipmi-fan-control -c /etc/ipmi-fan-control.toml
+Restart=on-failure
+KillMode=process
+
+[Install]
+WantedBy=multi-user.target

CPU issues with ipmi-fan-control

Over time (hours) something in ipmi-fan-control starts eating cpu causing the system to feel sluggish. While it doesn't eat a lot of actual CPU time (Load average 1.75 1.48. 1.35) opening htop and trying to navigate around takes several seconds to navigate between each row. After I kill and restart ipmi-fan-control my system is fine, and things like htop are immediately more responsive.

When I leave htop open for a while, I occasionally start to see zombie processes from smartctl. Could this have anything to do with it?

I tried switching from -A to -l scttempsts to see if the json payload size had anything to do with it, but that does not appear to be the case.

Anything I can do to debug this issue?

No standby with hd-idle

I have some hot disks which I would like to go on standby when not used. I am using hd-idle for this purpose. The disk temperature in ipmi-fan-control is being read through the "smart" source.

Unfortunately, fan control script keeps the disk awake and hd-idle is unable to make it sleep. On the other hand if the disk is on standby and the script is started afterwards it correctly does not wake it up.

One way to fix this is to give a larger interval to fan-control script compared with the timer in hd-idle. Thus hd-idle will always get higher priority than the fan-control.

If you have some other thoughts on this, please do share.

PS: I am assuming when the disk goes on standby it goes cold.

Fan keeps cycling back to full blast RPMs or 100% duty cycle

I'm not sure what's causing this but every couple minutes or so my fans go full blast before the script kicks in.

` [[zones]]

    ipmi_zones = [0]
    interval = 5
    sources = [
        { type = "ipmi", sensor = "CPU Temp" },
        { type = "file", path = "/sys/class/thermal/thermal_zone1/temp" },
    ]
    steps = [
        { temp = 30, dcycle = 10 },
        { temp = 35, dcycle = 15 },
        { temp = 40, dcycle = 20 },
        { temp = 50, dcycle = 30 },
        { temp = 60, dcycle = 50 },
    ]
    [[zones]]
    ipmi_zones = [1]
    interval = 5
    sources = [
        { type = "ipmi", sensor = "PCH Temp" },
        { type = "file", path = "/sys/class/thermal/thermal_zone1/temp" },
    ]
    steps = [
        { temp = 30, dcycle = 10 },
        { temp = 35, dcycle = 15 },
        { temp = 40, dcycle = 20 },
        { temp = 50, dcycle = 30 },
        { temp = 60, dcycle = 50 },
    ]
    [sessions]`