A pure Go IPMI v2.0 remote console.
License: GNU Lesser General Public License v3.0
This project implements an IPMI v2.0 remote console in pure Go, to interact with BMCs.
All section references in the code use the following documents:
Contributions in the form of bug reports and PRs are greatly appreciated.
Please see CONTRIBUTING.md for a few guidelines.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
It would be nice to use log.Fatal() here (and in the 3 other spots in that function which are treated as irrecoverable errors):
Lines 81 to 82 in 30abb01
When testing new versions, I like to loop through my BMCs to see whether this library can connect to them w/o errors (FYI, as of #51 (comment), they all work! 🎉). I'll work on getting approval for sending PRs in the future.
I am trying to adapt the example in ./cmd/describe/ to connect to a BMC using cipher suite 17. I had hoped that this diff would do the trick:
diff --git a/cmd/describe/main.go b/cmd/describe/main.go
index 4b02a03..af51e30 100644
--- a/cmd/describe/main.go
+++ b/cmd/describe/main.go
@@ -40,7 +40,7 @@ func main() {
ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
defer cancel()
- machine, err := bmc.Dial(ctx, *argBMCAddr)
+ machine, err := bmc.DialV2(*argBMCAddr)
if err != nil {
log.Print(err)
return
@@ -72,10 +72,21 @@ func main() {
printSystemGUID(guid)
}
- sess, err := machine.NewSession(ctx, &bmc.SessionOpts{
- Username: *flgUsername,
- Password: []byte(*flgPassword),
- MaxPrivilegeLevel: ipmi.PrivilegeLevelUser,
+ sess, err := machine.NewV2Session(ctx, &bmc.V2SessionOpts{
+ SessionOpts: bmc.SessionOpts{
+ Username: *flgUsername,
+ Password: []byte(*flgPassword),
+ MaxPrivilegeLevel: ipmi.PrivilegeLevelUser,
+ },
+ AuthenticationAlgorithms: []ipmi.AuthenticationAlgorithm{
+ ipmi.AuthenticationAlgorithmHMACSHA256,
+ },
+ IntegrityAlgorithms: []ipmi.IntegrityAlgorithm{
+ ipmi.IntegrityAlgorithmHMACSHA256128,
+ },
+ ConfidentialityAlgorithms: []ipmi.ConfidentialityAlgorithm{
+ ipmi.ConfidentialityAlgorithmAESCBC128,
+ },
})
if err != nil {
log.Print(err)However, when I try to use that, I get no response from the BMC for my first IPMI message:
$ ./describe --username=X--password=X X
2022/04/27 13:46:45 connected to 10.X.X.X:623 over IPMI v2.0
ASF Presence Pong capabilities:
IPMI: true
ASF v1.0: true
ASF security exts: false
DASH: false
DCMI: true
Channel Authentication Capabilities:
Channel: 0x1(Implementation-specific)
Extended: true
SupportsV2: true
K_G configured: false
Per-message auth: true
User-level auth: false
Non-null usernames: true
Null usernames: false
Anon login: false
OEM: 0(Unknown)
System:
GUID: X
2022/04/27 13:46:46 failed to get device id: read udp 10.X.X.X:35489->10.X.X.X:623: i/o timeoutI've tracked it down to an issue with the authentication algorithm. If I use HMACSHA1 instead of HMACSHA256 (while leaving the integrity and confidentiality algorithms unchanged), then I can connect just fine.
This is an issue because I am trying to use this library to connect to a new family of BMCs which requires the use of more modern ciphers. Currently I'm just falling back to my older, uglier solution of using ipmitool because when I use ipmitool -I lanplus -C17 I can connect without issues.
It's more flexible than you'd imagine: https://github.com/golang/go/wiki/Iota
Howdy. I appreciate the apparent large amount of work you've put into this project.
Would it be possible to add in-band (i.e. local) support? I ask because I'm trying to add functionality via Golang to issue BMC license activations locally for Super Micro. (Yes, their IPMI requires licensing for some features.)
It seems that it can at least be done remotely with RCMP+ (which I found out via a packet dump...), but it'd be a boon to datacenter operations if they're able to do it from a crashcart or before network is provisioned to the BMC.
I'm unsure about Windows (multi-platform infra is just... a joy) but it seems Linux provides certain character devices (/dev/ipmi0, /dev/imb, etc.) that allow for pretty easy in-band management.
If you do decide to support this, the IPMIUtil source may be a good starting point as it supports both remote and in-band for a multitude of OSes and may provide some hints.
Do we only increment the counter if we successfully decode the response? If so, needs changing, as decoding the response layer will almost always only work when completion code is normal.
And possibly v1.0 if you support it - check the layer formats vs. v1.1 and v1.5, which seem closer together.
Probably better represented as a nested bullet list for clarity.
Line 84 in e083998
I am testing out a new version of a BMC. This library is giving different readings from ipmitool. In the changelog for this version, they claim that ipmitool>=1.8.18 is required because "the issues with the older ipmitool was that the LUN of the sensor was not passed around with the sensor number".
Might this library have a similar issue as the older versions of ipmitool which would explain the discrepancy? I would be very willing to believe that the behavior exhibited by this BMC and by ipmitool is not according to spec.
It’s in Git - can restore it later. The current unusable implementation is the worst of both worlds.
name is generic; how about bmc_command_attempts_total{command="Get Device ID"} 1234?
Option to turn on dumping of each layer, as well as underlying hex.
Hello!
I was testing your code on my Supermicro card (Here is the output):
Sensors:
CPU1 Temp 30C
CPU2 Temp 44C
System Temp 20C
Peripheral Temp 26C
PCH Temp 43C
P1-DIMMA1 TEMP 19C
P1-DIMMA2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMA3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMB1 TEMP 20C
P1-DIMMB2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMB3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMC1 TEMP 22C
P1-DIMMC2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMC3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMD1 TEMP 21C
P1-DIMMD2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P1-DIMMD3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMME1 TEMP 25C
P2-DIMME2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMME3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMMF1 TEMP 27C
P2-DIMMF2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMMF3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMMG1 TEMP 24C
P2-DIMMG2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMMG3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMMH1 TEMP 29C
P2-DIMMH2 TEMP no reading/missing (response must be at least 2 bytes, got 0)
P2-DIMMH3 TEMP no reading/missing (response must be at least 2 bytes, got 0)
FAN1 2025RPM
FAN2 2100RPM
FAN3 2025RPM
FAN4 no reading/missing (response must be at least 2 bytes, got 0)
FAN5 no reading/missing (response must be at least 2 bytes, got 0)
FANA 3975RPM
VTT 0.992V
CPU1 Vcore 0.848V
CPU2 Vcore 0.9440000000000001V
VDIMM AB 1.488V
VDIMM CD 1.488V
VDIMM EF 1.488V
VDIMM GH 1.488V
3.3V 3.168V
+3.3VSB 3.3120000000000003V
5V 4.992V
Chassis Intru not analog
DCMI Capabilities:
Major version: 1
Minor version: 1
Supports pwr mgmt: true
DCMI Mandatory Platform Attributes:
Max SEL entries: 512
Temp sampling freq: 0s
DCMI Power Average Time Periods:
5s
15s
30s
1m0s
3m0s
7m0s
15m0s
30m0s
1h0m0s
And i got some questions:
| PS1 Status | | Presence detected. Power Supply Failure detected. Power Supply input lost (AC/DC).
| PS2 Status | | Presence detected.
Hi,
first of all, thank you so much for this great library!
We want to control some chassis (for instance power-up).
We currently face an issue similar to #38, where we get a Insufficient Privileges reply (even though the MaxPrivilegeLevel is correctly set):
sess, err := machine.NewSession(ctx, &bmc.SessionOpts{
Username: c.Username,
Password: []byte(c.Password),
MaxPrivilegeLevel: ipmi.PrivilegeLevelOperator,
})If we issue the same command from ipmi-chassis (FreeIPMI package), everything works fine:
ipmi-chassis -D LAN_2_0 -h XXX.XXX.XXX.XXX -u user -p secret -l user --chassis-control=POWER-UP
It seems that ipmi-chassis sends a 0x3b - Set Session Privilege Level command at the start, which is not available in this library yet.
According to the spec, page 88
All sessions start off at User Level privilege. It is necessary to issue a Set Session Privilege to raise the
operating privilege level before commands that required higher privilege can be executed. The maximum
operating privilege for a session is determined by Privilege Limits that are set both for the user and for the
overall channel. The more restrictive setting of the User Privilege Limit and the Channel Privilege Limit sets
the maximum operating privilege available for a session.
So I think this command should be added to the library and automatically sent if a command needs a privilege higher than User.
What to do you think about this ?
To reach this conclusion, we captured the packets sent by this library and ipmi-chassis using tcpdump. We then analyzed this .pcap file, however the commands are encrypted.
Since we know the password (and since the IPMI protocal has no forward-secrecy - we are lucky :-), we were able to very slowly decipher some packets using this guide: https://github.com/beingj/hash/blob/master/RMCP%2B%20Packet%20decrypt%20and%20authcode.org . It works, but involves a lot of manual (and error-prone) steps and gives us only the payload as []byte (which we now need to decode to the proper command and arguments).
To ease such analyze, we thought about writing a pcap analyzer, using github.com/google/gopacket/pcap and your library (this would also partially address #5 & #18, if integrated in this library ;).
In order to achieve this, we must be able to decode RAKP Message 1 packets, which is not available yet (PR is coming ;).
I now have a working prototype, up to the key construction.
However I am now struggling with gopacket to add the freshly made decipherLayer available to the gopacket analyzer.
If you think that such pcap analysis tool would be a nice addition to you repo, I can make a draft PR to show you where I am stuck (this is my first time working with github.com/google/gopacket/pcap)
User:
failed to get power reading: data too short for body code
Administrator:
failed to get power reading: 0-byte power reading response; this is known to happen when the BMC is not connected to the power supply
Currently it’s overly pessimistic - e.g. if a session expires and we fail to close it, there’s nothing we can do. It’ll still show up as a command execution failure.
SendCommand() can only record a normal completion code. Issue at the moment is we aren't sure whether the code is a sentinel or valid - returning ipmi.CompletionCodeUnspecified is a code smell. Need to split sendCommand() after ensuring the message is there, returning a code and err: code is only valid if err == nil. Can then use a separate method to parse the response.
Hi, wondering if you support Dell BMCs? when I run ChassisControl against Dell machines (r6525, r640) i get received non-normal completion code: 0xd4(Unknown)
Needs to be at minimum RTT of "furthest" BMC, plus a few seconds to allow it to respond. The impact of setting it too low is corruption. Effects can be reduced by lowering the latency between the app and BMC, however the main constraint is slow BMCs rather than the network.
When I compare the results of ipmitool sensor list with the list of sensors in the result from RetrieveSDRRepository(), I see many more sensors returned by ipmitool. They are all "discrete" (binary?) sensors - here is an example:
Presence | 0x0 | discrete | 0x0180| na | na | na | na | na | na
From a printf I added, it looks to me like the filtering happens on this line:
Line 91 in 455e487
Is there any way to list those filtered sensors using this library rather than discarding them?
We currently - accidentally - use cipher suite 3 (RAKP-HMAC-SHA1/HMAC-SHA1-96/AES-CBC-128). 17 (RAKP-HMAC-SHA256/HMAC-SHA256-128/AES-CBC-128) is newer, but less compatible.
Wireshark does not show all fields, and cannot decrypt. We can do both. Input is the packet source and password; from that, the decrypted (if necessary) packet stream is printed. We print as much as possible about unknown commands - can at least get the message layer, then print spaced hex for what we don’t know.
I noticed today that I was failing to scrape a lot of nodes with errors like this:
create session error: none of the provided cipher suite options were supported by the BMC
When I switched from NewSesssion to NewV2Session explicitly setting CipherSuites: []ipmi.CipherSuite{ipmi.CipherSuite17}, things got much better! I can now scrape ~260 nodes intead of ~180 nodes.
[17,3], so RetrieveSupportedCipherSuites should have successfully determined that suite 17 was ok to use.RetrieveSupportedCipherSuites once, it puts the BMC into a new state where your library can now run RetrieveSupportedCipherSuites without errors. I don't understand that.RetrieveSupportedCipherSuites, useful error messages such as RAKP2 HMAC fail (this indicates the BMC is using a different password) are masked, and the library prints an error about unsupported cipher suites inteadIt's harder to forget to call .Close() when you are given a function explicitly when dialling a BMC or creating a session.
machine, close, err := bmc.DialV2(...)
if err != nil {
return err
}
defer close() // not machine.Close()
...
I want to execute date comand bmc instance but not able to do. Can you please help?
I tried like this
type DateCommand struct{}
func (c *DateCommand) Name() string {
return "date"
}
func (c *DateCommand) Operation() *ipmi.Operation {
return &ipmi.Operation{}
}
func (c *DateCommand) Request() gopacket.SerializableLayer {
return nil
}
func (c *DateCommand) Response() gopacket.DecodingLayer {
return nil
}
func (upirdu *Upirdusession) ExecuteCmdOnUPIbaremetalInstance() error {
argBMCAddr := upirdu.BMCHOSTNAME
username := upirdu.BMCUSERNAME
password := upirdu.BMCPASSWORD
cmd := &DateCommand{}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
defer cancel()
machine, err := bmc.Dial(ctx, argBMCAddr)
if err != nil {
o.Expect(err).NotTo(o.HaveOccurred())
}
defer machine.Close()
e2e.Logf("connected to %v over IPMI v%v", machine.Address(), machine.Version())
sess, err := machine.NewSession(ctx, &bmc.SessionOpts{
Username: username,
Password: []byte(password),
MaxPrivilegeLevel: ipmi.PrivilegeLevelOperator,
})
if err != nil {
o.Expect(err).NotTo(o.HaveOccurred())
}
defer sess.Close(ctx)
oy, err := sess.SendCommand(ctx, cmd)
if err == nil {
e2e.Logf("tttt :: %s :: %v", oy, oy.String())
}
return err
}
We indirectly consume this project when we build the Tinkerbell PBNJ binary.
go mod graph on PBNJ project:
github.com/tinkerbell/pbnj github.com/bmc-toolbox/[email protected]
github.com/bmc-toolbox/[email protected] github.com/gebn/[email protected]
However since it is licensed as LGPLv3, it causes compliance and legal issues and makes it incompatible for our use-case. I believe LGPLv3 also requires that libraries be dynamically linked to applications if the original license of the application is to be preserved (that is, if you statically link against a LGPL library, you must license the entire application code with LGPLv3). I was curious behind the reason for choosing a copyleft license like LGPL, and whether you've had any considerations to vend this project under more permissive licenses. Also could you suggest other options to maintain compatibility with licenses such as Apache-2.0 and MIT License?
Upper bounds may be too low
OEM ID and OEM aux data will be null "if no OEM authentication type available". The describe command shows this as 0(Unknown).
Currently implemented in V2Sessionless only.
The following code prints the v4 address when example.com has both A and AAAA records. This results in us connecting to dual stack BMCs over v4. We should aim for v6 and fall back to v4.
package main
import (
"net"
"log"
)
func main() {
addr, err := net.ResolveUDPAddr("udp", "example.com:623")
if err != nil {
log.Fatal(err)
}
log.Println(addr)
}Double check you always copy from the DecodeFromBytes slice input and never reference a subslice - it will change, and this will cause corruption.
Hi, I don't see boot device support in this library. This fork has added boot device support. https://github.com/baas-project/bmc/tree/GetSystemBootOptionsCmd. I'm not affiliated with this fork but I'm wondering if boot device support is something you'd be open to accepting? If so, I could see about contacting the maintainers to gauge interest in contributing here.
# TYPE bmc_command_failures_total counter
bmc_command_failures_total{command="Get Power Reading"} 1
bmc_command_failures_total{command="Get Sensor Reading"} 5419
The comment cannot be true as bmc_command_responses_total{code="0xcb(Unknown)"} is 5419. The command executes successfully, returning a bad status code, which is counted as a command failure. Isn't it more useful to include bad response codes - assuming we can accurately identify them?
It would be good to distinguish between timeouts, incorrect passwords (i.e. RAKP2 HMAC mismatch), and other errors.
Only decrement the gauge if Close() returns no error.
Hey @gebn , really like and appreciate this library. Thanks for all the work!
I'm trying to write an IPMI v2.0 simulator and wanted to see if you could help point me in the direction of any good documentation, besides the spec, by chance?
I'm following the general pattern of the v1.5 simulator found here, but having trouble following the v2 spec. Any help or guidance would be really appreciated!
The BMC is entitled to omit everything after the completion code (apart from the checksum), so the error should be able the non-normal completion code, not about the missing defining body code.
Is it valid to query multiple sensors in parallel (from multiple worker coroutines)? In the IPMI v2.0 spec, I see a section about "Previous Sequence Number Tracking". I realize that's probably for handling multi-packet responses, but I was hoping I'd be able to send multiple requests in parallel, and that this library would automatically filter the response packets by their sequence number. But it's not working. Sometimes I get errors like read udp 10.20.170.132:41744->172.22.6.24:623: i/o timeout, and once I got a big backtrace like this:
panic: crypto/cipher: input not full blocks
goroutine 38 [running]:
crypto/cipher.(*cbcEncrypter).CryptBlocks(0xc0002d0b40, 0xc0002b603d, 0x1c, 0x33, 0xc0002b603d, 0x1c, 0x33)
/usr/lib/go/src/crypto/cipher/cbc.go:59 +0x306
github.com/gebn/bmc/pkg/ipmi.(*AES128CBC).SerializeTo(0xc0002980c0, 0xa8ccd8, 0xc000290190, 0xc000070101, 0x0, 0x0)
/home/lyeager/go/pkg/mod/github.com/gebn/[email protected]/pkg/ipmi/aes_128_cbc.go:110 +0x27a
github.com/google/gopacket.SerializeLayers(0xa8ccd8, 0xc000290190, 0xc000290101, 0xc00007baf0, 0x5, 0x5, 0x0, 0x0)
/home/lyeager/go/pkg/mod/github.com/google/[email protected]/writer.go:210 +0xcd
github.com/gebn/bmc.(*V2Session).buildAndSend.func1(0xc00029cae0, 0xc00029cae0)
/home/lyeager/go/pkg/mod/github.com/gebn/[email protected]/v2session.go:173 +0x1e9
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer(0xc00007bc88, 0xa82dc0, 0xc00029cae0, 0x0, 0xa86470, 0xc00028c2e8, 0x0, 0x0)
/home/lyeager/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:55 +0x102
github.com/cenkalti/backoff/v4.RetryNotify(...)
/home/lyeager/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:34
github.com/cenkalti/backoff/v4.Retry(...)
/home/lyeager/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:28
github.com/gebn/bmc.(*V2Session).buildAndSend(0xc0002bc000, 0xa88480, 0xc00028e1e0, 0xa88870, 0xc0002b4240, 0xc0002d2480, 0x96f980)
/home/lyeager/go/pkg/mod/github.com/gebn/[email protected]/v2session.go:211 +0x3d9
github.com/gebn/bmc.(*V2Session).SendCommand(0xc0002bc000, 0xa88480, 0xc00028e1e0, 0xa88870, 0xc0002b4240, 0xc0002b0d00, 0x0, 0x0)
/home/lyeager/go/pkg/mod/github.com/gebn/[email protected]/v2session.go:120 +0x1ee
github.com/gebn/bmc.(*linearSensorReader).Read(0xc0002b4240, 0xa88480, 0xc00028e1e0, 0xa8f780, 0xc0002bc000, 0x0, 0x0, 0x0)
/home/lyeager/go/pkg/mod/github.com/gebn/[email protected]/sensor_reader.go:80 +0x64
main.poll.func1.2()
/home/lyeager/code/cluster-monitoring/services/bmc-sensor-exporter/main.go:211 +0x2ee
created by main.poll.func1
/home/lyeager/code/cluster-monitoring/services/bmc-sensor-exporter/main.go:222 +0x633
exit status 2
Is this a limitation of your library, or of the IPMI spec? It looks to me like you simply assume that the next UDP packet is the response to your question, rather than parsing the sequence number and doing a "UDP sliding window" or something like that.
bmc/internal/pkg/transport/transport.go
Lines 108 to 140 in cefdef9
Folks currently have to know to look in cmd.
This is a bit of a mess, but some people smarter than me have decided on a solution.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.