ironcore-dev / oob

oob

Overview

The operator is doing a "light" lights out management of the hardware deployed in an Ironcore installation. It is handling the user management, power states, identification and service data collection via/of the baseboard management controllers by leveraging different industry standards and protocols (e.g. Redfish/IPMI).

It is scanning the known IP Addresses/MAC Addresses from the out-of-band network and handling only specific endpoints that are determined by a filtering mechanism that is preconfigured. The filtering mechanism is used to assign/find the preferred protocol and the proper manufacturer default credentials.

After proper credentials have been found, a new set of credentials is created, existing state for the device will be enforced and all service information is read. A new OOB object is created or updated if already existing.

Installation, using and developing

To find out more, please refer to documentation folder docs

Contributing

We'd love to get feedback from you. Please report bugs, suggestions or post questions by opening a GitHub issue.

How it works

This project aims to follow the Kubernetes Operator pattern

It uses Controllers which provides a reconcile function responsible for synchronizing resources untile the desired state is reached on the cluster

License

Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

oob's Issues

[rl-assigned_teams-3] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-assigned_teams-3
Explanation: Does it have enough admins on GitHub? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Remove fscom support

We'd like to remove fscom support as we're not going to use this anymore.
Also, get rid of expect and all its dependencies.

Comply to SAP reuse policy

firmware version field in status

can we have the firmware version field back ?

so we know what version of firmware is on the box?

(i know we had that before and removed it for not being used, but we will need it for updates, sorry)

[rl-vulnerability_alerts-1] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-vulnerability_alerts-1
Explanation: Are vulnerability alerts enabled? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Collecting the additional stats of the machine in oob-object

[rl-vulnerability_alerts-1] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-vulnerability_alerts-1
Explanation: Are vulnerability alerts enabled? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

HP server reset type 'GracefulShutdown' is not supported when spec set to power off

What would you like to be added:
HP server has diverted from the redfish standard and as a result power state off is not working as usual.

Dell and Lenovo support reset type 'GracefulShutdown' , when spec set to power off for an IP object it internally uses 'GracefulShutdown' option and works as expected.

In case of HP server when you set spec power= Off, you get following error,
"cannot power off machine: unable to power off the system: reset type 'GracefulShutdown' is not supported by this service"

When we checked supported options in case of HP, 'PushPowerButton' reset type looks closer match.

Currently Power = OffImmediate works fine and there is no blocker for HP server.

Please find attached investigation snapshots.

Why is this needed:

Redfish support for HP servers

What would you like to be added:
We'd like to be able to onboard HP servers.

Why is this needed:
The onboarding should just work out of the box with redfish, with what we currently have, but unfortunately HP's redfish implementation is not really following the standard so in our case the user creation doesn't work.
We are using HP 380G8 (Gen. 9) as baseline.

Remove power capability on certain oob objects

We'd like to have the option to remove the power capability from certain oob objects so that the bmc/machine/whatever will just be onboarded and not have the ability to power cycle or reset.

[rl-assigned_teams-5] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-assigned_teams-5
Explanation: Does teams have enough members on GitHub? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Investigate long duration of CI runs

Do they run multiple times?

oob-operator new feature ideas

What would you like to be added:
Hardware and devops team has suggested few ideas those can be implemented in the oob-operator, This task is created to document and discuss those ideas, which can later be considered for the implementation.

Why is this needed:

[rl-assigned_teams-2] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-assigned_teams-2
Explanation: Does it have an admin team on GitHub? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Implement a mechanism to exclude devices by MAC ID

What would you like to be added:

• currently we define in the oob-mac-prefixes.yaml file, the complete MAC address all devices except those ones which should't be managed by operator itself: like management server (aka zero server) + partition cluster
• the idea would be to have the possibility to define exclude e.g for 3 machines and other ones could be defined via MAC prefix without need to specify MAC and separate configuration for each machine..

Why is this needed:

Documentation missing

What would you like to be added:
Documenation

Why is this needed:
Docu is needed for proper implementation and understanding how such operator works. Thnx

OOB objects in error state

we have many objects in error state? probably too old can we reinit everything? error persists?

Avoid multiple reconciling of oob object in case of empty credentials

What would you like to be added:

Why is this needed:

status missing

NAMESPACE   NAME                                   UUID                                   POWER   LED   READY
oob         01c17340-4993-11ea-a1c8-0a94efa93719   01c17340-4993-11ea-a1c8-0a94efa93719   On      Off
oob         0962fed4-4993-11ea-8277-0a94efa9514d   0962fed4-4993-11ea-8277-0a94efa9514d   On      Off
oob         13f3378c-4992-11ea-a557-0a94efa93409   13f3378c-4992-11ea-a557-0a94efa93409   On      Off
oob         185e8394-4995-11ea-8354-0a94efa94e75   185e8394-4995-11ea-8354-0a94efa94e75   On      Off
oob         1e30d752-4992-11ea-be30-0a94efa937ed   1e30d752-4992-11ea-be30-0a94efa937ed   On      Off

shows a ready field but status is not yet part of crd

maybe the column should be named state and can have fields between ready, pending and failed (more?)

Gather MC fw version

What would you like to be added:
Gather the MC firmware version.

Why is this needed:
We would like to have the baseboard management controller firmware version gathered from the systems and stored in the oob objects to be used for upgrades, etc.

[rl-assigned_teams-4] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-assigned_teams-4
Explanation: Does it have a team with push privileges on GitHub? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

Move to ironcore

What would you like to be added:

Why is this needed:

Introduce go linter as a PR gatekeeper

Prior to merging a PR, a go linter with some well-defined configuration must have successfully run.

access to dell redfish seems to be broken

Dell machines are whitelisted in the oob mac database, but did not get recognized.

According to @5kt there is a bug in the redfish connectivity.

mgmt1 oob is at the latest idrac of Dell 6.xx. Please test against this.

ip is not shown on list action

NAMESPACE   NAME                                   UUID                                   POWER   LED   READY
oob         01c17340-4993-11ea-a1c8-0a94efa93719   01c17340-4993-11ea-a1c8-0a94efa93719   On      Off
oob         0962fed4-4993-11ea-8277-0a94efa9514d   0962fed4-4993-11ea-8277-0a94efa9514d   On      Off
oob         13f3378c-4992-11ea-a557-0a94efa93409   13f3378c-4992-11ea-a557-0a94efa93409   On      Off
oob         185e8394-4995-11ea-8354-0a94efa94e75   185e8394-4995-11ea-8354-0a94efa94e75   On      Off
oob         1e30d752-4992-11ea-be30-0a94efa937ed   1e30d752-4992-11ea-be30-0a94efa937ed   On      Off

there was an IP column but not populated
can this be fixed.

[rl-assigned_teams-1] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-assigned_teams-1
Explanation: Does it have enough teams on GitHub? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

[rl-reuse_tool-1] Violation against OSS Rules of Play

A violation against the OSS Rules of Play has been detected.

Rule ID: rl-reuse_tool-1
Explanation: Does README mention REUSE? No

Find more information at: https://sap.github.io/fosstars-rating-core/oss_rules_of_play_rating.html

please act on a label selector

currently oob-operator runs on a hard coded namespace oob

this should be not hardcoded ... it should work on the namespace it is installed to. but oob is triggered by ip addresses in that namespace and there should be a selector to this ... working with the label selector logic of kubernetes
e.g.

IPSelector:
    matchLabels:
      key: oob

when we have something like this we need a configmap for the oob operator, maybe it is also good to explicitly name the mac-db configmap and e.g. the prefix for the secrets and stuff

Add go test run as a PR gatekeeper

Add test run github action for go

only process ip objects with the oob label

please make sure we use a labelselector for processing the ip objects (and turning them into oob objects) atm we select on the label oob (should be configurable)

update of the expiry date in secret does not trigger pw renewal

it seems if the password in the password secret is updated (set to today) the password is not renewed. Seems to be a reconciler problem becaue the password expiry must work that way, or is it broken ?

Use case:
Operator with access to oob data needs to tell the password to e.g. customer support. So the operator sets the password to expire in 24hrs.
We need to make sure the password is guaranteed reset.

Add tests for all components

Than one shall go without description ;)

protocol ssh has a console capability

capabilities list the possible management functions of an oob machine.

a machine with protocol: ssh should have also console capabilities, since we support ssh consoles.

Move `oob-console` to `ironcore-dev`