RegistrationError, Failed to get power state for node Error: IPMI call failed: power status. about metal3-dev-env

[metal3-io-bot]

This issue is currently awaiting triage.
If Metal3.io contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rakeshk121]

I have also tried with redfish as BMC_DRIVER , could not get a sucesfull installation with that as well

After running the make command , at the end it is stuck with the following message:

- Waiting for task completion (up to 2400 seconds)  - Command: 'check_bm_hosts node-0 redfish+http://192.168.111.1:8000/redfish/v1/Systems/6be5c1d8-935f-48c3-a3a0-83482d408675 admin password 00:3a:e4:33:47:3c'

When I describe the baremetalhost node-0, i see the below error

Normal  RegistrationError  7m25s  metal3-baremetal-controller  Failed to get power state for node e84715bb-2fae-43d4-8241-94916138fba2. Error: HTTP GET http://192.168.111.1:8000/redfish/v1/Systems/6be5c1d8-935f-48c3-a3a0-83482d408675 returned code 500. Base.1.0.GeneralError: Error when connecting to the libvirt URI "qemu+ssh://[email protected]/system?&keyfile=/root/ssh/id_rsa_virt_power&no_verify=1&no_tty=1": Cannot recv data: [email protected]: Permission denied (publickey).: Connection reset by peer Extended information: [{'@odata.type': '/redfish/v1/$metadata#Message.1.0.0.Message', 'MessageId': 'Base.1.0.GeneralError'}]

Error Message:  Failed to get power state for node e84715bb-2fae-43d4-8241-94916138fba2. Error: HTTP GET http://192.168.111.1:8000/redfish/v1/Systems/6be5c1d8-935f-48c3-a3a0-83482d408675 returned code 500. Base.1.0.GeneralError: Error when connecting to the libvirt URI "qemu+ssh://[email protected]/system?&keyfile=/root/ssh/id_rsa_virt_power&no_verify=1&no_tty=1": Cannot recv data: [email protected]: Permission denied (publickey).: Connection reset by peer Extended information: [{'@odata.type': '/redfish/v1/$metadata#Message.1.0.0.Message', 'MessageId': 'Base.1.0.GeneralError'}]

[dtantsur]

@rakeshk121 the BMC emulator reports that it cannot access the virtual host (the machine where the VMs are running). There may be many reasons for that. If you look for a vbmc (IPMI) or sushy-tools (Redfish) containers and log into each of them, you can try SSH'ing into the virtual host with the provided IP and key file and checking why it fails.

/triage needs-information

[rakeshk121]

@dtantsur thanks for the suggestion

These are the logs of vbmc container

core@nodea14:~$ docker logs -f a19488a16e26
libvirt: XML-RPC error : Cannot recv data: Warning: Permanently added '192.168.111.1' (ED25519) to the list of known hosts.
[email protected]: Permission denied (publickey).: Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/virtualbmc/utils.py", line 47, in __enter__
    self.conn = libvirt.openReadOnly(self.uri)
  File "/usr/lib64/python3.9/site-packages/libvirt.py", line 349, in openReadOnly
    raise libvirtError('virConnectOpenReadOnly() failed')
libvirt.libvirtError: Cannot recv data: Warning: Permanently added '192.168.111.1' (ED25519) to the list of known hosts.
[email protected]: Permission denied (publickey).: Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pyghmi/ipmi/bmc.py", line 175, in handle_raw_request
    return self.get_chassis_status(session)
  File "/usr/lib/python3.9/site-packages/pyghmi/ipmi/bmc.py", line 91, in get_chassis_status
    powerstate = self.get_power_state()
  File "/usr/lib/python3.9/site-packages/virtualbmc/vbmc.py", line 129, in get_power_state
    with utils.libvirt_open(readonly=True, **self._conn_args) as conn:
  File "/usr/lib/python3.9/site-packages/virtualbmc/utils.py", line 54, in __enter__
    raise exception.LibvirtConnectionOpenError(uri=self.uri, error=e)
virtualbmc.exception.LibvirtConnectionOpenError: Fail to establish a connection with libvirt URI "qemu+ssh://[email protected]/system?&keyfile=/root/ssh/id_rsa_virt_power&no_verify=1&no_tty=1". Error: Cannot recv data: Warning: Permanently added '192.168.111.1' (ED25519) to the list of known hosts.
[email protected]: Permission denied (publickey).: Connection reset by peer
libvirt: XML-RPC error : Cannot recv data: [email protected]: Permission denied (publickey).: Connection reset by peer

I have exec into the vbmc container and tried to ssh into the virtual host , Got permission denied for the ssh. Below are the vebose output for the same:

[root@nodea14 /]# ssh -v -i /root/ssh/id_rsa_virt_power  [email protected]
OpenSSH_8.7p1, OpenSSL 3.0.7 1 Nov 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to 192.168.111.1 [192.168.111.1] port 22.
debug1: Connection established.
debug1: identity file /root/ssh/id_rsa_virt_power type 0
debug1: identity file /root/ssh/id_rsa_virt_power-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.9
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.9 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.111.1:22 as 'root'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:yOoKWNnaN1jPxj+BRFRNyOr+aAHj++yLLxZjP39A3D4
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '192.168.111.1' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /root/ssh/id_rsa_virt_power RSA SHA256:Ot5mJxupY0a//xkfk457i7ggM2kDR+aGCiDbN43Xrm0 explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected]>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /root/ssh/id_rsa_virt_power RSA SHA256:Ot5mJxupY0a//xkfk457i7ggM2kDR+aGCiDbN43Xrm0 explicit
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

[Rozzii]

/question
/help

[metal3-io-bot]

@Rozzii:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/question
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[adilGhaffarDev]

cc @adilGhaffarDev

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[Rozzii]

If you are planning to work on this @adilGhaffarDev please remove the stale label

[metal3-io-bot]

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

[metal3-io-bot]

@metal3-io-bot: Closing this issue.

In response to this:

Stale issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle stale.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.