UNKNOWN - "http response error: 401 - invalid content type about check_by_powershell

[pdorschner]

At the moment there is no option for AD-Users. This should be implemented.

We evaluate this problem. I discussed that with @LordHepipud

[lazyfrosch]

Usually NTLM should work with domains, but be aware of the shall handling for \.

You need to use one of these:

• --user 'domain\user'
• --user "domain\\user"

Icinga will take care of escaping a single \ correctly from Director, Icinga 2 config will require vars.xxx = "domain\\user"

I still need to test --user "fqdn.corp.de\\user"

Also see #9

[pdorschner]

We have a new pre-release version. I would appreciate if you could test the new build v0.2.0 pre-release.

The error message: UNKNOWN - "http response error: 401 - invalid content type" could be a problem with the configuration of the windows system, e.g. Firewall, WinRM. On my tests I can execute a command like that with no errors:

./check_by_powershell-darwin-amd64 --host 'example.ip' --auth NTLM --user "DOMAIN\Username" --password 'examplePass' --icingacmd "Invoke-IcingaCheckCPU"

[OK] Check package "CPU Load" | 'core_2'=47.41397%;;;0;100 'core_total'=49.99663%;;;0;100 'core_0'=70.58625%;;;0;100 'core_1'=29.40701%;;;0;100 'core_3'=45.4517%;;;0;100

Unfortunately NTLM won't work like @lazyfrosch mentioned above

I still need to test --user "fqdn.corp.de\user"

If you have feedback, please write us in #11

[lazyfrosch]

Okay, let's discuss the problems here, #11 is for a TLS client certificate, and not TLS connections by default.

[lazyfrosch]

@K0nne wrote in #11

--snip--

Hi!

I tested the latest release against our own test server:

./check_by_powershell-linux-amd64 --version
check_by_powershell version 0.2.0 - 72488a89b2e9fcf36e244b3394687e23838af55f

Test1:

./check_by_powershell-linux-amd64 --host 'IP' --auth NTLM --user 'DOMAIN\\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'FQDN' --auth NTLM --user 'DOMAIN\\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'

Here we get the following error:

UNKNOWN - execution failed: execution of remote cmd failed: http response error: 401 - invalid content type

Test2:

./check_by_powershell-linux-amd64 --host 'IP' --auth NTLM --user 'DOMAIN\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'IP' --auth NTLM --user 'USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'FQDN' --auth NTLM --user 'DOMAIN\USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'
./check_by_powershell-linux-amd64 --host 'FQDN' --auth NTLM --user 'USR' --password 'PWD'  --icingacmd 'Invoke-IcingaCheckCPU'

With the other spelling of the user we get this message:
UNKNOWN - execution failed: execution of remote cmd failed: Command has already been closed

We will investigate this with our colleagues.

[lazyfrosch]

@K0nne: You are not using a HTTPS/TLS connection, therefor you would need to add --tls as option, or reconfigure winrm:

winrm set winrm/config/service '@{AllowUnencrypted="true"}'

I guess it might be a good idea to change the --tls switch, so TLS is used by default, and HTTP can be used when needed...

[K0nne]

@aheinhold can you please take over?
@lazyfrosch I'm currently on vacation.

[lazyfrosch]

@K0nne enjoy! 🍹

[lazyfrosch]

@aheinhold I've updated the README, please tell me if I can help 😉

[aheinhold]

Hi @lazyfrosch
i downloaded and compiled the check_by_winrm package.

After this i tested the check as follows:

./check_by_winrm -H FQDN --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU'
UNKNOWN - Timeout reached
./check_by_winrm -H FQDN --user 'USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU'
UNKNOWN - Timeout reached
./check_by_winrm -H FQDN --user 'USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://FQDN:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H FQDN --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://FQDN:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H IP --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://IP:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H IP --user 'USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://IP:5986/wsman": dial tcp IP:5986: i/o timeout
./check_by_winrm -H IP --user 'USR@DOMAIN' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU' --timeout 60
UNKNOWN - execution failed: execution of remote cmd failed: unknown error Post "https://IP:5986/wsman": dial tcp IP:5986: i/o timeout

The i/o timeout occurs after ~30 seconds
There is no additional output when using the argument --verbose

[lazyfrosch]

• Have you configured WinRM on the host? When yes, how?
• Is the firewall opened or disabled?
• Can you access the HTTPs URL manually from your desktop?

[aheinhold]

Hi @lazyfrosch

sorry for my late response.
NOW i have configured the host for tls communication... I couldn't find out how the host was configured before.

I configured the host as follows:

• requested a certificate for the host.
• imported the certificate in the local machine cert store
• Get-ChildItem -Path cert:\LocalMachine\My -Recurse; -> copied the thumbprint
• $CertThumbprint = 'my-certificates-thumbprint';
• Enable-PSRemoting -SkipNetworkProfileCheck -Force;
• New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $CertThumbprint -Force;
• Set-Item WSMan:\localhost\Service\EnableCompatibilityHttpsListener -Value true;
• tested the connection from the satellite via telnet over port 5986
• tested the check_by_winrm-command

I got the following output:

./check_by_winrm -H FQDN --user 'DOMAIN\USR' --password 'PWD' --icingacmd 'Invoke-IcingaCheckCPU
UNKNOWN - execution failed: execution of remote cmd failed: http response error: 401 - invalid content type

[lazyfrosch]

How about enabling Basic Auth?

Or using --auth ntlm

[aheinhold]

--auth ntlm

UNKNOWN - execution failed: execution of remote cmd failed: Command has already been closed

when I try to configure basic auth:

winrm set winrm/config/service/Auth '@{Basic="true"}'
WSManFault
    Message
        ProviderFault
            WSManFault
                Message = The config setting Basic cannot be changed because is controlled by policies. The policy would need to be set to "Not Configured" in order to change the config setting.

Error number:  -2144108406 0x8033808A
Cannot change GPO controlled setting.

Changing this settings is prohibited by GPO. Would it help for the tests if i let this be changed?

[lazyfrosch]

Well Basic Auth might not be the best choice for default, I opened #15 for that.

I've seen the error "Command has already been closed" once in the CI, but not sure where that comes from, I thought of a timing issue during setup of the CI VM.

Have you restarted winrm Restart-Service winrm or tried again later?

[aheinhold]

I've seen the error "Command has already been closed" once in the CI, but not sure where that comes from, I thought of a timing issue during setup of the CI VM.

I don't know. Maybe @K0nne can say something about it next week.

Have you restarted winrm Restart-Service winrm or tried again later?

Yes, i restarted WinRM after the configuration.

Result/Plugin output:

UNKNOWN - execution failed: execution of remote cmd failed: http response error: 401 - invalid content type

[lazyfrosch]

Invalid content type basically means that authentication didn't work. You should find some info in the security log of the server.

But when you use NTLM with TLS - which is default in master now - it should be fine.

I'm not sure what can cause the error UNKNOWN - execution failed: execution of remote cmd failed: Command has already been closed. Maybe we should check the logs on server side here as well.

It could also be related to additional security policies.

/cc @mwaldmueller

[lazyfrosch]

Current status: We can't really determine why NTLM authentication fails.

There is a limitation with the library, that you can only auth with users of the same domain as the computer, and not even local users. Azure/go-ntlmssp#16

For this issue that shouldn't be the cause.

Events recorded:

Audit Success	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4627	Group Membership	"Group membership information.

<non interesting details>

Audit Success	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4624	Logon	"An account was successfully logged on.

<non interesting details>

Audit Failure	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4776	Credential Validation	"The computer attempted to validate the credentials for an account.

Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account:	user
Source Workstation:	REMOTECOMPUTER
Error Code:	0xC0000064"

Audit Success	27.10.2020 07:23:49	Microsoft-Windows-Security-Auditing	4634	Logoff	"An account was logged off.

<non interesting details>

What stands out here, is that the logon is successful, and afterwards some verification fails.

[K0nne]

On Monday I will provide you some audit details from our pwsh-version and winrm.

[K0nne]

I executed both checks from the same satellite against the same target. To my surprise check_by_winrm produced no security eventlog entries. Instead it thows the http 401 message in the same second it is executed. Is it posssible that the check dies right after execution before it can send anything?

I've sent the logs from our incarnation to @LordHepipud.