Palmetto slower after firmware update about op-build

[skachm]

Update: The frequency was reduced by 1/3rd. Here's the old /proc/cpuinfo:

processor : 0
cpu : POWER8 (raw), altivec supported
clock : 3000.000000MHz
revision : 2.0 (pvr 004d 0200)

...

processor : 31
cpu : POWER8 (raw), altivec supported
clock : 3000.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : PowerNV
model : palmetto
machine : PowerNV palmetto
firmware : OPAL v3

And here's the new /proc/cpuinfo:

processor : 0
cpu : POWER8 (raw), altivec supported
clock : 2061.000000MHz
revision : 2.0 (pvr 004d 0200)

...

processor : 31
cpu : POWER8 (raw), altivec supported
clock : 2061.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : PowerNV
model : TN71-BP012
machine : PowerNV TN71-BP012
firmware : OPAL v3

[ozbenh]

On Wed, 2015-04-08 at 13:59 -0700, skachm wrote:

Update: The frequency was reduced by 1/3rd. Here's the
old /proc/cpuinfo:

Two things:

• Check this isn't just Linux cpufreq, ie, change you linux governor to
  "performance" on all CPUs
• Check the resulting actual frequency and whether it matches
  what /proc/cpuinfo is displaying (ppc64_cpu --frequency)

Cheers,
Ben.

processor : 0
cpu : POWER8 (raw), altivec supported
clock : 3000.000000MHz
revision : 2.0 (pvr 004d 0200)

...

processor : 31
cpu : POWER8 (raw), altivec supported
clock : 3000.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : PowerNV
model : palmetto
machine : PowerNV palmetto
firmware : OPAL v3

And here's the new /proc/cpuinfo:

processor : 0
cpu : POWER8 (raw), altivec supported
clock : 2061.000000MHz
revision : 2.0 (pvr 004d 0200)

...

processor : 31
cpu : POWER8 (raw), altivec supported
clock : 2061.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : PowerNV
model : TN71-BP012

machine : PowerNV TN71-BP012

firmware : OPAL v3

—
Reply to this email directly or view it on GitHub.

[skachm]

The frequency measures at 2.067 GHz for all of the cores:

$ sudo ppc64_cpu --frequency
min: 2.067 GHz (cpu 31)
max: 2.067 GHz (cpu 1)
avg: 2.067 GHz

I tried several options in linux power governor: performance, userspace set to max frequency, etc. without any change in the frequency.

$ sudo cpufreq-info -c 0
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
driver: powernv-cpufreq
CPUs which run at the same hardware frequency: 0 1 2 3 4 5 6 7
CPUs which need to have their frequency coordinated by software: 0 1 2 3 4 5 6 7
maximum transition latency: 4294.55 ms.
hardware limits: 2.06 GHz - 4.32 GHz
available frequency steps: 4.32 GHz, 4.29 GHz, 4.26 GHz, 4.22 GHz, 4.19 GHz, 4.16 GHz, 4.12 GHz, 4.09 GHz, 4.06 GHz, 4.02 GHz, 3.99 GHz, 3.96 GHz, 3.92 GHz, 3.89 GHz, 3.86 GHz, 3.82 GHz, 3.79 GHz, 3.76 GHz, 3.72 GHz, 3.69 GHz, 3.66 GHz, 3.62 GHz, 3.59 GHz, 3.56 GHz, 3.52 GHz, 3.49 GHz, 3.46 GHz, 3.42 GHz, 3.39 GHz, 3.36 GHz, 3.33 GHz, 3.29 GHz, 3.26 GHz, 3.23 GHz, 3.19 GHz, 3.16 GHz, 3.13 GHz, 3.09 GHz, 3.06 GHz, 3.03 GHz, 2.99 GHz, 2.96 GHz, 2.93 GHz, 2.89 GHz, 2.86 GHz, 2.83 GHz, 2.79 GHz, 2.76 GHz, 2.73 GHz, 2.69 GHz, 2.66 GHz, 2.63 GHz, 2.59 GHz, 2.56 GHz, 2.53 GHz, 2.49 GHz, 2.46 GHz, 2.43 GHz, 2.39 GHz, 2.36 GHz, 2.33 GHz, 2.29 GHz, 2.26 GHz, 2.23 GHz, 2.19 GHz, 2.16 GHz, 2.13 GHz, 2.09 GHz, 2.06 GHz
available cpufreq governors: conservative, ondemand, userspace, powersave, performance
current policy: frequency should be within 2.06 GHz and 4.32 GHz.
The governor "userspace" may decide which speed to use
within this range.
current CPU frequency is 2.06 GHz (asserted by call to hardware).
cpufreq stats: 4.32 GHz:39.38%, 4.29 GHz:0.00%, 4.26 GHz:0.00%, 4.22 GHz:0.00%, 4.19 GHz:0.00%, 4.16 GHz:0.00%, 4.12 GHz:0.02%, 4.09 GHz:0.01%, 4.06 GHz:0.00%, 4.02 GHz:0.01%, 3.99 GHz:0.01%, 3.96 GHz:0.01%, 3.92 GHz:0.00%, 3.89 GHz:0.01%, 3.86 GHz:0.01%, 3.82 GHz:0.01%, 3.79 GHz:0.01%, 3.76 GHz:0.00%, 3.72 GHz:0.01%, 3.69 GHz:0.02%, 3.66 GHz:0.05%, 3.62 GHz:0.00%, 3.59 GHz:0.04%, 3.56 GHz:0.01%, 3.52 GHz:0.00%, 3.49 GHz:0.00%, 3.46 GHz:0.00%, 3.42 GHz:0.00%, 3.39 GHz:0.00%, 3.36 GHz:0.01%, 3.33 GHz:0.00%, 3.29 GHz:0.01%, 3.26 GHz:0.01%, 3.23 GHz:0.01%, 3.19 GHz:0.00%, 3.16 GHz:0.01%, 3.13 GHz:0.01%, 3.09 GHz:0.01%, 3.06 GHz:0.01%, 3.03 GHz:0.00%, 2.99 GHz:0.01%, 2.96 GHz:0.00%, 2.93 GHz:0.00%, 2.89 GHz:0.00%, 2.86 GHz:0.01%, 2.83 GHz:0.00%, 2.79 GHz:0.00%, 2.76 GHz:0.00%, 2.73 GHz:0.00%, 2.69 GHz:0.00%, 2.66 GHz:0.00%, 2.63 GHz:0.00%, 2.59 GHz:0.00%, 2.56 GHz:0.00%, 2.53 GHz:0.01%, 2.49 GHz:0.00%, 2.46 GHz:0.00%, 2.43 GHz:0.00%, 2.39 GHz:0.01%, 2.36 GHz:0.00%, 2.33 GHz:0.00%, 2.29 GHz:0.00%, 2.26 GHz:0.00%, 2.23 GHz:0.00%, 2.19 GHz:0.00%, 2.16 GHz:0.00%, 2.13 GHz:0.00%, 2.09 GHz:0.00%, 2.06 GHz:60.21% (121630)

[skachm]

Also the output for performance:

$ sudo cpufreq-info -c 0
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
driver: powernv-cpufreq
CPUs which run at the same hardware frequency: 0 1 2 3 4 5 6 7
CPUs which need to have their frequency coordinated by software: 0 1 2 3 4 5 6 7
maximum transition latency: 4294.55 ms.
hardware limits: 2.06 GHz - 4.32 GHz
available frequency steps: 4.32 GHz, 4.29 GHz, 4.26 GHz, 4.22 GHz, 4.19 GHz, 4.16 GHz, 4.12 GHz, 4.09 GHz, 4.06 GHz, 4.02 GHz, 3.99 GHz, 3.96 GHz, 3.92 GHz, 3.89 GHz, 3.86 GHz, 3.82 GHz, 3.79 GHz, 3.76 GHz, 3.72 GHz, 3.69 GHz, 3.66 GHz, 3.62 GHz, 3.59 GHz, 3.56 GHz, 3.52 GHz, 3.49 GHz, 3.46 GHz, 3.42 GHz, 3.39 GHz, 3.36 GHz, 3.33 GHz, 3.29 GHz, 3.26 GHz, 3.23 GHz, 3.19 GHz, 3.16 GHz, 3.13 GHz, 3.09 GHz, 3.06 GHz, 3.03 GHz, 2.99 GHz, 2.96 GHz, 2.93 GHz, 2.89 GHz, 2.86 GHz, 2.83 GHz, 2.79 GHz, 2.76 GHz, 2.73 GHz, 2.69 GHz, 2.66 GHz, 2.63 GHz, 2.59 GHz, 2.56 GHz, 2.53 GHz, 2.49 GHz, 2.46 GHz, 2.43 GHz, 2.39 GHz, 2.36 GHz, 2.33 GHz, 2.29 GHz, 2.26 GHz, 2.23 GHz, 2.19 GHz, 2.16 GHz, 2.13 GHz, 2.09 GHz, 2.06 GHz
available cpufreq governors: conservative, ondemand, userspace, powersave, performance
current policy: frequency should be within 2.06 GHz and 4.32 GHz.
The governor "performance" may decide which speed to use
within this range.
current CPU frequency is 2.06 GHz (asserted by call to hardware).
cpufreq stats: 4.32 GHz:39.53%, 4.29 GHz:0.00%, 4.26 GHz:0.00%, 4.22 GHz:0.00%, 4.19 GHz:0.00%, 4.16 GHz:0.00%, 4.12 GHz:0.02%, 4.09 GHz:0.01%, 4.06 GHz:0.00%, 4.02 GHz:0.01%, 3.99 GHz:0.01%, 3.96 GHz:0.01%, 3.92 GHz:0.00%, 3.89 GHz:0.01%, 3.86 GHz:0.01%, 3.82 GHz:0.01%, 3.79 GHz:0.01%, 3.76 GHz:0.00%, 3.72 GHz:0.01%, 3.69 GHz:0.02%, 3.66 GHz:0.05%, 3.62 GHz:0.00%, 3.59 GHz:0.04%, 3.56 GHz:0.01%, 3.52 GHz:0.00%, 3.49 GHz:0.00%, 3.46 GHz:0.00%, 3.42 GHz:0.00%, 3.39 GHz:0.00%, 3.36 GHz:0.01%, 3.33 GHz:0.00%, 3.29 GHz:0.01%, 3.26 GHz:0.01%, 3.23 GHz:0.01%, 3.19 GHz:0.00%, 3.16 GHz:0.01%, 3.13 GHz:0.01%, 3.09 GHz:0.01%, 3.06 GHz:0.01%, 3.03 GHz:0.00%, 2.99 GHz:0.01%, 2.96 GHz:0.00%, 2.93 GHz:0.00%, 2.89 GHz:0.00%, 2.86 GHz:0.01%, 2.83 GHz:0.00%, 2.79 GHz:0.00%, 2.76 GHz:0.00%, 2.73 GHz:0.00%, 2.69 GHz:0.00%, 2.66 GHz:0.00%, 2.63 GHz:0.00%, 2.59 GHz:0.00%, 2.56 GHz:0.00%, 2.53 GHz:0.01%, 2.49 GHz:0.00%, 2.46 GHz:0.00%, 2.43 GHz:0.00%, 2.39 GHz:0.01%, 2.36 GHz:0.00%, 2.33 GHz:0.00%, 2.29 GHz:0.00%, 2.26 GHz:0.00%, 2.23 GHz:0.00%, 2.19 GHz:0.00%, 2.16 GHz:0.00%, 2.13 GHz:0.00%, 2.09 GHz:0.00%, 2.06 GHz:60.06% (121630)

(Ubuntu 14.10, Palmetto without any hardware additions or changes, op-build firmware v1.0 built natively on a Palmetto.)

[ozbenh]

On Thu, 2015-04-09 at 12:34 -0700, skachm wrote:

The frequency measures at 2.067 GHz for all of the cores:

What governor is active and how did you configure it ?

If it's running the "on demand" governor, it will go to the lowest
frequency when the machine isn't under load.

You can try echo'ing "performance" in all the scaling_governor files
of /sys/devices/system/cpu/cpu* and see what the result is.

Then make sure, using ppc64_cpu --frequency, that the frequency measured
corresponds to what's in /proc/cpuinfo

clarity@clarity22:/sys/devices/system/cpu/cpu0/cpufreq$ sudo
cpufreq-info -c 0
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email protected], please.
analyzing CPU 0:
driver: powernv-cpufreq
CPUs which run at the same hardware frequency: 0 1 2 3 4 5 6 7
CPUs which need to have their frequency coordinated by software: 0 1 2
3 4 5 6 7
maximum transition latency: 4294.55 ms.
hardware limits: 2.06 GHz - 4.32 GHz
available frequency steps: 4.32 GHz, 4.29 GHz, 4.26 GHz, 4.22 GHz,
4.19 GHz, 4.16 GHz, 4.12 GHz, 4.09 GHz, 4.06 GHz, 4.02 GHz, 3.99 GHz,
3.96 GHz, 3.92 GHz, 3.89 GHz, 3.86 GHz, 3.82 GHz, 3.79 GHz, 3.76 GHz,
3.72 GHz, 3.69 GHz, 3.66 GHz, 3.62 GHz, 3.59 GHz, 3.56 GHz, 3.52 GHz,
3.49 GHz, 3.46 GHz, 3.42 GHz, 3.39 GHz, 3.36 GHz, 3.33 GHz, 3.29 GHz,
3.26 GHz, 3.23 GHz, 3.19 GHz, 3.16 GHz, 3.13 GHz, 3.09 GHz, 3.06 GHz,
3.03 GHz, 2.99 GHz, 2.96 GHz, 2.93 GHz, 2.89 GHz, 2.86 GHz, 2.83 GHz,
2.79 GHz, 2.76 GHz, 2.73 GHz, 2.69 GHz, 2.66 GHz, 2.63 GHz, 2.59 GHz,
2.56 GHz, 2.53 GHz, 2.49 GHz, 2.46 GHz, 2.43 GHz, 2.39 GHz, 2.36 GHz,
2.33 GHz, 2.29 GHz, 2.26 GHz, 2.23 GHz, 2.19 GHz, 2.16 GHz, 2.13 GHz,
2.09 GHz, 2.06 GHz
available cpufreq governors: conservative, ondemand, userspace,
powersave, performance
current policy: frequency should be within 2.06 GHz and 4.32 GHz.
The governor "userspace" may decide which speed to use
within this range.
current CPU frequency is 2.06 GHz (asserted by call to hardware).
cpufreq stats: 4.32 GHz:39.38%, 4.29 GHz:0.00%, 4.26 GHz:0.00%, 4.22
GHz:0.00%, 4.19 GHz:0.00%, 4.16 GHz:0.00%, 4.12 GHz:0.02%, 4.09
GHz:0.01%, 4.06 GHz:0.00%, 4.02 GHz:0.01%, 3.99 GHz:0.01%, 3.96
GHz:0.01%, 3.92 GHz:0.00%, 3.89 GHz:0.01%, 3.86 GHz:0.01%, 3.82
GHz:0.01%, 3.79 GHz:0.01%, 3.76 GHz:0.00%, 3.72 GHz:0.01%, 3.69
GHz:0.02%, 3.66 GHz:0.05%, 3.62 GHz:0.00%, 3.59 GHz:0.04%, 3.56
GHz:0.01%, 3.52 GHz:0.00%, 3.49 GHz:0.00%, 3.46 GHz:0.00%, 3.42
GHz:0.00%, 3.39 GHz:0.00%, 3.36 GHz:0.01%, 3.33 GHz:0.00%, 3.29
GHz:0.01%, 3.26 GHz:0.01%, 3.23 GHz:0.01%, 3.19 GHz:0.00%, 3.16
GHz:0.01%, 3.13 GHz:0.01%, 3.09 GHz:0.01%, 3.06 GHz:0.01%, 3.03
GHz:0.00%, 2.99 GHz:0.01%, 2.96 GHz:0.00%, 2.93 GHz:0.00%, 2.89
GHz:0.00%, 2.86 GHz:0.01%, 2.83 GHz:0.00%, 2.79 GHz:0.00%, 2.76
GHz:0.00%, 2.73 GHz:0.00%, 2.69 GHz:0.00%, 2.66 GHz:0.00%, 2.63
GHz:0.00%, 2.59 GHz:0.00%, 2.56 GHz:0.00%, 2.53 GHz:0.01%, 2.49
GHz:0.00%, 2.46 GHz:0.00%, 2.43 GHz:0.00%, 2.39 GHz:0.01%, 2.36
GHz:0.00%, 2.33 GHz:0.00%, 2.29 GHz:0.00%, 2.26 GHz:0.00%, 2.23
GHz:0.00%, 2.19 GHz:0.00%, 2.16 GHz:0.00%, 2.13 GHz:0.00%, 2.09
GHz:0.00%, 2.06 GHz:60.21% (121630)

—
Reply to this email directly or view it on GitHub.

[skachm]

I tried performance and userspace to 4.32 GHz, ppc64_cpu reported 2.067 GHz under both governors.

[ozbenh]

On Thu, 2015-04-09 at 16:18 -0700, skachm wrote:

I tried performance and userspace to 4.32 GHz, ppc64_cpu reported
2.067 GHz under both governors.

Ok. I observed that a while ago too but for me at least it seems fixed
with the latest PNOR we built from github and the latest AMI code, are
you fully up to date on your side ?

Cheers,
Ben.

[williamspatrick]

On the original Palmetto code we did not have the OCC supported. The OCC ("On-Chip Controller") is a device inside the P8 module that varies the voltages and frequencies based on thermal readings, power cap settings, and Linux governor requests. The original code simply set the frequency to 3ghz, while the latest code has the OCC which can run at a large frequency range. (It looks like 4.32 to 2.067 is your processor module's range)

I believe the /proc/cpuinfo represents the nominal frequency of the module that Hostboot told skiboot/Linux via the devtree. There are three frequency points that each module is qualified at: safe, nominal, turbo. Nominal is the "label" frequency.

In this case the /proc/cpuinfo is showing 2ghz which is the safe frequency for all modules. Whenever the OCC encounters a problem it will force the processor to safe frequency to minimize potential of hardware damage. So, since I suspect your processor was running in safe from the boot time (based on this /proc/cpuinfo value), that would seem to indicate that the OCC failed to start during Hostboot. Do you see any indication of this in the early console output? You might see a message somewhere between "ISTEP 18.0" and "ISTEP 21.1" messages indicating we failed to start the OCC.

It is also possible that the OCC is failing, for some reason, during runtime.

@ozbenh mentioned seeing this before. I think when we have seen this before (OCC failing at boot) many of the cases were to outdated APSS code. The APSS is an FPGA related to the power supplies and voltage regulators (Analog Power something something). The code for this FPGA is also on github but I don't know how to update it: https://github.com/open-power/apss

@guillesilva / @sbroylesibm - Is this something you can help with? I recall there were some i2c commands you could run from the BMC to check the APSS status and code level to see if it is running properly and what the state of it was. If all of the APSS code loads that Tyan originally shipped are out of date, we'd probably recommend that they put an update and instructions on their website.

@Erich-Hauptli / @kyle-tyan - You might want to keep tabs on this issue. I suspect as we get more "adventurous" users, like @skachm, others are going to run into similar problems.

[ozbenh]

On Thu, 2015-04-09 at 20:27 -0700, Patrick Williams wrote:

On the original Palmetto code we did not have the OCC supported. The
OCC ("On-Chip Controller") is a device inside the P8 module that
varies the voltages and frequencies based on thermal readings, power
cap settings, and Linux governor requests. The original code simply
set the frequency to 3ghz, while the latest code has the OCC which can
run at a large frequency range. (It looks like 4.32 to 2.067 is your
processor module's range)

I believe the /proc/cpuinfo represents the nominal frequency of the
module that Hostboot told skiboot/Linux via the devtree. There are
three frequency points that each module is qualified at: safe,
nominal, turbo. Nominal is the "label" frequency.

No, on any not-too-ancient kernel, it should display a frequency based
on the current pstate (provided cpufreq is active in Linux).

In this case the /proc/cpuinfo is showing 2ghz which is the safe
frequency for all modules. Whenever the OCC encounters a problem it
will force the processor to safe frequency to minimize potential of
hardware damage. So, since I suspect your processor was running in
safe from the boot time (based on this /proc/cpuinfo value), that
would seem to indicate that the OCC failed to start during Hostboot.
Do you see any indication of this in the early console output? You
might see a message somewhere between "ISTEP 18.0" and "ISTEP 21.1"
messages indicating we failed to start the OCC.

It is also possible that the OCC is failing, for some reason, during
runtime.

What we have observed in the past (but doesn't seem to be happening on
my Palmetto with the latest op-build stuff) was that the OCC started up
fine, gave us a pstate table, we picked the fastest state (4.3Ghz I
believe), but ppc64_cpu --frequency would still measure 2Ghz.

Reading the PMSR would still return the pstate corresponding to 4.3Ghz
though and the "safe mode" override bit in SCOM (whatever name it
actually has) wasn't set, it was like the OCC didn't properly program
the pstate frequencies under the hood.

There was no log of OCC failures from HBRT.

@ozbenh mentioned seeing this before. I think when we have seen this
before (OCC failing at boot) many of the cases were to outdated APSS
code. The APSS is an FPGA related to the power supplies and voltage
regulators (Analog Power something something). The code for this FPGA
is also on github but I don't know how to update it:
https://github.com/open-power/apss

Is v5 still current ? One problem I've had is the BMC i2c bus getting
stuck, in turn causing it to fail to talk to the APSS completely.

@guillesilva / @sbroylesibm - Is this something you can help with? I
recall there were some i2c commands you could run from the BMC to
check the APSS status and code level to see if it is running properly
and what the state of it was. If all of the APSS code loads that Tyan
originally shipped are out of date, we'd probably recommend that they
put an update and instructions on their website.

@Erich-Hauptli / @kyle-tyan - You might want to keep tabs on this
issue. I suspect as we get more "adventurous" users, like @skachm,
others are going to run into similar problems.

First make sure you have BMC and PNOR completely up to date, at least
from my perspective, it looks like something got fixed in the last 2
month or so.

Cheers,
Ben.

[sbroylesibm]

FYI: The latest APSS level is 05, you can check it from the BMC with:

Get APSS level

/usr/local/bin/i2c-test -m 1 -b 2 -s 0x38 -rc 1 -d 0x01

Also, as a general sanity check on the APSS you can scan the IIC bus to the APSS to make sure the IIC slave address is responding with:

Scan IIC bus, APSS is on address 0x70

/usr/local/bin/i2c-test --scan -b 2

[skachm]

I first checked it out less than 2 months ago (around March 26th), but I checked out a fresh copy of the repo, built it, reflashed the firmware and the server seems to be working now.

$ sudo ppc64_cpu --frequency
min: 4.334 GHz (cpu 31)
max: 4.334 GHz (cpu 1)
avg: 4.334 GHz

[williamspatrick]

Glad it is working for you. When you build a level for something other than testing, it is better to checkout a tag after the clone rather than HEAD. This way you get some content that has already been qualified by us. From: skachmSent: Monday, April 13, 2015 8:11 AMTo: open-power/op-buildReply To: open-power/op-buildCc: Patrick WilliamsSubject: Re: [op-build] Palmetto slower after firmware update (#132)I first checked it out less than 2 months ago (around March 26th), but I checked out a fresh copy of the repo, built it, reflashed the firmware and the server seems to be working now.

$ sudo ppc64_cpu --frequency
min: 4.334 GHz (cpu 31)
max: 4.334 GHz (cpu 1)
avg: 4.334 GHz

—Reply to this email directly or view it on GitHub.