RPI Zero 2W incorrect memory size about linux

[gugolple]

Here are the logs of executing the same commands on my Raspberry Pi Zero W. These are coherent with a device with ~500MB of ram with ~64MB separated for the GPU split.

$ dmesg | egrep 'Memory|CMA|cma' --color -C3
[    0.000000] Memory: 374380K/458752K available (8694K kernel code, 1324K rwdata, 2828K rodata, 424K init, 836K bss, 18836K reserved, 65536K cma-reserved)

$ cat /proc/meminfo
MemTotal:         440340 kB

$ vcgencmd get_mem gpu
gpu=64M

[popcornmix]

Can you identify the exact update which caused this. See:
https://github.com/raspberrypi/rpi-firmware/commits/master

If you click on each commit the end of the url contains a git hash. Run
sudo rpi-update <hash>
to revert back to that version. Report the first version with this error.

[pelwell]

Some before and after comparisons would be useful - that looks to be all after.

I note that you've opted into the 64-bit kernel (and perhaps even the 64-bit OS - it's hard to tell). This is not recommended on a Zero 2 W because it will eat into your limited RAM.

[gugolple]

Can you identify the exact update which caused this. See: https://github.com/raspberrypi/rpi-firmware/commits/master

The issue was introduced at the commit e632362b0399b4ce331aacd9386685bc60938ab7

Some before and after comparisons would be useful - that looks to be all after.

I note that you've opted into the 64-bit kernel (and perhaps even the 64-bit OS - it's hard to tell). This is not recommended on a Zero 2 W because it will eat into your limited RAM.

Free before the commit (using commit af3ab905d42829d4ce860f19a0e049c3bcdbe35f):

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           466Mi       165Mi       138Mi       1.2Mi       218Mi       300Mi

Free after appling the commit e632362b0399b4ce331aacd9386685bc60938ab7:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           400Mi       154Mi        88Mi       1.2Mi       209Mi       246Mi

[gugolple]

Some before and after comparisons would be useful - that looks to be all after.

I note that you've opted into the 64-bit kernel (and perhaps even the 64-bit OS - it's hard to tell). This is not recommended on a Zero 2 W because it will eat into your limited RAM.

Indeed I am running Raspberry Pi OS 64 bit version, but I do not see how it affects the topic at hand.

[popcornmix]

Ah - so was this the kernel update rather than the firmware?
That commit did move from 6.1 kernel to 6.6.

You should be able to confirm this by copying /boot/firmware/{start*.elf,fixup*.dat} from the "good" rpi-update commit and the "bad" commit somewhere safe, and check if just swapping those files over changes free memory available.

You can check firmware and kernel versions after a reboot with:

uname -a
vcgencmd version

[pelwell]

but I do not see how it affects the topic at hand.

Do you care about how much usable RAM your Z2W has?

[pelwell]

Note that as well as having unavoidably larger page tables, lower code density, etc., there is a common 64-bit kernel that runs on Pi 3s, 4s, 5 and Z2W. This means that your Z2W has used some of its RAM to hold support for booting a Pi 5 to the point where it can access its SD card, USB, and Ethernet interfaces. kernel7.img is only for Pi 3s and Z2W, therefore it is much smaller. Note also that the 64-bit kernels are GZIP compressed - uncompressed they are roughly 25MB.

[gugolple]

Ah - so was this the kernel update rather than the firmware? That commit did move from 6.1 kernel to 6.6.

You should be able to confirm this by copying /boot/firmware/{start*.elf,fixup*.dat} from the "good" rpi-update commit and the "bad" commit somewhere safe, and check if just swapping those files over changes free memory available.

You can check firmware and kernel versions after a reboot with:

uname -a
vcgencmd version

I have run out of time for the moment, but copying either set of files from the "bad" commit to the "good" one does not appear to solve the issue. I am missing to test the other permutation.

[gugolple]

Note that as well as having unavoidably larger page tables, lower code density, etc., there is a common 64-bit kernel that runs on Pi 3s, 4s, 5 and Z2W. This means that your Z2W has used some of its RAM to hold support for booting a Pi 5 to the point where it can access its SD card, USB, and Ethernet interfaces. kernel7.img is only for Pi 3s and Z2W, therefore it is much smaller. Note also that the 64-bit kernels are GZIP compressed - uncompressed they are roughly 25MB.

Correct, that affects memory utilization, not the amount of memory available to the system, that is the issue I am suffering.

[popcornmix]

I have run out of time for the moment, but copying either set of files from the "bad" commit to the "good" one does not appear to solve the issue. I am missing to test the other permutation.

I'd expect copying firmware from "bad" to "good" would either have no effect (i.e. it's not a firmware issue) or to make memory available worsen (i.e. it is a firmware issue). Can you confirm which result you had ("solving" wasn't an expected outcome from this test).

[gugolple]

The results from the improvised experiment:

	af3ab9	e63236
start*.elf af3ab9 fixup*.dat af3ab9	Correct	Bad
start*.elf e63236 fixup*.dat af3ab9	Correct	Bad
start*.elf af3ab9 fixup*.dat e63236	Correct	Does not boot
start*.elf e63236 fixup*.dat e63236	Correct	Bad

After seeing the results, I do believe the issue is with the new 6.6 kernel and not with the firmware.
Thank you so much @popcornmix for the support given. I believe the issue should be closed here and open in the kernel repo. Would you kindly confirm?

[popcornmix]

start*.elf and fixup*.dat should always be of the same version, so not booting is reasonable.

I've transferred this issue to the linux repo as that appears to be the cause of the increased memory.

As @pelwell says, it is worth testing how much of the increased memory is due to 64-bit kernel (which now also supports pi5, which the 32-bit kernel does not).

Is your OS 32-bit or 64-bit? If it's 32-bit then booting without arm_64bit=1 would be informative.
If it's 64-bit then it's trickier to test (will need a 32-bit install).

[gugolple]

As @pelwell says, it is worth testing how much of the increased memory is due to 64-bit kernel (which now also supports pi5, which the 32-bit kernel does not).

Is your OS 32-bit or 64-bit? If it's 32-bit then booting without arm_64bit=1 would be informative. If it's 64-bit then it's trickier to test (will need a 32-bit install).

Currently I'm running "Raspberry Pi OS (64-bit) : Raspberry Pi OS Lite".

What tests and versions would be intriguing to execute? I am open for suggestions.

[pelwell]

Although I've never quite got 460M for total mem, my results are close, seeing a ~60M drop when switching to the 6.6 kernel - whether by rpi-update or building my own.

Looking at the start of the /proc/iomem output with some annotations is instructive.

Before:

MemTotal:         440568 kB
00000000-1effffff : System RAM [-0.0M -> 496.0M]
  00000000-00000fff : reserved [-0.0M -> 496.0M]
  00210000-0121ffff : Kernel code [-16.1M -> 479.9M]
  01220000-0165ffff : reserved [-4.2M -> 475.7M]
  01660000-0199ffff : Kernel data [-3.2M -> 472.4M]
  0b200000-0ddfffff : reserved [-44.0M -> 428.4M]
  0de32000-0def2fff : reserved [-0.8M -> 427.7M]
  0def3000-0df66fff : reserved [-0.5M -> 427.2M]
  0df69000-0df69fff : reserved [-0.0M -> 427.2M]
  0df6a000-0df6afff : reserved [-0.0M -> 427.2M]
  0df6b000-1e08efff : reserved [-257.1M -> 170.1M]
  1e08f000-1e096fff : reserved [-0.0M -> 170.0M]
  1e097000-1effffff : reserved [-15.4M -> 154.6M]

After:

MemTotal:         372896 kB
00000000-1effffff : System RAM [-0.0M -> 496.0M]
  00000000-00000fff : reserved [-0.0M -> 496.0M]
  00210000-0136ffff : Kernel code [-17.4M -> 478.6M]
  01370000-0183ffff : reserved [-4.8M -> 473.8M]
  01840000-01b7ffff : Kernel data [-3.2M -> 470.6M]
  07140000-0ddfffff : reserved [-108.8M -> 361.8M]
  0de2c000-0deecfff : reserved [-0.8M -> 361.1M]
  0deed000-0df64fff : reserved [-0.5M -> 360.6M]
  0df67000-0df67fff : reserved [-0.0M -> 360.6M]
  0df68000-0df68fff : reserved [-0.0M -> 360.6M]
  0df69000-0df79fff : reserved [-0.1M -> 360.5M]
  0df7a000-1e08dfff : reserved [-257.1M -> 103.4M]
  1e08e000-1e08efff : reserved [-0.0M -> 103.4M]
  1e08f000-1e096fff : reserved [-0.0M -> 103.4M]
  1e097000-1effffff : reserved [-15.4M -> 88.0M]

Here you can clearly see that the third reserved section has gone from 44MB to 108MB, accounting for the ~60MB difference.
(Ah - I had reserved 5MB for kernel logging; without that I get similar numbers to you.)

The obvious next question is what that area is reserved for. The answer is "software IO TLB" - DMA bounce buffers:

[    0.000000] software IO TLB: mapped [mem 0x0000000009600000-0x000000000d600000] (64MB)

This memory is used as a staging post for DMA transfers when the target memory is out of range of the DMA controller. There is no such allocation in the 6.1 kernel, which is puzzling. I'll have to dig deeper.

By the way, the spi1-3cs in dtoverlay=vc4-kms-v3d,spi1-3cs is wrong. It's the name of an overlay, not a parameter, so must be at the start of its own dtoverlay line if want it to have any effect.

[pelwell]

The extra 64MB usage is caused by this commit: 370645f

Prior to this, the initialisation of the swiotlb buffers was smart - if all of RAM is addressable by the DMA controller(s) then no bounce buffers are required. This is certainly true on a Z2W with only 512MB, so you save the 64MB carve-out. The above commit, which appeared in Linux 6.5, says that bounce buffers may be required if DMA transfers are not cache-line-aligned, so it is better to have the bounce buffers unconditionally allocated. The DMA_BOUNCE_UNALIGNED_KMALLOC option is turned on by the arm64 Kconfig file. I don't see any way to opt-out without modifying the kernel source - as a config setting without a description it isn't one you can alter from a defconfig file.

Needless to say, the 32-bit (ARCH=arm) Kconfig files don't set DMA_BOUNCE_UNALIGNED_KMALLOC - yet another reason you would be better off with a 32-bit build.

[JamesH65]

This is an interesting and related read https://lwn.net/Articles/940973/#:~:text=The%20kernel's%20software%20I%2FO,as%20newer%20peripherals%20came%20along.

[pelwell]

Given that 6.1 was quite happy without this option being defined, we opted to not enable it on 6.6 - see 3e0065d. This change will be in future releases, but if you wait about 40 minutes (until around 15:10GMT) you should be able to install an early build of this beta kernel by running sudo rpi-update rpi-6.6.y.

For the record, this is the same RAM usage information taken from 6.6 with the patch applied (and without my oversized kernel log buffer):

MemTotal:         476064 kB
00000000-1effffff : System RAM [-0.0M -> 496.0M]
  00000000-00000fff : reserved [-0.0M -> 496.0M]
  00210000-0136ffff : Kernel code [-17.4M -> 478.6M]
  01370000-0183ffff : reserved [-4.8M -> 473.8M]
  01840000-01b7ffff : Kernel data [-3.2M -> 470.6M]
  0d600000-0ddfffff : reserved [-8.0M -> 462.6M]
  0de2c000-0deecfff : reserved [-0.8M -> 461.8M]
  0deed000-0df64fff : reserved [-0.5M -> 461.3M]
  0df67000-0df67fff : reserved [-0.0M -> 461.3M]
  0df68000-0df68fff : reserved [-0.0M -> 461.3M]
  0df69000-0df79fff : reserved [-0.1M -> 461.3M]
  0df7a000-1e08dfff : reserved [-257.1M -> 204.2M]
  1e08e000-1e08efff : reserved [-0.0M -> 204.2M]
  1e08f000-1e096fff : reserved [-0.0M -> 204.1M]
  1e097000-1effffff : reserved [-15.4M -> 188.7M]

[popcornmix]

This is an interesting and related read

The patch series described there is in our 6.6 kernel (it wasn't in 6.5 and prior ones).
So it isn't solving this issue.

[gugolple]

Given that 6.1 was quite happy without this option being defined, we opted to not enable it on 6.6 - see 3e0065d. This change will be in future releases, but if you wait about 40 minutes (until around 15:10GMT) you should be able to install an early build of this beta kernel by running sudo rpi-update rpi-6.6.y.

Seeing that this is not the only project with this issue, maybe the functionality could be kept with a smaller fix just be setting the kernel boot parameter of "swiotlb=0" as they did in the Yocto project for qemu images.

[pelwell]

That's not a bad solution, but this works for us for now.

[gugolple]

Can confirm the fix is working.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           465Mi       161Mi       133Mi       1.4Mi       222Mi       303Mi

$ uname -a
Linux raspberrypi 6.6.17-v8+ #1 SMP PREEMPT Wed Feb 21 16:23:00 UTC 2024 aarch64 GNU/Linux

$ vcgencmd version
Oct 17 2023 15:42:53 
Copyright (c) 2012 Broadcom
version 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 (clean) (release) (start_cd)

Thank you everyone for the support.

[pelwell]

Thank you - without your observation all Z2W owners would be missing 12.5% of their RAM for no reason.