'get-message' continuation fired more than once? about fibers

[civodul]

Adding pk calls in channels.scm shows that (put-message n) is called twice with the same value of n when things go wrong.

With GUILE_JIT_THRESHOLD=-1, the problem seems to go away.

Could we have a program counter synchronization issue or similar when returning from JITted code? Thoughts, @wingo?

(This reminds me of https://issues.guix.gnu.org/28211#23, though that one didn't involve JIT particularly.)

[wingo]

AFAICS this program is deterministic. On aarch64, is the number at which we see a problem deterministic, too?

[wingo]

Does the problem reproduce on qemu?

[civodul]

Hey!

AFAICS this program is deterministic. On aarch64, is the number at which we see a problem deterministic, too?

@wingo No, the number changes at each run.

In qemu-aarch64 running on x86_64 (via binfmt_misc support), the problem does not occur.

FWIW, the AArch64 machine I'm testing on is a SoftIron OverDrive with this CPU:

$ lscpu 
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A57
    Model:               2
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r1p2
    BogoMIPS:            500.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-3
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Vulnerable
  Srbds:                 Not affected
  Tsx async abort:       Not affected

[wingo]

On your machine if you taskset to a specific CPU does that change anything? Trying to understand here.

[civodul]

@wingo The problem almost disappears (happens once every 10-20 runs instead of once every run or so) when binding the process to a single core, like so:

guix shell guile guile-fibers hwloc -- hwloc-bind core:0 -- sh -c 'while guile ../fibers-channel.scm ; do : ; done'

[Millak]

guix shell guile guile-fibers hwloc -- hwloc-bind core:0 -- sh -c 'while guile ../fibers-channel.scm ; do : ; done'

I used the same snippet on one of my machines. On my pbp as far as I can tell core0 is an A53 core. In the loop I had failures at 10, 5, 7, 1, 27, 3, 1, 20, 3, 2, 7, and then I stopped it.

Without pinning it to one cpu I had failures at 1, 6, 3, 2, 1, 1, 1, 3, 2, 7, 7, 2 and then I stopped it.

[wingo]

If this really only happens for JIT-compiled code, then one possibility would be bad codegen for atomics. It would be in https://gitlab.com/wingo/lightening/-/blob/main/lightening/aarch64-cpu.c#L2520-2565. But that would be weird given that this test is single-threaded!