Bug in restoring a checkpoint using SimpleProcessor and SwitchableProcessor in ARM & KVM setup about gem5

[powerjg]

It's doesn't surprise me too much that the SimpleSwitchableProcessor doesn't work well. This was never meant to be used for anything except the absolute simplest use cases.

What happens if you use just the KVM processor to take the checkpoint and then restore with atomic or the simple timing?

[mbabaie]

Hi Jason,
Thank you very much for your reply.

I have tested the same experiments with simpleProcessor(), using the same repo and run script.
Here are the results for testing:

1. To generate the checkpoint, one should use this command:
   Note: we use a simpleProcessor(KVM) to take the checkpoint.

build/ARM/gem5.opt --outdir=m5out_chkpt_kvm_simpleProcessor configs/example/gem5_library/arm-ubuntu-run-with-kvm.py --take-chkpt=True --chkpt-cpu-switchable=False

2. To restore the checkpoint with a simpleProcessor(Timing), use the command below:

build/ARM/gem5.opt --outdir=m5out_rstr_simpleTiming --debug-flags=ExecAll configs/example/gem5_library/arm-ubuntu-run-with-kvm.py --take-chkpt=False --chkpt-dir=m5out_chkpt_kvm_simpleProcessor/checkpoint --rstr-cpu-switchable=False

Here's the terminals output:

info: Using default config
Reading the checkpoint.
warn: Setting the checkpoint path via the Simulator constructor is deprecated and will be removed in future releases of gem5. Please set this through via the appropriate workload function (i.e., `set_se_binary_workload` or `set_kernel_disk_workload`). If both are set the workload function set takes precedence.
Global frequency set at 1000000000000 ticks per second
warn: No dot file generated. Please install pydot to generate the dot file and pdf.
src/mem/dram_interface.cc:690: warn: DRAM device capacity (16384 Mbytes) does not match the address range assigned (1024 Mbytes)
src/mem/dram_interface.cc:690: warn: DRAM device capacity (16384 Mbytes) does not match the address range assigned (1024 Mbytes)
src/sim/kernel_workload.cc:46: info: kernel located at: /home/babaie/.cache/gem5/arm64-linux-kernel-5.4.49
src/base/loader/symtab.cc:95: warn: Cannot insert a new symbol table due to name collisions. Adding prefix to each symbol's name can resolve this issue.
src/base/statistics.hh:279: warn: One of the stats is a legacy stat. Legacy stat is a stat that does not belong to any statistics::Group. Legacy stat is deprecated.
board.vncserver: Listening for connections on port 5913
board.terminal: Listening for connections on port 3508
board.realview.uart1.device: Listening for connections on port 3509
board.realview.uart2.device: Listening for connections on port 3510
board.realview.uart3.device: Listening for connections on port 3511
board.remote_gdb: Listening for connections on port 7013
src/arch/arm/isa.cc:1559: warn: Checkpoint value for register id_pfr1 does not match current configuration (checkpointed: 0x1, current: 0x1011)
src/arch/arm/isa.cc:1559: warn: Checkpoint value for register id_aa64pfr0_el1 does not match current configuration (checkpointed: 0x1100000022, current: 0x1100002222)
src/arch/arm/isa.cc:1559: warn: Checkpoint value for register id_pfr1 does not match current configuration (checkpointed: 0x1, current: 0x1011)
src/arch/arm/isa.cc:1559: warn: Checkpoint value for register id_aa64pfr0_el1 does not match current configuration (checkpointed: 0x1100000022, current: 0x1100002222)
src/dev/arm/energy_ctrl.cc:252: warn: Existing EnergyCtrl, but no enabled DVFSHandler found.
src/sim/simulate.cc:199: info: Entering event queue @ 1317305722654.  Starting simulation...

While using the debug-flags=ExecAll that should print all committed instructions by the cores, nothing is printed. The simulation also does not end. Using debug-flags=Event shows some events keep being rescheduled and executed:

...
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 executed @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 executed @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 executed @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 executed @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 executed @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 executed @ 1317305924784
1317305924784: board.processor.cores1.core.wrapped_function_event: EventFunctionWrapped 288 rescheduled @ 1317305924784
...

[powerjg]

@giactra do you have any hints as to how to debug taking a checkpoint with Arm KVM and restoring?

[kaustav-goswami]

Hi, for the SimpleProcessor case, ARM KVM uses ArmDefaultRelease.for_kvm() as the release for the ArmBoard. This removes SECURITY and VIRTUALIZATION extensions from ArmDefaultRelease(). When restoring the system using anything other than KVM CPUs, the ArmBoard uses release=ArmDefaultRelease(). Therefore the warn for id_pfr1 and id_aa64pfr0_el1 were shown during restore. The simulation starts but it gets stuck at gem5::ArmISA::ISA while reading the wrong register value. The ArmBoard has an init param release= which can be used to get around this problem. I'll document this case, and, add checks within the board and open a new PR.

[giactra]

I don't know if changing the release will fix the issue, but IMHO we shouldn't just document things better, we should provide a less obscure platform.

I think we should remove complexity from the ArmBoard:

https://github.com/gem5/gem5/blob/develop/src/python/gem5/components/boards/arm_board.py#L119

Which is changing the release secretly to make it work with KVM depending on the CPUs in use.
We already have a KVM specific config, which is arm-ubuntu-run-with-kvm.py. It means that if you want to use KVM, you should use that platform. We should move the ArmRelease.for_kvm from the ArmBoard to that config.

You want to boot linux without KVM?

Use arm-ubuntu-run.py

You want to use KVM?

Use arm-ubuntu-run-with-kvm.py

[kaustav-goswami]

Okay, I'll open a new PR fixing the release.