Discussion:
gic-v2 and SGI interrupts on boot CPU
Dmitry Skorodumov via freebsd-arm
2021-04-20 22:02:21 UTC
Permalink
Hi


It looks like code for gic-v2 in FreeBSD not quite correctly relies on implementation defined behaviour of GIC.

The g<file:///Users/sdmitry/Downloads/IHI0048B_b_gic_architecture_specification.pdf>ic 2.0 spec https://developer.arm.com/documentation/ihi0048/bb chapter 3.2.2 "Interrupt controls in the GIC" states the following:

"Whether SGIs are permanently enabled, or can be enabled and disabled by writes to the GICD_ISENABLERn and GICD_ICENABLERn, is IMPLEMENTATION DEFINED."

But code in sys/arm/arm/gic.c assumes that SGI are always enabled and doesn't configure them at initialization. They are initialized only for secondary CPUs - in arm_gic_init_secondary().

For sure it is a rather minor issue, since all appears to be ok in gic-v3 (v3 code enables SGIs for all CPUs, including the boot one). And even if platform supports only gic-v2, likely SGIs are always enabled anyway. So, my post is rather pedantic notice without real life case.

Dmitry
Michael Tuexen
2021-04-20 22:38:44 UTC
Permalink
Post by Dmitry Skorodumov via freebsd-arm
Hi
It looks like code for gic-v2 in FreeBSD not quite correctly relies on implementation defined behaviour of GIC.
"Whether SGIs are permanently enabled, or can be enabled and disabled by writes to the GICD_ISENABLERn and GICD_ICENABLERn, is IMPLEMENTATION DEFINED."
But code in sys/arm/arm/gic.c assumes that SGI are always enabled and doesn't configure them at initialization. They are initialized only for secondary CPUs - in arm_gic_init_secondary().
For sure it is a rather minor issue, since all appears to be ok in gic-v3 (v3 code enables SGIs for all CPUs, including the boot one). And even if platform supports only gic-v2, likely SGIs are always enabled anyway. So, my post is rather pedantic notice without real life case.
Dear all,

if I understand things correctly, the problem described is the cause which does not
allow to use more than one CPU core in FreeBSD when running on Parallels Desktop on
an M1 based Mac. It runs perfectly well with one core, but with multiple cores it
locks up during boot.

It would be great if some could propose a patch. I would be more than happy to test it.

Best regards
Michael
Post by Dmitry Skorodumov via freebsd-arm
Dmitry
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
Greg V
2021-04-21 00:40:22 UTC
Permalink
Post by Michael Tuexen
Post by Dmitry Skorodumov via freebsd-arm
Hi
It looks like code for gic-v2 in FreeBSD not quite correctly relies on implementation defined behaviour of GIC.
"Whether SGIs are permanently enabled, or can be enabled and disabled by writes to the GICD_ISENABLERn and GICD_ICENABLERn, is IMPLEMENTATION DEFINED."
But code in sys/arm/arm/gic.c assumes that SGI are always enabled and doesn't configure them at initialization. They are initialized only for secondary CPUs - in arm_gic_init_secondary().
For sure it is a rather minor issue, since all appears to be ok in gic-v3 (v3 code enables SGIs for all CPUs, including the boot one). And even if platform supports only gic-v2, likely SGIs are always enabled anyway. So, my post is rather pedantic notice without real life case.
Dear all,
if I understand things correctly, the problem described is the cause which does not
allow to use more than one CPU core in FreeBSD when running on Parallels Desktop on
an M1 based Mac. It runs perfectly well with one core, but with multiple cores it
locks up during boot.
Hmm if I'm reading it correctly, the gicv2 driver *does* do this on secondary CPUs, just not on the boot one. Which doesn't sound like something that would cause SMP boot to break but single-core to still work.

Seems like people using QEMU with Hypervisor.framework patches do have SMP working fine:
https://gist.github.com/ctsrc/a1f57933a2cde9abc0f07be12889f97f so go bother Parallels about their bugs ;)
Michael Tuexen
2021-04-21 06:44:25 UTC
Permalink
Post by Greg V
Post by Michael Tuexen
Post by Dmitry Skorodumov via freebsd-arm
Hi
It looks like code for gic-v2 in FreeBSD not quite correctly relies on implementation defined behaviour of GIC.
"Whether SGIs are permanently enabled, or can be enabled and disabled by writes to the GICD_ISENABLERn and GICD_ICENABLERn, is IMPLEMENTATION DEFINED."
But code in sys/arm/arm/gic.c assumes that SGI are always enabled and doesn't configure them at initialization. They are initialized only for secondary CPUs - in arm_gic_init_secondary().
For sure it is a rather minor issue, since all appears to be ok in gic-v3 (v3 code enables SGIs for all CPUs, including the boot one). And even if platform supports only gic-v2, likely SGIs are always enabled anyway. So, my post is rather pedantic notice without real life case.
Dear all,
if I understand things correctly, the problem described is the cause which does not
allow to use more than one CPU core in FreeBSD when running on Parallels Desktop on
an M1 based Mac. It runs perfectly well with one core, but with multiple cores it
locks up during boot.
Hmm if I'm reading it correctly, the gicv2 driver *does* do this on secondary CPUs, just not on the boot one. Which doesn't sound like something that would cause SMP boot to break but single-core to still work.
https://gist.github.com/ctsrc/a1f57933a2cde9abc0f07be12889f97f so go bother Parallels about their bugs ;)
I reported the issue in https://forum.parallels.com/threads/freebsd-support.352663/ and Dmitry said
that they found the issue and fixed it within Parallels, so FreeBSD 13 will be supported by
a future update of Parallels. But since the cause is based on an implementation depended behaviour
and FreeBSD seems to assume one particular one, I suggested to report it here. So I think the
argument that there is some platform, which works, does not imply that the FreeBSD behaviour is correct.


As Vincent figured out, a potential fix for FreeBSD is in https://reviews.freebsd.org/D26975. I can't
comment on if the patch is architecturally correct, I can confirm that it resolves the problem.

Best regards
Michael
Michael Tuexen
2021-04-21 06:48:30 UTC
Permalink
This may actually be related to the Parallels issue. We had a similar issue with ESXi ARM Fling initially where it would only work on 1 vCPU but not more, and it was related to the GIC code. This very much is worth investigating for getting FreeBSD functional under Parallels!
Oh, and yes, I've bothered Parallels a lot... I unofficially got a reply essentially stating that the bug "isn't a priority" (they primarily care about Windows VMs and seems to only marginally care about Linux, with FreeBSD and other OSes not even remotely in their sights)
Eventually they looked into the issue https://forum.parallels.com/threads/freebsd-support.352663/, figured
out what is going wrong and will fix it in their code base...
I can understand that they have to prioritise on which guest OS they focus first. So I'm happy now.

Best regards
Michael
Post by Michael Tuexen
Post by Dmitry Skorodumov via freebsd-arm
Hi
It looks like code for gic-v2 in FreeBSD not quite correctly relies on implementation defined behaviour of GIC.
"Whether SGIs are permanently enabled, or can be enabled and disabled by writes to the GICD_ISENABLERn and GICD_ICENABLERn, is IMPLEMENTATION DEFINED."
But code in sys/arm/arm/gic.c assumes that SGI are always enabled and doesn't configure them at initialization. They are initialized only for secondary CPUs - in arm_gic_init_secondary().
For sure it is a rather minor issue, since all appears to be ok in gic-v3 (v3 code enables SGIs for all CPUs, including the boot one). And even if platform supports only gic-v2, likely SGIs are always enabled anyway. So, my post is rather pedantic notice without real life case.
Dear all,
if I understand things correctly, the problem described is the cause which does not
allow to use more than one CPU core in FreeBSD when running on Parallels Desktop on
an M1 based Mac. It runs perfectly well with one core, but with multiple cores it
locks up during boot.
Hmm if I'm reading it correctly, the gicv2 driver *does* do this on secondary CPUs, just not on the boot one. Which doesn't sound like something that would cause SMP boot to break but single-core to still work.
https://gist.github.com/ctsrc/a1f57933a2cde9abc0f07be12889f97f so go bother Parallels about their bugs ;)
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
Michael Tuexen
2021-04-21 06:49:41 UTC
Permalink
Doing some more digging, I came across this post from a while ago.
https://reviews.freebsd.org/D26975
I just built a kernel with this patch, and am now running 8 vCPUs under Parallels on my MacBook Air M1.
I can also confirm that the patch fixes the issue. Thanks a lot for finding the patch!

Best regards
Michael
You can see evidence of it here!
https://twitter.com/DarkainMX/status/1384724684902600706
This may actually be related to the Parallels issue. We had a similar issue with ESXi ARM Fling initially where it would only work on 1 vCPU but not more, and it was related to the GIC code. This very much is worth investigating for getting FreeBSD functional under Parallels!
Oh, and yes, I've bothered Parallels a lot... I unofficially got a reply essentially stating that the bug "isn't a priority" (they primarily care about Windows VMs and seems to only marginally care about Linux, with FreeBSD and other OSes not even remotely in their sights)
Post by Michael Tuexen
Post by Dmitry Skorodumov via freebsd-arm
Hi
It looks like code for gic-v2 in FreeBSD not quite correctly relies on implementation defined behaviour of GIC.
"Whether SGIs are permanently enabled, or can be enabled and disabled by writes to the GICD_ISENABLERn and GICD_ICENABLERn, is IMPLEMENTATION DEFINED."
But code in sys/arm/arm/gic.c assumes that SGI are always enabled and doesn't configure them at initialization. They are initialized only for secondary CPUs - in arm_gic_init_secondary().
For sure it is a rather minor issue, since all appears to be ok in gic-v3 (v3 code enables SGIs for all CPUs, including the boot one). And even if platform supports only gic-v2, likely SGIs are always enabled anyway. So, my post is rather pedantic notice without real life case.
Dear all,
if I understand things correctly, the problem described is the cause which does not
allow to use more than one CPU core in FreeBSD when running on Parallels Desktop on
an M1 based Mac. It runs perfectly well with one core, but with multiple cores it
locks up during boot.
Hmm if I'm reading it correctly, the gicv2 driver *does* do this on secondary CPUs, just not on the boot one. Which doesn't sound like something that would cause SMP boot to break but single-core to still work.
https://gist.github.com/ctsrc/a1f57933a2cde9abc0f07be12889f97f so go bother Parallels about their bugs ;)
_______________________________________________
https://lists.freebsd.org/mailman/listinfo/freebsd-arm
Loading...