# Issue with seL4 & ARMv8 config for MMU

**URL:** <https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940>\
**Category:** seL4 kernel\
**Tags:** arm\
**Created:** [January 28, 2025, 3:01pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940 "2025-01-28T15:01:40Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [January 28, 2025, 3:01pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/1 "2025-01-28T15:01:40Z")

</div>

Dear Community members,

I have used maaxboard bsp in amother armv8 cortex-a53 processor (same as maaxboard) & facing below issue specially related to mmu:

1. In the elfloader crt0.s file if i add the below code to disable mmu  
MRS x3, SCTLR\_EL1  
BIC x3, x3, (1\<\<0)  
BIC x3, x3, (1\<\<2)  
BIC x3, x3, (1\<\<12)  
MSR SCTLR\_EL1, x3  
ISB  
In the maaxboard it works fine, but the other armv8 the execution stops for a while (few sec) and then continues.

2. The the “kernel/src/arch/arm/64/kernel/vspace.c” activete\_kernel\_vspace()  
I have to comment below function to boot kernel in the other armv8 cortex-a53 h/w as:  
BOOT\_CODE void activate\_kernel\_vspace(void)  
{  
cleanInvalidateL1Caches();  
setCurrentKernelVSpaceRoot(ttbr\_new(0, addrFromKPPtr(armKSGlobalKernelPGD)));

3. When the user space “setCurrentUserVSpaceRoot()” is called by the root process multiple times to update the pagetable. During the process the system gets hanged abruptly. MSR(“ttbr0\_el1”, ttbr.words[0]); is not stable and works just by chance for some time, but mostly hangs.

I am able to boot the system till the user land but mmu is not working properly with two observations  
a) disabling mmu using MRS x3, SCTLR\_EL1  
BIC x3, x3, (1\<\<0) works but stops for several sec.  
b) updating ttbr0\_el1 is not stable & the system hangs abruptly.

Did any one faced the similar issue & can you please suggest, because i tried a lot of mmu related config change and nothing helped.

Regards,  
Misbah

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [January 29, 2025, 10:41am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/2 "2025-01-29T10:41:50Z")

</div>

seL4 expects to run with MMU enabled, why are you disabling it?

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [January 29, 2025, 10:48am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/3 "2025-01-29T10:48:44Z")

</div>

I tried it in the elfloader crt0.s in the \_start function file before calling main(). This is just for experimental purpose to disable mmu and reenable it. Because “ttbr0\_el1” update is causing problem in aarch64 cortex-a53 board, but the same code works fine in imx8 (ARM cortex-a53) board.

seL4 release for maaxboard works fine for mmu like update to “ttbr0\_el1” on maaxboard but not on another cortex-a53 based NXP chipset having same ARM architecture. ?

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [January 29, 2025, 11:54am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/4 "2025-01-29T11:54:22Z")

</div>

What SoC are you using? Does your SoC implement FEAT\_D128 and is it enabled perhaps? Which values of ttbr0\_el1 give problems and which values work?

I highly recommend reverting all MMU disabling code and either debug this problem fully within Elfloader or in the seL4 kernel, but not both. Also try using a non-SMP kernel to simplify debugging, if you haven’t already.

Edit: Also, what exception level are you running, EL1 or EL2? Try enabling or disabling hypervisor support to see if it makes a difference.

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [January 30, 2025, 9:56am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/5 "2025-01-30T09:56:46Z")

</div>

Thanks for your reply.

Please find the answers to your questions as below:

> > >

Does your SoC implement FEAT\_D128

> > > No

Which values of ttbr0\_el1 give problems and which values work?

> > > Any Value of ttbr0\_el1 gives the problem.

debug this problem fully within Elfloader or in the seL4 kernel. Also try using a non-SMP kernel

> > > I am trying to debug in the elfloader. I am trying non-SMP kernel as well.

Also, what exception level are you running, EL1 or EL2

> > > Running EL1.

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [January 31, 2025, 1:22pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/6 "2025-01-31T13:22:12Z")

</div>

Your post got a bit mangled.

BIC x3, x3, (1\<\<2)  
BIC x3, x3, (1\<\<12)

Disables the data and instruction caches. I don’t think that’s safe to do without cleaning all caches first.

Anyway, your problem seems to be unrelated to seL4 as such, as any update to ttbr0\_el1 gives problems. You might be hitting some errata or there is a mismatch in expectations between what Elfloader/seL4 does and what your hardware expects. Is Linux running on your board?

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 3, 2025, 6:09am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/7 "2025-02-03T06:09:17Z")

</div>

Hi Indan,

My hardware is running Linux, issue is with seL4.  
My h/w & maaxboard are same interms of ARM architecture (Cortex-a53) on maaxboard mmu works fine hence i cloned the same ARM settings and hence everythings works fine except the mmu.  
There are 3 observations with respect to the mmu problem:

1. disabling mmu in elfloader crt0.s file in mmaxboard has no effect but in my board the program execution stops for few sec before proceeding with the boot.
2. In kernel after “after ttbr0\_el1” is updated InvalidateLocalTLB() hangs
3. In user space update to “ttbr0\_el1” abroptly hangs at any time. (unstable)

There is no Errata with respect to ARM on this issue. Hence the question is what is missed or not considered while writing the ARM specific code in seL4 for Cortex-a53/aarch64.

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [February 3, 2025, 12:11pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/8 "2025-02-03T12:11:26Z")

</div>

If you’re using the same SoC as the maaxboard board, your issue is most likely related to bootup. E.g. there is a bug in your board bring-up which corrupts some memory or system registers somewhere that later gives those MMU problems.

If you are using a different SoC that only also uses the Cortex-A53 CPU, you really can’t just use the maaxboard seL4 platform, you have to do a proper port of seL4.

NXP’s BSP and the Linux in there support all NXP hardware, so the maaxboard bsp working on your SoC doesn’t say that much.

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 3, 2025, 1:52pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/9 "2025-02-03T13:52:47Z")

</div>

To validate the same i want to run seL4 in HYPERVISOR mode.  
What setting/sample project is need to build seL4 in ARM\_HYPERWISOR mode such that different configuration of mmu shall be used in EL2.

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [February 4, 2025, 11:04am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/10 "2025-02-04T11:04:12Z")

</div>

For sel4test, all you need to do is enable ARM\_HYP, you don’t need to actually run a VM or anything. It may not officially be supported for the maaxboard, but it should work.

`../init-build.sh -DAARCH64=TRUE -DPLATFORM=maaxboard -DARM_HYP=TRUE`

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 4, 2025, 2:40pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/11 "2025-02-04T14:40:24Z")

</div>

Can you please hint as how the mode shall be switched to EL2 when compiled with -DARM\_HYP=1 Because in maaxboard BSP i could not find the code to switch mode from EL1(default mode) to EL2. Because the function “is\_hyp\_mode()” is returning mode as EL1 in my case below boot log for reference-

ELF-loader started on CPU: ARM Ltd. Cortex-A53 r0p4  
paddr=[80a7c000…80fc51bf]  
No DTB passed in from boot loader.  
Looking for DTB in CPIO archive…found at 80bc3810.  
Loaded DTB from 80bc3810.  
paddr=[80248000…80254fff]  
ELF-loading image ‘kernel’ to 80000000  
paddr=[80000000…80247fff]  
vaddr=[8080000000…8080247fff]  
virt\_entry=8080000000  
ELF-loading image ‘sel4test-driver’ to 80255000  
paddr=[80255000…8068bfff]  
vaddr=[400000…836fff]  
virt\_entry=40f0b0  
Enabling MMU and paging  
Jumping to kernel-image entry point…

ELF-LOADER: Synchronous exception received:  
esr\_el1: 86000004  
elr\_el1: 8080000000  
spsr\_el1: 600003c5  
far\_el1: 8080000000  
abort() called.

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [February 4, 2025, 6:26pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/12 "2025-02-04T18:26:06Z")

</div>

You probably need to modify U-Boot to stay in EL2. But if your BSP doesn’t support EL2 by default, I wouldn’t bother trying to get it to work and focus on fixing your original problem instead.

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 5, 2025, 8:32am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/13 "2025-02-05T08:32:14Z")

</div>

We have cloned the maaxboard BSP for our hardware, both being Cortex-a53 & made changes related to peripheral mapping. There is no arm specific changes, we expect arm specific changes should work the same way its working in maaxboard:

When i try to build maaxboard image and test it on maaxboard both EL1 & EL2 mode works properly, but in my hardware El1 mode have mmu related issue as explained in the previous thread & EL2 is not getting selected while reading “currentEL” register. The maaxboard build and run log is as below-

## ../init-build.sh -DAARCH64=TRUE -DPLATFORM=maaxboard -DARM\_HYP=TRUE Build Command:$ninja

Bootlogs:

u-boot=\> fatload mmc 0:1 0x40000000 sel4test-driver-image-arm-maaxboardinhyp  
5739160 bytes read in 268 ms (20.4 MiB/s)  
u-boot=\> bootelf 0x40000000

## Starting application at 0x40a81000 …

ELF-loader started on CPU: ARM Ltd. Cortex-A53 r0p4  
paddr=[40a81000..40fde137]  
No DTB passed in from boot loader.  
Looking for DTB in CPIO archive…found at 40be2d88.  
Loaded DTB from 40be2d88.  
paddr=[40246000..40250fff]  
ELF-loading image ‘kernel’ to 40000000  
paddr=[40000000..40245fff]  
vaddr=[8040000000..8040245fff]  
virt\_entry=8040000000  
ELF-loading image ‘sel4test-driver’ to 40251000  
paddr=[40251000..40687fff]  
vaddr=[400000..836fff]  
virt\_entry=40f0b0  
Enabling hypervisor MMU and paging  
Jumping to kernel-image entry point…

Build Command:$../init-build.sh -DAARCH64=TRUE -DPLATFORM=maaxboard  
Build Command:$ninja

* * *

Bootlogs:

u-boot=\> fatload mmc 0:1 0x40000000 sel4test-driver-image-arm-maaxboardinosmode  
5620456 bytes read in 250 ms (21.4 MiB/s)  
u-boot=\> bootelf 0x40000000

## Starting application at 0x40a77000 …

ELF-loader started on CPU: ARM Ltd. Cortex-A53 r0p4  
paddr=[40a77000..40fb1137]  
No DTB passed in from boot loader.  
Looking for DTB in CPIO archive…found at 40bbb530.  
Loaded DTB from 40bbb530.  
paddr=[40241000..4024bfff]  
ELF-loading image ‘kernel’ to 40000000  
paddr=[40000000..40240fff]  
vaddr=[ffffff8040000000..ffffff8040240fff]  
virt\_entry=ffffff8040000000  
ELF-loading image ‘sel4test-driver’ to 4024c000  
paddr=[4024c000..40682fff]  
vaddr=[400000..836fff]  
virt\_entry=40f0b0  
Enabling MMU and paging  
Jumping to kernel-image entry point…

* * *

The question is which part of the code selects EL2/EL1 while compiling for maaxboard using the flag “-DARM\_HYP=TRUE” because the bootloader for maaxboard is the same.

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [February 5, 2025, 10:23am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/14 "2025-02-05T10:23:39Z")

</div>

> The question is which part of the code selects EL2/EL1 while compiling for maaxboard using the flag “-DARM\_HYP=TRUE” because the bootloader for maaxboard is the same.

The bootloader determines the initial exception level, either EL1 or EL2. Elfloader drops down to EL1 if seL4 is compiled without hypervisor support and Elfloader is booted in EL2, otherwise it stays in EL2. You can’t change from EL1 to EL2, you can only drop exception levels. What exception level the bootloader runs at depends on either the hardware and/or the earlier board bootup code and configuration.

Considering your hardware is different, you want to use either the `KernelCustomDTSOverlay` or `KernelCustomDTSOverlay` options to modify the maaxboard’s DTS to match your actual hardware. Also double check whether all the other settings in `src/plat/maaxboard/config.cmake` are correct for your board (should be fine if you’re using an i.MX 8M).

From what you’ve told me, I think on your hardware Elfloader/seL4 is trying to use some device memory as normal memory, and that won’t work of course. It would explain the one second delay and other weird behaviour.

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 6, 2025, 11:28am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/15 "2025-02-06T11:28:15Z")

</div>

“From what you’ve told me, I think on your hardware Elfloader/seL4 is trying to use some device memory as normal memory, and that won’t work of course. It would explain the one second delay and other weird behavior.”

I am trying to get the latest bootloader for my hardware platform because with the current bootloader is not loading EL2 seL4 Image in EL2 mode.

Can you please provide some hint where are the possibilities of accessing device memory as Normal. For debugging purpose i already disabled almost all the peripherals in my dts file except serial.  
The ddr memory in dts is configured as \<0x00 0x80000000 0x00 0x20000000\>. (With respect to maaxboard this is the only change & the serial driver memory address). The TCR, MIDR, TTBR, etc are exactly same as that of maaxboard (No ARM specific code changed)

---

<div class="post-metadata">

**Author:** ![Indan](https://yyz2.discourse-cdn.com/free1/user_avatar/sel4.discourse.group/indan/32/185_2.png) [@Indan](https://sel4.discourse.group/u/Indan)\
**Post date:** [February 6, 2025, 12:35pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/16 "2025-02-06T12:35:57Z")

</div>

All the memory addresses used by the kernel can be found in build/kernel/gen\_headers/plat/machine/devices\_gen.h and build/kernel/gen\_headers/plat/machine/platform\_gen.yaml, both generated from the DTS files. Any memory not in there is treated as device memory.

Check all addresses in those files with your SoC’s memory map to see if they are correct. If not, you may need to add one or more `reserved-memory` memory regions with the `no-map` attribute to your `KernelCustomDTSOverlay` file to tell seL4 not to use it.

If you are using the same SoC and SPL as for the maaxboard, with the only difference the memory amount and serial output, it should work fine. Check the `bdinfo` U-Boot output for both the maaxboard and your board and compare the differences and see if they match your DTS.

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 26, 2025, 8:53am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/17 "2025-02-26T08:53:20Z")

</div>

We communicated to NXP about the issue because we are using the NXP chip similar to i.mx8 and cortex-A53. They asked for some change in the u-boot and with that we are able to launch seL4 image as hypervisor. But still our main issue remains the same in EL2 mode also that is:

1. Invalidate local TLB is getting hanged.
2. If we bypass it, then update to “ttbr0\_el” is suddenly getting stopped in the user space.

I have written to ARM community & NXP community to understand the reason as why/when InvalidateLocalTLB “asm volatile(“tlbi vmalle1”);”  
Because linux is running this platform hence NXP is not interested it. And there was no support from ARM either.

InvalidateLocalTLB is called in kernel “kernel/src/arch/arm/64/kernel/vspace.c”

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 26, 2025, 8:59am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/18 "2025-02-26T08:59:16Z")

</div>

I tried to create " `KernelCustomDTSOverlay" as suggested it didn’t help.

Its not the same SoC as i.MX8 but same ARM core cortexA-53, the ram address starts from 0x80000000 & in maaxboard its 0x40000000.

---

<div class="post-metadata">

**Author:** ![kent-mcleod2](https://avatars.discourse-cdn.com/v4/letter/k/34f0e0/32.png) [@kent-mcleod2](https://sel4.discourse.group/u/kent-mcleod2)\
**Post date:** [February 26, 2025, 10:26am UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/19 "2025-02-26T10:26:24Z")

</div>

Are you able to provide any details of the device tree for the platform that you are targeting?

---

<div class="post-metadata">

**Author:** ![Misbah](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Misbah](https://sel4.discourse.group/u/Misbah)\
**Post date:** [February 26, 2025, 1:30pm UTC](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940/20 "2025-02-26T13:30:01Z")

</div>

# The dts file is too big to be attached or copied here … Coping the overlay & platform\_gen.yaml as below:

## overlay.dts

/ {  
memory@80000000 {  
device\_type = “memory”;  
reg = \< 0x00 0x80000000 0x00 0x40000000 \>;  
};

```
chosen {
	seL4,elfloader-devices =
		"serial0",
		&{/firmware/psci},
		&{/timer};

	seL4,kernel-devices =
		"serial0",
		&{/soc/interrupt-controller@50800000},
		&{/timer};
};
    reserved-memory {
            #address-cells = <0x02>;
            #size-cells = <0x02>;
            ranges;

            shm@50800000 {
                    compatible = "nxp,s32cc-shm";
                    reg = <0x00 0x50800000 0x00 0x10000>;
                    no-map;
            };

            shm@50900000 {
                    compatible = "nxp,s32cc-shm";
                    reg = <0x00 0x50900000 0x00 0x100000>;
                    no-map;
            };

	shm@50000000 {
                    compatible = "nxp,s32cc-shm";
                    reg = <0x00 0x50000000 0x00 0x30000000>;
                    no-map;
            };

};

```

};

## ================================================================= platform\_gen.yaml

devices:

- end: 0x50800000  
start: 0x0
- end: 0x50900000  
start: 0x50810000
- end: 0x80000000  
start: 0x50a00000
- end: 0x835e0000  
start: 0x83200000
- end: 0x84080000  
start: 0x84000000
- end: 0x10000000000  
start: 0xc0000000  
memory:
- end: 0x83200000  
start: 0x80000000
- end: 0x84000000  
start: 0x835e0000
- end: 0xc0000000  
start: 0x84080000

===================================================================  
devices\_gen sending as below

[Next page](https://sel4.discourse.group/t/issue-with-sel4-armv8-config-for-mmu/940.md?page=2)