I thought that would be desired? I think MCS is the only thing that is difficult to represent as a const that dependencies need to adapt to, though I’d consider MCS to be a temporary transition. Are there other cases?
I think that it would be desirable to have the ability to access kernel configuration information at compile-time outside of the sel4 crate. It may not be necessary often, but grepping for CONFG_ in seL4_libs yields results from the kernel configuration beyond just CONFIG_KERNEL_MCS.
One basic case where this would be useful is for conditional debugging and benchmarking code in higher-level userspace crates.
1 Like
Indeed. I suppose that would help with supporting the seL4 way of building user space code. Thanks.
The IceCap crates are designed to stand on their own, outside of the IceCap build system. However, I recognize that this is not obvious. In order to frame the IceCap crates in a more familiar context, I’ve put together a little demo using the IceCap crates in a project which is built by seL4_tools/cmake_tool:
I hope that this makes the prototype I posted above more accessible and useful.
One approach to enabling a simple “getting started” workflow for newcomers without a bunch of build magic under the hood could be to add optional support for simple dynamic loading of the root task to the elfloader. For example, a development-and-debugging-oriented configuration of the elfloader could find the root task by semihosting or in the device tree (at the initrd-* nodes) at runtime instead of at link time. A feature like this would be very easy to add.
A “getting started” workflow could look like:
$ prebuilt=./sel4-12.1.0-dev-prebuilt-with-config-5576d1fc
$ SEL4_CONFIG_ABI_ETC=$prebuilt/config-abi-etc.whatever cargo build
$ qemu-system-aarch64 ... -kernel $prebuilt/elfloader.elf -initrd target/aarch64-icecap/debug/my-root-task.elf
Even if we don’t distribute prebuilt binaries, having a configuration where elfloader.elf does not depend on my-root-task.elf allows one to opt for a much more simple build system where the builds of elfloader.elf and my-root-task.elf aren’t intertwined.
That sounds like it could be quite nice, yes, not only for Rust builds but also for what the Core Platform is trying to do. I wasn’t in the discussions back then about the elfloader design, it’s entirely possible that the current state is just historically grown and not purposefully designed that way. It’d be good to know if there are known issues, though. @kent-mcleod2 might remember something there.
Haven’t had a chance yet to play with the demo repo you posted, it looks like the setup as such is working and I hope to be able to get to it in the next few days (moving house next week, so things are a bit chaotic).
I can’t think of any reasons why this couldn’t be added. Originally there was no device tree pass-through and the elfloader grew out of a necessity to try and minimize the size of the kernel boot code. I think the elfloader still doesn’t include any code to parse a device tree blob at runtime. I’d welcome a feature like @nspin suggests.
Didn’t I read somewhere that there was interest in having the kernel do what the elfloader currently does, like on x86_64?
That’d surprise me, the kernel should be doing less, not more.
I suppose the only minor hesitation I have is that currently we can expect the build system to produce an image which contains a root task and kernel built for the same kernel configuration, and once you can pass the root task separately from the kernel, we could imagine a scenario in which these expectations are no longer in sync. If there are any mechanisms in place that a would catch a mismatched kernel/root process early during startup, i’m not aware of them but I haven’t actually looked at it very much.
I’ve put together a PoC: Colias Group / IceCap (Colias Group branches) / rust-meta-demo · GitLab
This PoC includes:
- A branch of
seL4_toolsin which I’ve added support for some dynamism to the elfloader. I will eventually open a PR detailing the various design points I encountered and how I landed on my implementation. - Source for building seL4 artifacts (
libsel4,kernel.elf,kernel.dtb,elfloader, etc.) independently of a root task. - The corresponding output, which serves as an example collection of prebuilt artifacts.
- An example which compiles and simulates a root task (written in Rust) against those prebuilt artifacts.
One note worth including inline here is that a simple -initrd flag to QEMU will not suffice, unless we disguise the elfloader as Linux. QEMU only observes the -initrd flag in the case of a “direct Linux boot”, and such a case requires -kernel to be in the typical arm64 Linux image format. The alternative I’ve included in this PoC is to use U-Boot as -kernel, with a script (u-boot-script.uimg, source) which loads the elfloader and the CPIO payload, adds information about the location of the payload to the device tree, and then passes control to the elfloader.
Good point. One approach to catching mismatches at runtime is by embedding the expected hash of the kernel into the elfloader, and requiring that root tasks include the hash of the kernel they are compiled for in something like an ELFNOTE. The elfloader could verify that everything lines up while it is loading the various images.
2 Likes
I’ve noted that build systems for Rust seL4 userspaces would be simpler if the elfloader didn’t have to be rebuilt to embed the root task image. I’ve proposed adding support for the elfloader to be passed the root task image at runtime as a potential solution. Now, I’ve come up with an idea for what I believe might be a stronger solution.
My idea is to first build the elfloader without a CPIO payload, as before. Then, at a different phase of the build, once the CPIO payload, including the root task, is assembled, the empty elfloader image is rendered into a new ELF file along with the CPIO payload. A few variables in the elfloader part of this new aggregate image are then modified to point to the payload.
One advantage of this approach over dynamism in the elfloader is that arranging the elfloader’s address space is now the responsibility of the build system, rather than that of the previous stage bootloader. We have higher-level tools at our disposal at build time, and we only have to implement the relevant logic once rather than in a bootloader-specific way for each platform. As an added bonus, this allows U-Boot to be removed from the QEMU simulation setup.
Another advantage of this approach is that it could also apply to the CapDL loader. It isn’t as immediate to apply, however, because some of the data that would need to be injected to the image, namely the CapDL spec, is C compilation output with symbols floating about which must be concretized. I can think of a few potentially low-resistance ways to handle this, e.g. by linking the CapDL spec as a relocatable image and then performing the relocation as part of this rendering phase, but this would take some additional consideration.
In keeping with tradition, I’ve prepared a PoC demonstrating this solution. Note that, in my previous post, I failed to link to sufficiently specific branches in a few cases and I’ve missed my chance to make an edit. This time I’ve linked to branches named link-target/defer-linking-elfloader-payload. If any links in my previous post end up seeming like they’re targeting an overly general branch of a given repository, just switch to the branch named link-target/dynamic-elfloader-payload.
This PoC, much like my previous one, includes:
- A branch of
seL4_toolsin which I’ve added support for this “deferred linking” to the elfloader. - A program called
render-elf-with-datawhich performs this “deferred linking” in a general way. Example invocation here. -
Source for building seL4 artifacts (
libsel4,kernel.elf,kernel.dtb,elfloader, etc.) independently of a root task. - The corresponding output, which serves as an example collection of prebuilt artifacts.
- An example which compiles and simulates a root task (written in Rust) against those prebuilt artifacts, including using the tool mentioned above for deferred linking.
A few notes about the render-elf-with-data tool: Its job is quite simple, and it contains no seL4-specific logic. All it does is “load” the input ELF file into a new ELF file by copying PT_LOAD segments over unmodified, and then add new PT_LOAD segments containing the input data at the next available appropriately aligned virtual addresses. Then, it records some information about the layout of this new data for the program by modifying the values of a few variables (i.e. making a few patches to the new image). The new ELF file has no sections, but all of the information in the original ELF file is still valid, and thus useful for debugging, etc.
2 Likes
I agree that removing the need to rebuild the elfloader from source each time would simplify development.
Just briefly documenting an additional approach before I forget about it: merge all of the object files into an intermediate object file using -Wl,--relocatable. Then linking in the CPIO archive would only require linking the single elfloader.o with a cpio.o file + a platform specific linker script.
CMake snippet: support creating relocatable object target in add_library (#16977) · Issues · CMake / CMake · GitLab
At the seL4 Summit in October, I presented some thoughts on Rust support in seL4 userspace. A few weeks ago, I started implementing some of the ideas that have been floating around in this area; from this thread, that talk, and discussions at the summit. My progress can be found in this repository:
So far, this repository includes:
This repository also contains some code for building and testing these crates. However, the crates are in no way bound to this build system code.
sel4-sys
This sel4-sys crate is mostly generated code. Rust equivalents of the types and consts which are defined by hand in the C libsel4 source are generated from the C libsel4 headers using bindgen. Types and functions which are generated for the C libsel4 are generated in the same way for this sel4-sys crate. That is, at build time, Rust code parses .pbf files and .xml files and generates bitfield structs and object invocation functions. In this way, to the greatest extent possible, items in sel4-sys are generated directly from where they are originally defined, whether that be C headers shared between libsel4 and the kernel, or .pbf and .xml files. Furthermore, the result is a pure Rust crate, which does not require linking against the C libsel4.
The build-time inputs for sel4-sys are a JSON representation of the kernel configuration along with the .h, .pbf, and .xml files comprising the C libsel4. The locations of these inputs are passed to the relevant crates by environment variables. They can be pointed to in a fine-grained manner (for when invoking Cargo as part of a typical cmake-tool build), or all together in the single environment variable SEL4_PREFIX (for cases where the seL4 kernel is built separately).
For now, sel4-sys depends on a few patches to libsel4 which can be found at coliasgroup/seL4:rust. Most of these patches simply move a few files around to make the more readily available. The most significant patch adds CMake logic to generate a JSON representation of the kernel configuration alongside the C header.
sel4-sys aims to be a minimal expression of the seL4 API, covering the functionality of libsel4, but nothing more. Its API is low-level, and mostly generated. The idea here is that it is the responsibility of a slightly higher-level crate to wrap all of this in a nicer, more idiomatic Rust API. I’m working on one such crate now, but sel4-sys should also be available to be used directly (e.g. for cases where one wishes to use a different higher-level wrapper around it). This intent follows the Rust convention for *-sys crates.
Similar to how Sparrow tests its sel4-sys crate, this sel4-sys crate has a feature called wrappers which, when enabled, generates extern "C" wrappers which cover the entire API of the C libsel4. I have a hacked branch of the seL4 kernel which extends libsel4 with a configuration which replaces C definitions of functions with declarations which link against the Rust definitions. This allows us to run sel4test against sel4-sys instead of the C libsel4.
If you’re interested, have a look at the code or rendered rustdoc. I’d love to hear your thoughts.
Loader
One thread of discussion here has been about how elfloader is tightly integrated with cmake-tool. I’ve proposed extending elfloader with configurations to provide it with images at run-time or in a second build phase, and Kent just mentioned another approach which splits the build of the elfloader into a compile phase which can happen without the images and a link phase which can happen later, even as part of a different build system, with the images.
Another approach is to just provide a simple loader in Rust. This approach maximizes build system convenience in the cases we are focused on here, but minimizes code reuse with elfloader. I wrote such a loader to explore this idea. It’s still a work in progress, but so far it works on aarch64, specifically qemu -m virt and the Raspberry Pi 4. Such a loader wouldn’t have the feature-richness, configurability, breadth of platform support, or test coverage of elfloader, but it would make getting started developing Rust userspace much easier for some limited set of configurations. This example shows how simple the build can be.
1 Like