While calling
unsafe {
value = ptr::read_volatile(region.starting_address as *const u32)
}
I get
Caught cap fault in send phase at address 0
while trying to handle:
vm fault on data at address 0xa000000 with status 0x93880006
in thread 0x807fee9400 "rootserver" at address 0x24aaec
With stack:
0x2f9530: 0x0
0x2f9538: 0xa000000
0x2f9540: 0xa000000
...
What is the issue and how can I resolve it?
It seems like the address (region.starting_address) you are accessing is not mapped in the thread’s address space.
Are you able to provide more context for this code?
Hi @nspin sure. Many thanks in advance.
I assume to get the address space mapped as follows (code bellow).
init_free_page_addr is taken from the serial-device example.device_ut_ix contains the ID of the untyped, I before selected by the address range it covers. The address range encloses my desired address.Remarks: I feel not confident with the type sel4::cap_type::Granule::object_blueprint() but I was not able to get any constellation running (compiled yes but runtime failure) e.g., with sel4::FrameObjectType::from_bits(device_ut_desc.size_bits()).unwrap().blueprint(). Here the unwrap results in a runtime failure, because called Option::unwrap() on a None value.
let device_ut_cap = bootinfo.untyped().index(*device_ut_ix).cap();
// Take a CSpace slot
let device_frame_slot = empty_slots
.pop()
.unwrap()
.downcast::<sel4::cap_type::Granule>();
// Retype the untyped memory and put it into the CSpace slot
device_ut_cap
.untyped_retype(
&sel4::cap_type::Granule::object_blueprint(),
&sel4::init_thread::slot::CNODE.cap().relative_self(),
device_frame_slot.index(),
1,
)
.unwrap();
// Take the capability out of the slot
let device_frame_cap = device_frame_slot.cap();
// Map the physical address to a page table entry
let device_frame_addr = init_free_page_addr(bootinfo);
// Map the page table entry to the virtual address
device_frame_cap
.frame_map(
sel4::init_thread::slot::VSPACE.cap(),
device_frame_addr,
sel4::CapRights::read_write(),
sel4::VmAttributes::default(),
)
.unwrap();
I assume region.starting_address is the physical address of the region you are trying to access.
Your device_frame_cap.frame_map(...) invocation maps the page whose physical address corresponds to that of the untyped at device_ut_ix to the virtual address device_frame_addr. You can access it with:
value = ptr::read_volatile(device_frame_addr as *const u32)
Thanks worked perfect 
@nspin yet another question. My device-id information from the DTS file, fiven to qemu, is no passed to the bootinfo additional information. Maybe can you support me with that too? Many thanks in advance
One feature the Rust kernel loader (the sel4-kernel-loader crate) is still missing is being able to propagate the run-time device tree to the root task. So, the device tree that the root task get via the bootinfo is the one that is built by the seL4 CMake build system.
What information are you trying to pass to userspace via the bootinfo? Could you add it to the build-time device tree, by, for example, putting it in an overlay? See [1].
[1] seL4/src/plat/qemu-riscv-virt/overlay-qemu-riscv-virt.dts at master · seL4/seL4 · GitHub
@nspin thx. Got it. I am trying to test a virtio VFS implementation. I try to use the overlay.
Fixed it by adding --dtb parameter to sel4-kernel-loader-add-payload
@nspin yet another question.
I am now able to access VirtIO devices, but after the 5th device I receive the following failure while doing a volatile read:
vm fault on data at address 0x203800 with status 0x92000010
in thread 0x807ff49400 "rootserver" at address 0x253504
With stack:
0x2a29f0: 0x2a2960
@nspin yet another question.
While accessing the 5th VirtIO device in row I receive the following failure:
vm fault on data at address 0x203800 with status 0x92000010
in thread 0x807ff49400 "rootserver" at address 0x253504
With stack:
0x2a29f0: 0x2a2960
What could be the reason?
Can you provide some more context here, such as the code of your root task?
@nspin sure. Happy to do so. After the first VirtIO device was accessed, no further read/ write on other VirtIO devices is possible. Read/write on VirtIO devices means the code section print_device_info e.g., ptr::read_volatile((mapped_address + 0x000) as *const u32);
Root application:
#![no_std]
#![no_main]
extern crate alloc;
//use core::error;
use core::{any::{Any, TypeId}, f32::consts::E, fmt::Debug, ops::Index};
use sel4::{
cap_type,
init_thread::{self, Slot},
CapRights, CapTypeForFrameObjectOfFixedSize,
};
use core::ops::Range;
use sel4_sys::*;
use alloc::{borrow::ToOwned, string::ToString};
use alloc::vec::Vec;
use buddy_system_allocator::{FrameAllocator, LockedHeap};
use flat_device_tree::{node::FdtNode, standard_nodes::Compatible, Fdt};
use virtio_drivers::{
device::{
blk::VirtIOBlk,
console::VirtIOConsole,
gpu::VirtIOGpu,
net::VirtIONetRaw,
socket::{
VirtIOSocket, VsockAddr, VsockConnectionManager, VsockEventType, VMADDR_CID_HOST,
},
},
transport::{
mmio::{MmioTransport, VirtIOHeader},
pci::{
bus::{BarInfo, Cam, Command, DeviceFunction, MemoryBarType, PciRoot},
virtio_device_type, PciTransport,
},
DeviceType, Transport,
},
};
use core::{
mem::size_of,
alloc::Layout,
panic::PanicInfo,
ptr::{self, NonNull},
};
use virtio_drivers::{BufferDirection, Hal, PhysAddr, PAGE_SIZE};
use sel4_sync::{lock_api::Mutex, PanickingRawMutex};
use sel4::*;
use log::{debug, error, info, trace, warn, LevelFilter};
use sel4_immutable_cell::ImmutableCell;
#[global_allocator]
static HEAP_ALLOCATOR: LockedHeap<128> = LockedHeap::<128>::new();
use sel4::*;
use sel4_root_task::{debug_println, root_task, Never};
// RUST global allocator
const HEAP_SIZE : usize = 102400;
static mut HEAP: [u8; HEAP_SIZE] = [0; HEAP_SIZE];
fn traverse_device_tree(bootinfo: &sel4::BootInfoPtr, empty_slots: &mut Vec<sel4::init_thread::Slot>, node: FdtNode) {
// Never access memory as an I/O device
if node.name.contains(& "memory"){
return;
}
print_device_info(node, bootinfo, empty_slots);
for (position, element ) in node.children().enumerate(){
traverse_device_tree(bootinfo,empty_slots, element);
}
}
fn load_device_tree( bootinfo: &sel4::BootInfoPtr, empty_slots: &mut Vec<sel4::init_thread::Slot>){
if bootinfo.extra().count() > 0 {
for (position, element) in bootinfo.extra().enumerate(){
if element.id == BootInfoExtraId::Fdt {
let dt = Fdt::new(element.content()).unwrap();
for node in dt.all_nodes(){
traverse_device_tree(bootinfo,empty_slots,node);
}
}
}
}
}
fn map_u8_to_usize(bytes: &[u8]) -> Option<usize> {
if bytes.len() >= size_of::<usize>() {
let mut array = [0u8; size_of::<usize>()];
for (i, byte) in bytes.iter().take(size_of::<usize>()).enumerate() {
array[i] = *byte;
}
Some(usize::from_ne_bytes(array))
} else {
None
}
}
fn print_device_info(node: FdtNode,bootinfo: &sel4::BootInfoPtr, empty_slots: &mut Vec<sel4::init_thread::Slot>) -> usize {
// CNode stores capabilities and represents a logical component.
let cnode = sel4::init_thread::slot::CNODE.cap();
sel4::debug_println!("Name {} ", node.name);
// VirtIO in 2 ways
// 1. Via MMIO
if let (Some(compatible), Some(region)) = (node.compatible(), node.reg().next()) {
if compatible.all().any(|s| s.contains("virtio,mmio") )
&& region.size.unwrap_or(0) > size_of::<VirtIOHeader>()
{
sel4::debug_println!("Region size {}", region.size.unwrap_or(0) );
let (mapped_address, dev_ut_idx) = init_device(bootinfo,empty_slots, region.starting_address as usize, region.size.unwrap_or(0) );
if mapped_address == 0 {
if dev_ut_idx != 0{
sel4::debug_println!("Invalid device ");
//unmap_slot(dev_ut_idx);
}
return 0;
}
unsafe {
let magic_value = ptr::read_volatile((mapped_address + 0x000) as *const u32);
let version = ptr::read_volatile((mapped_address + 0x004) as *const u32);
let device_id = ptr::read_volatile((mapped_address + 0x008) as *const u32);
if magic_value != 0x74726976 {
sel4::debug_println!("Invalid Virtio device {} {} {}",magic_value, version, device_id);
//unmap_slot(dev_ut_idx);
return 0;
}
if device_id == 0 {
sel4::debug_println!("Invalid Virtio device {} {} {}",magic_value, version, device_id);
//unmap_slot(dev_ut_idx);
return 0;
}
let mut device_status = ptr::read_volatile((mapped_address + 0x070) as *const u32);
sel4::debug_println!("Device status {}",device_status);
ptr::write_volatile((mapped_address + 0x070) as *mut u32, device_status | 1 << 0);
ptr::write_volatile((mapped_address + 0x070) as *mut u32, device_status | 1 << 1);
let mut device_features = ptr::read_volatile((mapped_address + 0x010) as *const u32);
ptr::write_volatile((mapped_address + 0x020) as *mut u32, 0 | device_features);
ptr::write_volatile((mapped_address + 0x030) as *mut u32, 0);
ptr::write_volatile((mapped_address + 0x034) as *mut u32, 2);
ptr::write_volatile((mapped_address + 0x030) as *mut u32, 1);
ptr::write_volatile((mapped_address + 0x034) as *mut u32, 2);
ptr::write_volatile((mapped_address + 0x044) as *mut u32, 1);
ptr::write_volatile((mapped_address + 0x050) as *mut u32, 1);
ptr::write_volatile((mapped_address + 0x070) as *mut u32, device_status | 1 << 3);
device_status = ptr::read_volatile((mapped_address + 0x070) as *const u32);
sel4::debug_println!("Device status {}",device_status);
sel4::debug_println!("Init device of virtio type no. {} ", device_id);
}
}
}
// 2. Via PCI(e)
// 3. Special devices
return 0;
}
fn unmap_slot(dev_ut_idx: usize) {
let slot: Slot<cap_type::Granule> = init_thread::Slot::<cap_type::Granule>::from_index(dev_ut_idx);
let cap = slot.cap();
cap.frame_unmap();
}
fn device_by_address(bootinfo: &BootInfoPtr, address: usize, size : usize) -> Vec<(usize, &UntypedDesc,usize)> {
let mut memory_range: Vec<(usize, &UntypedDesc, usize)> = Vec::new();
let mut allocated_bits: usize = 0;
let mut allocated_address = address;
while allocated_bits < size{
// It is not ensured that the address spaces are sequentially given
// in the untyped list of the bootinfo
let _result = find_untyped_memory(bootinfo, allocated_address);
if _result.is_ok() {
let mut _tmp = _result.unwrap();
allocated_bits = allocated_bits + ( 1 << _tmp.1.size_bits() );
allocated_address = address + allocated_bits +1;
memory_range.push( _tmp );
}
else {
sel4::debug_println!("Failure ");
}
}
sel4::debug_println!("Required pages {}", memory_range.len() );
return memory_range;
}
fn find_untyped_memory(bootinfo: &BootInfoPtr, address: usize) -> Result<(usize, &UntypedDesc, usize)> {
for ( position, element) in bootinfo.untyped_list().iter().enumerate() {
let _from = element.paddr() as usize;
let _to = _from + ( 1 << element.size_bits() );
let _tmp_range = ( _from .. _to );
if ( _tmp_range.contains(&address)) {
let _offset = &address - _tmp_range.min().unwrap();
sel4::debug_println!("Memory index found, with free space of {}", _to -&address );
return Result::Ok( (position, element, _offset) );
}
}
return Result::Err(Error::AlignmentError);
}
#[root_task]
fn main(bootinfo: &sel4::BootInfoPtr) -> sel4::Result<Never> {
unsafe {
// Safe because `HEAP` is only used here and `entry` is only called once.
HEAP_ALLOCATOR.lock().init(HEAP.as_mut_ptr() as usize, HEAP.len());
}
sel4::debug_println!("Simplify");
/* # Manage untyped memory.
#
*/
let mut empty_capability_slots: Vec<sel4::init_thread::Slot> = Vec::new();
for (position, element) in bootinfo.empty().range().enumerate() {
empty_capability_slots.push(sel4::init_thread::Slot::from_index(element));
}
sel4::debug_println!("Amount of capability slots: {}",empty_capability_slots.len() );
/* # Init root CNode
#
*/
let cnode = sel4::init_thread::slot::CNODE;
let cnode_capability = cnode.cap();
/* # Object allocatpr
#
*/
load_device_tree(bootinfo,&mut empty_capability_slots);
sel4::debug_println!("Simply initialized ...");
loop {}
}
fn init_device(bootinfo: &BootInfoPtr, empty_slots: &mut Vec<sel4::init_thread::Slot>, address: usize, size : usize) -> (usize,usize) {
let _untyped_vector: Vec<(usize, &UntypedDesc, usize)> = device_by_address(bootinfo,address as usize, size);
if _untyped_vector.len() > 0{
let (device_ut_ix, device_ut_desc, device_addr_offset ) = _untyped_vector.index(0);
sel4::debug_println!("Index {} ", &device_ut_ix );
if device_ut_desc.is_device() {
unmap_slot(*device_ut_ix);
return map_addressspace_into_thread(bootinfo, device_ut_ix, device_ut_desc, device_addr_offset, address, size, empty_slots);
}
}
return (0,0);
}
fn map_addressspace_into_thread(bootinfo: &BootInfoPtr, device_ut_ix: &usize,device_ut_desc: &UntypedDesc, device_addr_offset:&usize, address: usize, size : usize, empty_slots: &mut Vec<Slot>) -> (usize, usize) {
sel4::debug_println!("Kernel mapping ");
let device_ut_cap = bootinfo.untyped().index(*device_ut_ix).cap();
sel4::debug_println!("Untyped size {}",device_addr_offset );
// Take a CSpace slot
let device_frame_slot = empty_slots
.pop()
.unwrap()
.downcast::<sel4::cap_type::SmallPage>();
// Retype the untyped memory and put it into the CSpace slot
let cap_status = device_ut_cap
.untyped_retype(
&sel4::cap_type::SmallPage::object_blueprint(),
&sel4::init_thread::slot::CNODE.cap().relative_self(),
device_frame_slot.index(),
1,
);
if cap_status.is_err() {
sel4::debug_println!("Faile 1 ");
return (0,*device_ut_ix);
}
cap_status.unwrap();
// Take the capability out of the slot
let device_frame_cap = device_frame_slot.cap();
// Map the physical address to a page table entry
let mut device_page_addr_ptr: usize = init_free_page_addr(bootinfo) as usize;
let cap_status = device_frame_cap
.frame_map(
sel4::init_thread::slot::VSPACE.cap(),
device_page_addr_ptr,
sel4::CapRights::read_write(),
sel4::VmAttributes::default(),
);
if cap_status.is_err() {
sel4::debug_println!("Faile 2 ");
return (0,*device_ut_ix);
}
cap_status.unwrap();
//return ( device_frame_cap.frame_get_address().unwrap() + device_addr_offset, *device_ut_ix);
return ( device_page_addr_ptr as usize + device_addr_offset, *device_ut_ix);
}
#[derive(Copy)]
#[derive(Clone)]
#[repr(C, align(4096))]
struct SmallPage(#[allow(dead_code)] [u8; SMALL_PAGE_PLACEHOLDER_SIZE]);
static SMALL_PAGE_PLACEHOLDERS: [SmallPage; SMALL_PAGE_NUMBER] = [SmallPage([0; SMALL_PAGE_PLACEHOLDER_SIZE]); SMALL_PAGE_NUMBER];
static mut OFFSET: usize = 0;
fn init_free_page_addr(bootinfo: &sel4::BootInfoPtr) -> usize {
let addr = unsafe { ptr::addr_of!( SMALL_PAGE_PLACEHOLDERS[OFFSET] ) as usize };
sel4::debug_println!("Bytes {} ", cap_type::Granule::FRAME_OBJECT_TYPE.bytes() );
sel4::debug_println!("ALLOC Page {} {} ", unsafe {OFFSET}, addr );
unsafe { OFFSET +=1 };
addr
}
extern "C" {
static __executable_start: usize;
static _end: usize;
}
fn user_image_bounds() -> Range<usize> {
unsafe { addr_of_ref(&__executable_start)..addr_of_ref(&_end) }
}
fn addr_of_ref<T>(x: &T) -> usize {
x as *const T as usize
}
const SMALL_PAGE_NUMBER: usize = 256;
const SMALL_PAGE_PLACEHOLDER_SIZE: usize = 4096;
QEMU startup:
qemu-system-aarch64 \
-machine virt,virtualization=on \
-cpu cortex-a57 \
-smp 2 \
-m 2048 \
-nographic \
-append "console=ttyAMA0" \
-drive file=./disk.img,if=none,format=raw,id=blk-device-0 \
-device virtio-blk-device,drive=blk-device-0 \
-netdev user,id=net0 \
-device virtio-net-device,netdev=net0 \
-device virtio-rng-device \
-device virtio-balloon-device \
-device virtio-gpu-device \
-device virtio-serial-device \
-serial mon:stdio \
-dtb ./devicetree.dtb \
-kernel $(image)
Example output:
qemu-system-aarch64 -machine virt,virtualization=on -cpu cortex-a57 -smp 2 -m 2048 -nographic -append "console=ttyAMA0" -drive file=./disk.img,if=none,format=raw,id=blk-device-0 -device virtio-blk-device,drive=blk-device-0 -netdev user,id=net0 -device virtio-net-device,netdev=net0 -device virtio-rng-device -device virtio-balloon-device -device virtio-gpu-device -device virtio-serial-device -serial mon:stdio -dtb ./devicetree.dtb -gdb tcp::3333 -S -kernel build/image.elf
seL4 kernel loader | INFO Starting loader
seL4 kernel loader | DEBUG Platform info: PlatformInfo {
memory: [
0x60000000..0x80000000,
],
devices: [
0x0..0x8000000,
0x8001000..0x8010000,
0x8011000..0x8030000,
0x8031000..0x60000000,
0x80000000..0x100000000000,
],
}
seL4 kernel loader | DEBUG Loader footprint: 0x6027f000..0x60477b17
seL4 kernel loader | DEBUG Payload info: PayloadInfo {
kernel_image: ImageInfo {
phys_addr_range: 0x60000000..0x6023f000,
phys_to_virt_offset: 0x8000000000,
virt_entry: 0x8060000000,
},
user_image: ImageInfo {
phys_addr_range: 0x7fe5e000..0x80000000,
phys_to_virt_offset: 0xffffffff803a2000,
virt_entry: 0x3345f8,
},
fdt_phys_addr_range: Some(
0x7fe5c000..0x7fe5dddc,
),
}
seL4 kernel loader | DEBUG Payload regions:
seL4 kernel loader | DEBUG 0x60000000..60031da0 true
seL4 kernel loader | DEBUG 0x60031da0..6023f000 false
seL4 kernel loader | DEBUG 0x7fe5e000..7ff877ac true
seL4 kernel loader | DEBUG 0x7ff887b0..7ffd4d74 true
seL4 kernel loader | DEBUG 0x7ffd5d78..7ffd5d90 true
seL4 kernel loader | DEBUG 0x7ffd5d90..7ffff208 false
seL4 kernel loader | DEBUG 0x7fe5c000..7fe5dddc true
seL4 kernel loader | DEBUG Copying payload data
seL4 kernel loader | INFO Entering kernel
Bootstrapping kernel
available phys memory regions: 1
[60000000..80000000]
reserved virt address space regions: 3
[8060000000..806023f000]
[807fe5c000..807fe5dddc]
[807fe5e000..8080000000]
Booting all finished, dropped to user space
Simplify
Amount of capability slots: 3592
Name /
Name psci
Name platform@c000000
Name fw-cfg@9020000
Name virtio_mmio@a000000
Region size 512
Memory index found, with free space of 33554432
Required pages 1
Index 21
Kernel mapping
Untyped size 0
Bytes 4096
ALLOC Page 0 2109440
Invalid Virtio device 1953655158 1 0
Name virtio_mmio@a000200
Region size 512
Memory index found, with free space of 33553920
Required pages 1
Index 21
Kernel mapping
Untyped size 512
Bytes 4096
ALLOC Page 1 2113536
Invalid Virtio device 1953655158 1 0
Name virtio_mmio@a000400
Region size 512
Memory index found, with free space of 33553408
Required pages 1
Index 21
Kernel mapping
Untyped size 1024
Bytes 4096
ALLOC Page 2 2117632
Invalid Virtio device 1953655158 1 0
Name virtio_mmio@a000600
Region size 512
Memory index found, with free space of 33552896
Required pages 1
Index 21
Kernel mapping
Untyped size 1536
Bytes 4096
ALLOC Page 3 2121728
Device status 0
Device status 8
Init device of virtio type no. 16
Name virtio_mmio@a000800
Region size 512
Memory index found, with free space of 33552384
Required pages 1
Index 21
Kernel mapping
Untyped size 2048
Bytes 4096
ALLOC Page 4 2125824
Caught cap fault in send phase at address 0
while trying to handle:
vm fault on data at address 0x207800 with status 0x92000010
in thread 0x807fe49400 "rootserver" at address 0x35048c
With stack:
0x39ec20: 0x39ebd0
0x39ec28: 0x207800
0x39ec30: 0x207800
0x39ec38: 0x32b4d0
0x39ec40: 0x3315c0
0x39ec48: 0x33152c
0x39ec50: 0x0
0x39ec58: 0x202480
0x39ec60: 0x202486
0x39ec68: 0x202480
0x39ec70: 0x202486
0x39ec78: 0x202480
0x39ec80: 0x3749d0
0x39ec88: 0x6
0x39ec90: 0x202480
0x39ec98: 0x100000006
This code’s intent is to, for each VirtIO MMIO region, (1) locate the corresponding untyped memory, (2) retype it into frames, (3) map those frames, and then (4) access them. I can spot at least one issue in (1):
In AArch64, pages are 4096 bytes. VirtIO MMIO regions are 512 bytes, they are packed in pages together, 8 at a time. However, your code attempts to map one page for each 512 byte MMIO region, which means that it retypes (i.e. allocates) 4096 bytes of untyped memory for each 512 byte region. It’s hard to tell exactly how this bug interacts with the other issues in this code, but what seems to be happening is that, by the time it reaches the 5th VirtIO MMIO region, the physical addresses corresponding to the untyped memory backing the frames you are mapping are not mapped in the platform’s physical address space.
@nspin See my fault. Many thanks.