Awkernel

Internal

Synchronization

The awkernel_lib/src/sync.rs module provides synchronization primitives. Mutex:awkernel_lib/src/sync/mutex.rs is used for mutual exclusion. RwLock:awkernel_lib/src/sync/rwlock.rs is used for read-write locks. Note that these locks are spin-based, and interrupts are disabled during the critical section.

Mutex

Awkernel adopts MCS lock [1] for mutual exclusion by default because the safety and liveliness of MCS lock have been formally verified [2]. Therefore, MCSNode have to be used when using Mutex as follows.

#![allow(unused)]
fn main() {
use awkernel_lib::sync::mutex::{MCSNode, Mutex};
use alloc::sync::Arc;

let data = Arc::new(Mutex::new(0));

// acquire the lock
let mut node = MCSNode::new();
let guard = data.lock(&mut node);

*guard = 10;
}

RwLock

RwLock can be used as normal read-write lock as follows.

#![allow(unused)]
fn main() {
use awkernel_lib::sync::rwlock::RwLock;
use alloc::sync::Arc;

let data = Arc::new(RwLock::new(0));

{
    // write lock
    let guard = data.write();
    *guard = 10;
}

{
    // read lock
    let guard = data.read();
    assert_eq!(*guard, 10);
}
}

References

1. J. M. Mellor-Crummey and M. L. Scott. Algorithms for scalable synchronization on shared- memory multiprocessors. ACM Trans. Comput. Syst., 9(1), Feb. 1991.
2. J. Kim, V. Sjöberg, R. Gu, Z. Shao. Safety and Liveness of MCS Lock - Layer by Layer. APLAS 2017: 273-297

Console

Console is a trait for the console and defined in awkernel_lib/src/console.rs as follows.

#![allow(unused)]
fn main() {
pub trait Console: Write + Send {
    /// Enable the serial port.
    fn enable(&mut self);

    /// Disable the serial port.
    fn disable(&mut self);

    /// Enable the reception interrupt.
    fn enable_recv_interrupt(&mut self);

    /// Disable the reception interrupt.
    fn disable_recv_interrupt(&mut self);

    /// Acknowledge to the reception interrupt.
    fn acknowledge_recv_interrupt(&mut self);

    /// Get IRQ#.
    fn irq_id(&self) -> u16;

    /// Read a byte.
    fn get(&mut self) -> Option<u8>;

    /// Write a byte.
    fn put(&mut self, data: u8);
}
}

There are several functions regarding the Console trait in awkernel_lib/src/console.rs.

When booting, an unsafe console should be registered by calling the register_unsafe_puts function. After that, the unsafe_puts and unsafe_print_hex functions can be used to print messages. Note that these functions are unsafe because they may cause data races.

After enabling mutual exclusion, a safe console should be registered by calling the register_console function. Then, the print, get, and put functions can be used to print messages.

Implementation

x86_64

x86_64 should equip UART 16550 serial ports, and Awkerenel uses the serial port to output messages as a console. UART 16550's device driver is implemented in awkernel_drivers/src/uart/uart_16550 whose original source code is from uart_16550. The Uart structure defined in kernel/src/arch/x86_64/console.rs implements the Console trait as follows.

#![allow(unused)]
fn main() {
pub struct Uart {
    port: uart_16550::SerialPort,
    enabled: bool,
}

impl Console for Uart {
    fn enable(&mut self) {
        self.enabled = true;
    }

    fn disable(&mut self) {
        self.enabled = false;
    }

    fn enable_recv_interrupt(&mut self) {
        self.port.enable_interrupt();
    }

    fn disable_recv_interrupt(&mut self) {
        self.port.disable_interrupt();
    }

    fn acknowledge_recv_interrupt(&mut self) {
        // nothing to do
    }

    fn irq_id(&self) -> u16 {
        36 // COM1
    }

    fn get(&mut self) -> Option<u8> {
        if self.enabled {
            self.port.try_receive()
        } else {
            None
        }
    }

    fn put(&mut self, data: u8) {
        if self.enabled {
            self.port.send(data);
        }
    }
}
}

AArch64

AArch64 should equip PL011 UART serial ports, and Awkerenel uses the serial port to output messages as a console. PL011 UART's device driver is implemented in awkernel_drivers/src/uart/pl011.rs, and it implements the Console trait as follows.

#![allow(unused)]
fn main() {
impl Console for PL011 {
    fn enable(&mut self) {
        use registers::CR;
        registers::UART0_CR.write(CR::EN | CR::RXE | CR::TXE, self.base_addr); // enable, Rx, Tx
    }

    fn disable(&mut self) {
        registers::UART0_CR.write(registers::CR::empty(), self.base_addr);
    }

    fn enable_recv_interrupt(&mut self) {
        registers::UART0_IMSC.setbits(IMSC_RXIM, self.base_addr);
    }

    fn disable_recv_interrupt(&mut self) {
        registers::UART0_IMSC.clrbits(IMSC_RXIM, self.base_addr);
    }

    fn acknowledge_recv_interrupt(&mut self) {
        registers::UART0_ICR.write(registers::ICR::RXIC, self.base_addr);
    }

    fn irq_id(&self) -> u16 {
        self.irq
    }

    fn get(&mut self) -> Option<u8> {
        if registers::UART0_FR.read(self.base_addr) & 0x10 != 0 {
            None
        } else {
            Some(registers::UART0_DR.read(self.base_addr) as u8)
        }
    }

    fn put(&mut self, data: u8) {
        // wait until we can send
        unsafe { asm!("nop;") };
        while registers::UART0_FR.read(self.base_addr) & 0x20 != 0 {
            core::hint::spin_loop();
        }

        // write the character to the buffer
        registers::UART0_DR.write(data as u32, self.base_addr);
    }
}
}

Logger

Awkernel uses log crate for logging. So, you can use the log macros like defined in this crate as follows. The logger uses the console module internally.

#![allow(unused)]
fn main() {
log::error!("This is an error message.");
log::warn!("This is a warning message.");
log::info!("Hello, world!");
log::debug!("This is a debug message.");
log::trace!("This is a trace message.");
}

log::debug! is useful when implementing and debugging the kernel because it displays a message with the file name and line number where the macro is called.

You can use log::set_max_level function to set the maximum log level as follows.

#![allow(unused)]
fn main() {
log::set_max_level(log::LevelFilter::Trace);
}

Buffered Logger

After booting Awkernel, the logger implementation defined in awkernel_lib/src/logger.rs is switched to a buffered logger defined in applications/awkernel_services/src/buffered_logger.rs. The buffered logger buffers log messages and writes them to the UART in a batch. Note that the buffered logger will discard messages if the buffer is full to avoid memory exhaustion. The buffered logger is executed as an async/await task and spawned in applications/awkernel_services/src/main.rs.

Boot

In this section, we explain how Awkernel boots.

x86_64

Primary Core

The primary core calls kernel_main of x86_64 first, which is called by UEFI.

1. kernel_main:kernel/src/arch/x86_64/kernel_main.rs
2. kernel_main2:kernel/src/arch/x86_64/kernel_main.rs
3. main:kernel/src/main.rs

graph TD;
    kernel_main:kernel_main.rs-->kernel_main2:kernel_main.rs;
    kernel_main2:kernel_main.rs-->main:main.rs;

During the primary core is booting, it wakes up non-primary cores by sending ACPI's IPIs.

Non-primary Cores

Non-primary cores calls _start_cpu defined in mpboot.S first, and it then calls non_primary_kernel_main. It eventually calls main like the primary core.

1. _start_cpu:kernel/asm/x86/mpboot.S
2. non_primary_kernel_main:kernel/src/arch/x86_64/kernel_main.rs
3. main:kernel/src/main.rs

graph TD;
    _start_cpu:mpboot.S-->non_primary_kernel_main:kernel_main.rs;
    non_primary_kernel_main:kernel_main.rs-->main:main.rs;

AArch64

Primary and Non-primary Cores

_start defined in boot.S is the entry point for both the primary and non-primary cores. _start eventually calls kernel_main in kernel_main.rs. After that, the primary core calls primary_cpu and non-primary cores call non_primary_cpu. Eventually, main is called.

1. _start:kernel/asm/aarch64/boot.S
2. kernel_main:kernel/src/arch/aarch64/kernel_main.rs
3. The primary core calls primary_cpu and non-primary cores call non_primary_cpu.
   1. primary_cpu:kernel/src/arch/aarch64/kernel_main.rs
   2. non_primary_cpu:kernel/src/arch/aarch64/kernel_main.rs
4. main:kernel/src/main.rs

graph TD;
    _start:boot.S-->kernel_main:kernel_main.rs;
    kernel_main:kernel_main.rs-->primary_cpu:kernel_main.rs;
    kernel_main:kernel_main.rs-->non_primary_cpu:kernel_main.rs;
    primary_cpu:kernel_main.rs-->main:main.rs;
    non_primary_cpu:kernel_main.rs-->main:main.rs;

Main Function

After booting, in the main function, the primary core wakes async/await tasks, and non-primary cores execute async/await tasks.

Primary Core

In main function, the primary core periodically calls wake_task and poll functions defined in awkernel_async_lib and awkernel_lib,

1. main:kernel/src/main.rs
2. wake_task:awkernel_async_lib/src/scheduler.rs
3. poll:awkernel_lib/src/net.rs

graph TD;
    main:kernel/src/main.rs-->wake_task:awkernel_async_lib/src/scheduler.rs;
    main:kernel/src/main.rs-->poll:awkernel_lib/src/net.rs;

wake_task is a function to wake sleeping async/await tasks up, and it will be explained in Sec. Scheduler. poll is a function to poll network interface controllers. If some events arrives, poll wakes async/await tasks related to the controllers.

Non-primary Cores

In main function, non-primary core periodically call run defined in awkernel_async_lib. run initializes variables regarding preemption by calling preempt::init. After that, it calls run_main to execute async/await tasks.

1. main:kernel/src/main.rs
2. run:awkernel_async_lib/src/task.rs
3. init:awkernel_async_lib/src/task/preempt.rs
4. run_main:awkernel_async_lib/src/task.rs

graph TD;
    main:kernel/src/main.rs-->run:awkernel_async_lib/src/task.rs;
    run:awkernel_async_lib/src/task.rs-->init:awkernel_async_lib/src/task/preempt.rs;
    run:awkernel_async_lib/src/task.rs-->run_main:awkernel_async_lib/src/task.rs;

run_main executes async/await tasks, and it will be explained in Sec. Scheduler.

Architecture Abstraction

Awkernel abstracts the architecture-specific details of the underlying hardware using the Arch trait, which is defined in the awkernel_lib/src/arch.rs. The Arch trait requires the Delay, Interrupt, CPU, and Mapper traits as follows.

#![allow(unused)]
fn main() {
#[allow(dead_code)]
#[cfg(not(feature = "std"))]
trait Arch:
    super::delay::Delay + super::interrupt::Interrupt + super::cpu::CPU + super::paging::Mapper
{
}

#[allow(dead_code)]
#[cfg(feature = "std")]
trait Arch: super::delay::Delay + super::interrupt::Interrupt + super::cpu::CPU {}
}

graph LR;
    Arch:awkernel_lib/src/arch.rs-->Delay:awkernel_lib/src/delay.rs;
    Arch:awkernel_lib/src/arch.rs-->Interrupt:awkernel_lib/src/interrupt.rs;
    Arch:awkernel_lib/src/arch.rs-->CPU:awkernel_lib/src/cpu.rs;
    Arch:awkernel_lib/src/arch.rs-->Mapper:awkernel_lib/src/paging.rs;

The Delay trait provides a way to wait for a certain amount of time and to get the time. The Interrupt trait provides a way to enable and disable interrupts. The CPU trait provides a way to get the current CPU ID. The Mapper trait provides a way to map and unmap virtual memory regions.

For the std environment, the Arch trait does not require the Mapper trait because user space applications do not need to manage memory mappings.

Implementation

x86_64

For x86_64, the X86:awkernel_lib/src/arch/x86_64.rs structure implements the Arch trait. In addition, it implements the Delay, Interrupt, CPU, and Mapper traits as follows.

• Delay: awkernel_lib/src/arch/x86_64/delay.rs
• Interrupt: awkernel_lib/src/arch/x86_64/interrupt.rs
• CPU: awkernel_lib/src/arch/x86_64/cpu.rs
• Mapper: awkernel_lib/src/arch/x86_64/paging.rs

AArch64

For AArch64, the AArch64:awkernel_lib/src/arch/aarch64.rs structure implements the Arch trait. In addition, it implements the Delay, Interrupt, CPU, and Mapper traits as follows.

• Delay: awkernel_lib/src/arch/aarch64/delay.rs
• Interrupt: awkernel_lib/src/arch/aarch64/interrupt.rs
• CPU: awkernel_lib/src/arch/aarch64/cpu.rs
• Mapper: awkernel_lib/src/arch/aarch64/paging.rs

Delay

The Delay trait provides a way to wait for a certain amount of time and to get the time. It is defined in awkernel_lib/src/delay.rs as follows.

#![allow(unused)]
fn main() {
pub trait Delay {
    /// Wait interrupt.
    fn wait_interrupt();

    /// Wait microseconds.
    fn wait_microsec(usec: u64);

    /// Never return.
    fn wait_forever() -> ! {
        loop {
            Self::wait_interrupt();
        }
    }

    /// Wait milliseconds.
    fn wait_millisec(msec: u64) {
        assert!(msec < u64::MAX / 1000);
        Self::wait_microsec(msec * 1000);
    }

    /// Wait seconds.
    fn wait_sec(sec: u64) {
        assert!(sec < u64::MAX / 1_000_000);
        Self::wait_microsec(sec * 1000 * 1000);
    }

    /// This function returns uptime in microseconds.
    fn uptime() -> u64;

    /// Return CPU cycle counter.
    fn cpu_counter() -> u64;

    /// Pause a CPU during busy loop to reduce CPU power consumption.
    fn pause() {
        core::hint::spin_loop();
    }
}
}

There are several functions regarding the Delay trait in awkernel_lib/src/delay.rs.

Implementation

x86_64

For x86_64, the the X86 structure implements the Delay trait in awkernel_lib/src/arch/x86_64/delay.rs as follows.

#![allow(unused)]
fn main() {
impl Delay for super::X86 {
    fn wait_interrupt() {
        unsafe { core::arch::asm!("hlt") };
    }

    fn wait_microsec(usec: u64) {
        let start = uptime();
        loop {
            let diff = uptime() - start;
            if diff >= usec {
                break;
            }

            core::hint::spin_loop();
        }
    }

    fn uptime() -> u64 {
        let base = HPET_BASE.load(Ordering::Relaxed);
        let hz = HPET_COUNTER_HZ.load(Ordering::Relaxed);
        let start = HPET_COUNTER_START.load(Ordering::Relaxed);

        if hz == 0 {
            0
        } else {
            let now = HPET_MAIN_COUNTER.read(base);
            let diff = now - start;

            diff * 1_000_000 / hz
        }
    }

    fn cpu_counter() -> u64 {
        unsafe { core::arch::x86_64::_rdtsc() }
    }
}
}

Awkernel currently uses High Precision Event Timer (HPET) to get uptime in microseconds. So, if you want to use Awkernel on KVM, you need to enable HPET in the virtual machine settings. To get the cpu cycle counter, Awkernel uses the _rdtsc instruction.

AArch64

For x86_64, the AArch64 structure implements the Delay trait in awkernel_lib/src/arch/aarch64/delay.rs as follows.

#![allow(unused)]
fn main() {
impl Delay for super::AArch64 {
    fn wait_interrupt() {
        unsafe { core::arch::asm!("wfi") };
    }

    fn wait_microsec(usec: u64) {
        let frq = awkernel_aarch64::cntfrq_el0::get();
        let t = awkernel_aarch64::cntvct_el0::get();

        let end = t + ((frq / 1000) * usec) / 1000;

        while awkernel_aarch64::cntvct_el0::get() < end {
            awkernel_aarch64::isb();
        }
    }

    fn uptime() -> u64 {
        let start = unsafe { read_volatile(addr_of!(COUNT_START)) };

        let frq = awkernel_aarch64::cntfrq_el0::get();
        let now = awkernel_aarch64::cntvct_el0::get();

        let diff = now - start;

        diff * 1_000_000 / frq
    }

    fn cpu_counter() -> u64 {
        awkernel_aarch64::pmccntr_el0::get()
    }
}
}

To get uptime in microseconds, Awkernel uses the counter-timer frequency register (CNTFRQ_EL0) and the counter-timer virtual count register (CNTVCT_EL0) of AArch64. To get the cpu cycle counter, Awkernel uses the performance monitors cycle count register (PMCCNTR_EL0).

CPU

The CPU trait provides a way to get the current CPU ID. It is defined in awkernel_lib/src/cpu.rs as follows.

#![allow(unused)]
fn main() {
pub trait CPU {
    /// CPU ID returns the ID of the CPU.
    /// The ID is unique for each CPU and starts from 0 to `num_cpu() - 1`.
    fn cpu_id() -> usize;

    /// Raw CPU ID returns the ID of the CPU without any modification.
    fn raw_cpu_id() -> usize;
}
}

The cpu_id method returns the current CPU ID. The ID is unique for each CPU and ranges from 0 to num_cpu() - 1. The num_cpu function, which returns the number of CPUs, is also defined in awkernel_lib/src/cpu.rs.

The raw_cpu_id method returns the ID of the CPU without any modification. The ID is unique for each CPU, but it may not be in the range of 0 to num_cpu() - 1.

There are functions regarding CPU in awkernel_lib/src/cpu.rs as follows.

Implementation

x86_64

For x86_64, the X86 structure implements the CPU trait in awkernel_lib/src/arch/x86_64/cpu.rs as follows.

#![allow(unused)]
fn main() {
impl CPU for super::X86 {
    fn cpu_id() -> usize {
        let cpuid_leaf_1 = unsafe { core::arch::x86_64::__cpuid(1) };

        // Check if x2APIC is supported
        if (cpuid_leaf_1.ecx & (1 << 21)) != 0 {
            // Get x2APIC ID from leaf 1FH or leaf 0BH (1FH is preferred)
            let max_leaf = unsafe { core::arch::x86_64::__cpuid(0) }.eax;
            let edx = if max_leaf >= 0x1F {
                unsafe { core::arch::x86_64::__cpuid(0x1F).edx }
            } else {
                unsafe { core::arch::x86_64::__cpuid(0x0B).edx }
            };
            edx as usize
        } else {
            (cpuid_leaf_1.ebx >> 24 & 0xff) as usize
        }
    }

    fn raw_cpu_id() -> usize {
        Self::cpu_id()
    }
}
}

Awkernel uses core::arch::x86_64::__cpuid to get the CPU ID. It assumes that the CPU ID is unique for each CPU and ranges from 0 to num_cpu() - 1. If you find any hardware that does not follow this assumption, please let us know or send us a PR.

AArch64

For x86_64, the AArch64 structure implements the CPU trait in awkernel_lib/src/arch/aarch64/cpu.rs as follows.

#![allow(unused)]
fn main() {
impl CPU for super::AArch64 {
    fn cpu_id() -> usize {
        let mpidr = mpidr_el1::get();

        let aff0 = mpidr & 0xff;
        let aff1 = (mpidr >> 8) & 0xff;
        let aff2 = (mpidr >> 16) & 0xff;
        let aff3 = (mpidr >> 32) & 0xff;

        let result =
            unsafe { aff0 + AFF0_MAX * aff1 + AFF0_X_AFF1 * aff2 + AFF0_X_AFF1_X_AFF2 * aff3 };

        result as usize
    }

    fn raw_cpu_id() -> usize {
        mpidr_el1::get() as usize
    }
}
}

For AArch64, Awkernel calculates the CPU ID based on the MPIDR_EL1 register and the affinity information variables.

Interrupt

The Interrupt trait provides a way to enable and disable interrupts. It is defined in awkernel_lib/src/interrupt.rs as follows.

#![allow(unused)]
fn main() {
pub trait Interrupt {
    fn get_flag() -> usize;
    fn disable();
    fn enable();
    fn set_flag(flag: usize);
}
}

The get_flag and set_flag methods get and set the interrupt flag. These methods are used to save and restore the interrupt flag when enabling and disabling interrupts and used in the InterruptGuard structure.

Interrupt Guard

The InterruptGuard structure defined in awkernel_lib/src/interrupt.rs is used to disable interrupts in a scope. After the scope, interrupts are enabled automatically as follows.

#![allow(unused)]
fn main() {
{
    use awkernel_lib::interrupt::InterruptGuard;
    let _int_guard = InterruptGuard::new();
    // interrupts are disabled.
}
}

There are enable and disable functions in awkernel_lib/src/interrupt.rs. You can use these functions rather than using the InterruptGuard.

Implementation

x86_64

For x86_64, the X86 structure implements the Interrupt trait in awkernel_lib/src/arch/x86_64/interrupt.rs as follows.

#![allow(unused)]
fn main() {
impl Interrupt for super::X86 {
    fn get_flag() -> usize {
        if x86_64::instructions::interrupts::are_enabled() {
            1
        } else {
            0
        }
    }

    fn disable() {
        x86_64::instructions::interrupts::disable();
    }

    fn enable() {
        x86_64::instructions::interrupts::enable();
    }

    fn set_flag(flag: usize) {
        if flag == 0 {
            x86_64::instructions::interrupts::disable();
        } else {
            x86_64::instructions::interrupts::enable();
        }
    }
}
}

For x86_64, Awkernel uses the x86_64 crate to enable and disable interrupts.

AArch64

For x86_64, the AArch64 structure implements the Interrupt trait in awkernel_lib/src/arch/aarch64/interrupt.rs as follows.

#![allow(unused)]
fn main() {
impl Interrupt for super::AArch64 {
    fn get_flag() -> usize {
        awkernel_aarch64::daif::get() as usize
    }

    fn disable() {
        unsafe { core::arch::asm!("msr daifset, #0b0010",) };
    }

    fn enable() {
        unsafe { core::arch::asm!("msr daifclr, #0b0010",) };
    }

    fn set_flag(flag: usize) {
        unsafe { awkernel_aarch64::daif::set(flag as u64) };
    }
}
}

Mapper (Virtual Memory Management)

Mapper is a trait that provides a way to map and unmap virtual memory. It is defined in awkernel_lib/src/paging.rs as follows.

#![allow(unused)]
fn main() {
pub trait Mapper {
    /// Return the physical address of `vm_addr`.
    fn vm_to_phy(vm_addr: VirtAddr) -> Option<PhyAddr>;

    /// Map `vm_addr` to `phy_addr` with `flag`.
    ///
    /// # Safety
    ///
    /// - Virtual memory must be enabled.
    /// - `flag` must be reasonable.
    /// - `phy_addr` must be being unmapped.
    unsafe fn map(vm_addr: VirtAddr, phy_addr: PhyAddr, flags: Flags) -> Result<(), MapError>;

    /// Unmap `vm_addr`.
    ///
    /// # Safety
    ///
    /// - Virtual memory must be enabled.
    /// - `vm_addr` must be being mapped.
    unsafe fn unmap(vm_addr: VirtAddr);
}
}

Mapper uses VirtAddr and PhyAddr types to represent virtual and physical addresses. These types are defined in awkernel_lib/src/addr/virt_addr.rs and awkernel_lib/src/addr/phy_addr.rs.

The Flags type is used to represent the flags of the page table entry. It is defined in awkernel_lib/src/paging.rs as follows.

#![allow(unused)]
fn main() {
/// Flag for a page.
/// Note that every page is readable.
#[derive(Debug, Clone, Copy)]
pub struct Flags {
    pub execute: bool,       // executable
    pub write: bool,         // writable
    pub cache: bool,         // enable cache
    pub write_through: bool, // write back if disabled
    pub device: bool,        // this page is for MMIO, ignored on x86
}
}

There are functions regarding the Mapper trait in awkernel_lib/src/paging.rs as follows.

Awkernel's page size is 4 KiB and it is defined by PAGESIZE:awkernel_lib/src/paging.rs.

#![allow(unused)]
fn main() {
pub const PAGESIZE: usize = 4 * 1024;
}

Implementation

x86_64

For x86_64, the X86 structure implements the Mapper trait in awkernel_lib/src/arch/x86_64/paging.rs. To handle page tables, the OffsetPageTable structure defined in the x86_64 crate is used.

AArch64

For AArch64, the AArch64 structure implements the Mapper trait in awkernel_lib/src/arch/aarch64/paging.rs. To handle page tables, the PageTable structure defined in the awkernel_lib/src/arch/aarch64/page_table.rs is used.

Page Table

PageTable defined in awkernel_lib/src/paging.rs is a trait that provides a way to abstract page tables. It is defined as follows.

#![allow(unused)]
fn main() {
pub trait PageTable<F, FA, E>
where
    F: Frame,
    FA: FrameAllocator<F, E>,
{
    /// Map `virt_addr` to `phy_addr` with `flag`.
    ///
    /// # Safety
    ///
    /// - virt_addr and phy_addr must be aligned to page size.
    unsafe fn map_to(
        &mut self,
        virt_addr: VirtAddr,
        phy_addr: PhyAddr,
        flags: Flags,
        page_allocator: &mut FA,
    ) -> Result<(), E>;
}
}

map_to method of PageTable is used to specify a page frame allocator, which allocates physical pages for the page table, when mapping pages. It is typically used when initializing the kernel's page tables or initializing device drivers.

Frame defined in awkernel_lib/src/paging.rs is a trait to represent a physical page frame. It is defined as follows.

#![allow(unused)]
fn main() {
pub trait Frame {
    fn start_address(&self) -> PhyAddr;
    fn set_address(&mut self, addr: PhyAddr);
    fn size(&self) -> usize;
}
}

FrameAllocator defined in awkernel_lib/src/paging.rs is a trait to allocate physical pages. It is used by PageTable as described above.

Implementation

x86_64

For x86_64, the PageTable structure is defined in awkernel_lib/src/arch/x86_64/page_table.rs as follows.

#![allow(unused)]
fn main() {
pub struct PageTable<'a> {
    offset_page_table: &'a mut OffsetPageTable<'static>,
}
}

The PageTable structure implements the PageTable:awkernel_lib/src/paging.rs trait as follows.

#![allow(unused)]
fn main() {
impl<'a> crate::paging::PageTable<super::page_allocator::Frame, VecPageAllocator, &'static str>
    for PageTable<'a>
{
    unsafe fn map_to(
        &mut self,
        virt_addr: crate::addr::virt_addr::VirtAddr,
        phy_addr: crate::addr::phy_addr::PhyAddr,
        flags: crate::paging::Flags,
        page_allocator: &mut VecPageAllocator,
    ) -> Result<(), &'static str> {
        let flags = flags_to_x86_flags(flags);

        let page = Page::containing_address(VirtAddr::new(virt_addr.as_usize() as u64));
        let frame =
            PhysFrame::<Size4KiB>::containing_address(PhysAddr::new(phy_addr.as_usize() as u64));

        match self
            .offset_page_table
            .map_to(page, frame, flags, page_allocator)
        {
            Ok(flusher) => {
                flusher.flush();
                Ok(())
            }
            Err(_) => Err("Failed to map page"),
        }
    }
}
}

AArch64

For AArch64, the PageTable structure is defined in awkernel_lib/src/arch/aarch64/page_table.rs as follows.

#![allow(unused)]
fn main() {
/// - 3 transition levels
/// - 4KiB page
/// - up to 512GiB memory
pub struct PageTable {
    root: PageTableEntry,
}
}

The PageTable structure implements the PageTable:awkernel_lib/src/paging.rs trait as follows.

#![allow(unused)]
fn main() {
impl crate::paging::PageTable<Page, PageAllocator<Page>, &'static str> for PageTable {
    unsafe fn map_to(
        &mut self,
        virt_addr: VirtAddr,
        phy_addr: PhyAddr,
        flags: crate::paging::Flags,
        page_allocator: &mut PageAllocator<Page>,
    ) -> Result<(), &'static str> {
        let mut f = FLAG_L3_AF | 0b11;

        if !flags.execute {
            f |= FLAG_L3_XN | FLAG_L3_PXN;
        }

        if flags.write {
            f |= FLAG_L3_SH_RW_N;
        } else {
            f |= FLAG_L3_SH_R_N;
        }

        match (flags.device, flags.cache) {
            (true, true) => f |= FLAG_L3_ATTR_MEM | FLAG_L3_OSH,
            (true, false) => f |= FLAG_L3_ATTR_DEV | FLAG_L3_OSH,
            (false, true) => f |= FLAG_L3_ATTR_MEM | FLAG_L3_ISH,
            (false, false) => f |= FLAG_L3_NS | FLAG_L3_ISH,
        }

        self.map_to_aarch64(virt_addr, phy_addr, f, page_allocator)
    }
}
}

Context Switch

Context defined in awkernel_lib/src/context.rs is a trait that enables preemptive multitasking. It provides methods to set the stack pointer, entry point, and argument of the context as follows.

#![allow(unused)]
fn main() {
pub trait Context: Default {
    /// # Safety
    ///
    /// Ensure that changing the stack pointer is valid at that time.
    unsafe fn set_stack_pointer(&mut self, sp: usize);

    /// # Safety
    ///
    /// This function must be called for only initialization purpose.
    unsafe fn set_entry_point(&mut self, entry: extern "C" fn(usize) -> !, arg: usize);

    /// # Safety
    ///
    /// This function must be called for only initialization purpose.
    unsafe fn set_argument(&mut self, arg: usize);
}
}

The context_switch function must be implemented for each architecture to enable context switching.

#![allow(unused)]
fn main() {
extern "C" {
    /// Switch context from `current` to `next`.
    pub fn context_switch(current: *mut ArchContext, next: *const ArchContext);
}
}

The context_switch function stores and restores CPU registers, and the ArchContext structure is an architecture-specific context structure.

Disable Preemption

In awkernel_async_lib/Cargo.toml, there is the no_preempt feature to disable preemption.

[features]
default = []
std = ["awkernel_lib/std", "no_preempt"]
no_preempt = []

If you want to disable preemption, please enable the no_preempt feature as default = ["no_preempt"].

Implementation

x86_64

For x86_64, the Context structure is defined in awkernel_lib/src/context/x86_64.rs as follows.

#![allow(unused)]
fn main() {
#[derive(Debug, Copy, Clone, Default)]
#[repr(C)]
pub struct Context {
    pub rbx: u64,
    pub rsp: u64,
    pub rbp: u64,
    pub r12: u64,
    pub r13: u64,
    pub r14: u64,
    pub r15: u64,
}
}

The Context structure is imported as the ArchContext in awkernel_lib/src/context.rs.

The context_switch function is implemented in assembly as follows. This stores and restores only general-purpose registers because the floating-point registers are stored and restored by each interrupt handler.

#![allow(unused)]
fn main() {
core::arch::global_asm!(
    "
.global context_switch
context_switch:
// Store general purpose registers
mov   [rdi], rbx
mov  8[rdi], rsp
mov 16[rdi], rbp
mov 24[rdi], r12
mov 32[rdi], r13
mov 40[rdi], r14
mov 48[rdi], r15

// Load general purpose registers
mov rbx,   [rsi]
mov rsp,  8[rsi]
mov rbp, 16[rsi]
mov r12, 24[rsi]
mov r13, 32[rsi]
mov r14, 40[rsi]
mov r15, 48[rsi]

ret
"
);
}

The Context:awkernel_lib/src/context.rs trait is implemented for the Context:awkernel_lib/src/context/x86_64.rs structure as follows. These methods set the stack pointer, entry point, and argument of the context.

#![allow(unused)]
fn main() {
impl crate::context::Context for Context {
    unsafe fn set_stack_pointer(&mut self, sp: usize) {
        self.rsp = sp as u64;
    }

    unsafe fn set_entry_point(&mut self, entry: extern "C" fn(usize) -> !, arg: usize) {
        self.r12 = arg as u64;
        self.r13 = entry as usize as u64;

        let entry_point_addr = entry_point as usize as u64;

        unsafe {
            core::arch::asm!("mov {}, rsp", lateout(reg) self.r15);
            core::arch::asm!("mov rsp, {}", in(reg) self.rsp);
            core::arch::asm!("push {}", in(reg) entry_point_addr);
            core::arch::asm!("mov rsp, {}", in(reg) self.r15);
        }

        self.rsp -= 8;
    }

    unsafe fn set_argument(&mut self, arg: usize) {
        self.r12 = arg as u64;
    }
}

extern "C" {
    fn entry_point() -> !;
}

global_asm!(
    "
.global entry_point
entry_point:
    mov  rdi, r12
    call r13
1:
    hlt
    jmp  1b
"
);
}

AArch64

For AArch64, the Context structure is defined in awkernel_lib/src/context/aarch64.rs as follows.

#![allow(unused)]
fn main() {
#[derive(Debug, Copy, Clone, Default)]
#[repr(C)]
pub struct Context {
    // 8 * 8 bytes
    pub fp_regs: FPRegs, // floating point registers

    //------------------------------ offset: 16 * 4 (+4)

    // 8 * 12 bytes
    pub gp_regs: GPRegs, // general purpose registers

    // ----------------------------- offset: 16 * 10 (+6)

    // 8 * 2 bytes
    pub sp: u64, // stack pointer
    _unused: [u8; 8],
}
}

The Context structure is imported as the ArchContext in awkernel_lib/src/context.rs.

The context_switch function is implemented in assembly as follows.

#![allow(unused)]
fn main() {
core::arch::global_asm!(
    "
.global context_switch
context_switch:
// Save the current context.

// Store floating-point registers.
stp      d8,  d9, [x0], #16
stp     d10, d11, [x0], #16
stp     d12, d13, [x0], #16
stp     d14, d15, [x0], #16

// Store general purpose registers.
stp     x19, x20, [x0], #16
stp     x21, x22, [x0], #16
stp     x23, x24, [x0], #16
stp     x25, x26, [x0], #16
stp     x27, x28, [x0], #16
stp     x29, x30, [x0], #16

// Store SP.
mov     x9, sp
str     x9, [x0]


// Restore the next context.

// Load floating-point registers.
ldp      d8,  d9, [x1], #16
ldp     d10, d11, [x1], #16
ldp     d12, d13, [x1], #16
ldp     d14, d15, [x1], #16

// Load general purpose registers.
ldp     x19, x20, [x1], #16
ldp     x21, x22, [x1], #16
ldp     x23, x24, [x1], #16
ldp     x25, x26, [x1], #16
ldp     x27, x28, [x1], #16
ldp     x29, x30, [x1], #16

// Load SP.
ldr     x9, [x1]
mov     sp, x9

ret
"
);
}

The Context:awkernel_lib/src/context.rs trait is implemented for the Context:awkernel_lib/src/context/aarch64.rs structure as follows. These methods set the stack pointer, entry point, and argument of the context.

#![allow(unused)]
fn main() {
impl crate::context::Context for Context {
    unsafe fn set_stack_pointer(&mut self, sp: usize) {
        self.sp = sp as u64;
    }

    unsafe fn set_entry_point(&mut self, entry: extern "C" fn(usize) -> !, arg: usize) {
        self.gp_regs.x19 = arg as u64;
        self.gp_regs.x20 = entry as usize as u64;
        self.gp_regs.x30 = entry_point as *const () as u64;
    }

    unsafe fn set_argument(&mut self, arg: usize) {
        self.gp_regs.x19 = arg as u64;
    }
}

extern "C" {
    fn entry_point() -> !;
}

global_asm!(
    "
.global entry_point
entry_point:
    mov     x0, x19
    blr     x20
1:
    wfi
    b       1b
"
);
}

Interrupt Controller

InterruptController is a trait for interrupt controllers. It is defined in awkernel_lib/src/interrupt.rs as follows.

#![allow(unused)]
fn main() {
pub trait InterruptController: Sync + Send {
    fn enable_irq(&mut self, irq: u16);
    fn disable_irq(&mut self, irq: u16);
    fn pending_irqs(&self) -> Box<dyn Iterator<Item = u16>>;

    /// Send an inter-process interrupt to `target` CPU.
    fn send_ipi(&mut self, irq: u16, cpu_id: u32);

    /// Send an inter-process interrupt to all CPUs.
    fn send_ipi_broadcast(&mut self, irq: u16);

    /// Send an inter-process interrupt to all CPUs except the sender CPU.
    fn send_ipi_broadcast_without_self(&mut self, irq: u16);

    /// Initialization for non-primary core.
    fn init_non_primary(&mut self) {}

    /// End of interrupt.
    /// This will be used by only x86_64.
    fn eoi(&mut self) {}

    /// Return the range of IRQs, which can be registered.
    /// The range is [start, end).
    fn irq_range(&self) -> (u16, u16);

    /// Return the range of IRQs, which can be used for PnP devices.
    /// The range is [start, end).
    fn irq_range_for_pnp(&self) -> (u16, u16);

    /// Set the PCIe MSI or MSI-X interrupt
    #[allow(unused_variables)]
    fn set_pcie_msi(
        &self,
        segment_number: usize,
        target: u32,
        irq: u16,
        message_data: &mut u32,
        message_address: &mut u32,
        message_address_upper: Option<&mut u32>,
    ) -> Result<IRQ, &'static str> {
        Err("Interrupt controller does not support PCIe MSI or MSI-X.")
    }
}
}

Some related functions are defined in awkernel_lib/src/interrupt.rs as follows.

Handling Interrupts

x86_64

handle_irq is called in interrupt handlers defined in. kernel/src/arch/x86_64/interrupt_handler.rs for x86_64 as follows.

#![allow(unused)]
fn main() {
macro_rules! irq_handler {
    ($name:ident, $id:expr) => {
        extern "x86-interrupt" fn $name(_stack_frame: InterruptStackFrame) {
            awkernel_lib::interrupt::eoi(); // End of interrupt.
            awkernel_lib::interrupt::handle_irq($id);
        }
    };
}
}

irq_handler macro is called in each interrupt handler.

AArch64

The handle_irqs function is called in interrupt handlers defined in. kernel/src/arch/aarch64/exception.rs for aarch64 as follows.

#![allow(unused)]
fn main() {
#[no_mangle]
pub extern "C" fn curr_el_spx_irq_el1(_ctx: *mut Context, _sp: usize, _esr: usize) {
    interrupt::handle_irqs();
}
}

Handling Preemption

A preemption request can be sent by an inter process interrupt (IPI) to the target CPU. This means that the target CPU should handle the preemption request if it receives the IPI.

x86_64

For x86_64, the handle_preempt function is called in a interrupt handler defined in kernel/src/arch/x86_64/interrupt_handler.rs as follows.

#![allow(unused)]
fn main() {
extern "x86-interrupt" fn preemption(_stack_frame: InterruptStackFrame) {
    awkernel_lib::interrupt::eoi(); // End of interrupt.
    awkernel_lib::interrupt::handle_preemption();
}
}

AArch64

For AArch64, handling preemption is performed in the handle_irqs function defined in awkernel_lib/src/interrupt.rs.

#![allow(unused)]
fn main() {
/// Handle all pending interrupt requests.
/// This function will be used by only aarch64 and called from CPU's interrupt handlers.
#[cfg(feature = "aarch64")]
pub fn handle_irqs() {
    use crate::{heap, unwind::catch_unwind};
    use core::mem::transmute;

    let handlers = IRQ_HANDLERS.read();
    let mut need_preemption = false;

    // omitted

    if need_preemption {
        let ptr = PREEMPT_FN.load(Ordering::Relaxed);
        let preemption = unsafe { transmute::<*mut (), fn()>(ptr) };
        preemption();
    }
}
}

Implementation

There are some device drivers for interrupt controllers in awkernel_drivers/src/interrupt_controller.

x86_64

xAPIC and x2APIC are supported for x86_64.

• xAPIC and x2APIC

AArch64

BCM2835's (Raspberry Pi 3) interrupt controller, GICv2 and GICv3 are supported for AAarch64.

• BCM2835's interrupt controller
• GICv2
• GICv3

Memory Allocator

Awkernel uses rlsf, which implements Two-Level Segregated Fit (TLSF) memory allocator. The Tallock structure represents an allocator in Awkernel, which contains a primary allocator and a backup allocator. Async/await tasks use only the primary allocator, but kernel tasks, such as interrupt handlers, use both the primary and the backup allocators for safety.

The following code shows how to use the primary and backup allocators in the task scheduler defined in awkernel_async_lib/src/task.rs.

#![allow(unused)]
fn main() {
pub fn run_main() {
    loop {
        if let Some(task) = get_next_task() {
            // Use the primary memory allocator.
            #[cfg(not(feature = "std"))]
            unsafe {
                awkernel_lib::heap::TALLOC.use_primary()
            };

            let result = catch_unwind(|| {
                guard.poll_unpin(&mut ctx)
            });

            // Use the primary and backup memory allocator.
            unsafe {
                awkernel_lib::heap::TALLOC.use_primary_then_backup()
            };
        }
    }
}
}

In run_main function, a executable task is taken from the task queue by get_next_task function. Before executing the task, awkernel_lib::heap::TALLOC.use_primary() is called to use only the primary memory allocator. The task is executed by calling poll_unpin method in the catch_unwind block to catch a panic. If the task exhausts the primary memory region, it will panic and run_main function will catch the panic. After catching the panic, awkernel_lib::heap::TALLOC.use_primary_then_backup() is called to use both the primary and backup memory allocators, and safely deallocate the task.

The Tallock structure is defined in awkernel_lib/src/heap.rs as follows.

#![allow(unused)]
fn main() {
struct Allocator(Mutex<TLSFAlloc>);
struct BackUpAllocator(Mutex<TLSFAlloc>);

pub struct Talloc {
    primary: Allocator,
    backup: BackUpAllocator,

    /// bitmap for each CPU to decide which allocator to use
    flags: [AtomicU32; NUM_MAX_CPU / 32],

    primary_start: AtomicUsize,
    primary_size: AtomicUsize,
    backup_start: AtomicUsize,
    backup_size: AtomicUsize,
}
}

The Talloc structure is defined as a global allocator as follows.

#![allow(unused)]
fn main() {
#[global_allocator]
pub static TALLOC: Talloc = Talloc::new();
}

There are 2 functions to initialize memory regions of the primary and backup allocators.

Initialization

x86_64

For x86_64, the primary and backup allocators are initialized in init_primary_heap and init_backup_heap functions defined in kernel/src/arch/x86_64/kernel_main.rs as follows. These functions initialize virtual memory regions for the primary and backup heaps before initializing the primary and backup allocators.

#![allow(unused)]
fn main() {
fn init_primary_heap(
    page_table: &mut OffsetPageTable<'static>,
    page_allocators: &mut BTreeMap<u32, VecPageAllocator>,
) {
    let primary_start = HEAP_START + BACKUP_HEAP_SIZE;

    let num_pages = map_primary_heap(page_table, page_allocators, primary_start);

    let heap_size = num_pages * PAGESIZE;
    unsafe { awkernel_lib::heap::init_primary(primary_start, heap_size) };

    // omitted
}
}

#![allow(unused)]
fn main() {
fn init_backup_heap(
    boot_info: &mut BootInfo,
    page_table: &mut OffsetPageTable<'static>,
) -> (usize, MemoryRegion, Option<PhysFrame>) {
    // omitted: Initialize virtual memory regions for the backup heap.

    // Initialize.
    // Enable heap allocator.
    unsafe {
        awkernel_lib::heap::init_backup(HEAP_START, BACKUP_HEAP_SIZE);
        awkernel_lib::heap::TALLOC.use_primary_then_backup();
    }

    (backup_pages, backup_heap_region, next_page)
}
}

AArch64

For x86_64, the primary and backup allocators are initialized in primary_cpu function defined in kernel/src/arch/aarch64/kernel_main.rs as follows.

#![allow(unused)]
fn main() {
unsafe fn primary_cpu(device_tree_base: usize) {
    // omitted

    // 5. Enable heap allocator.
    let backup_start = HEAP_START;
    let backup_size = BACKUP_HEAP_SIZE;
    let primary_start = HEAP_START + BACKUP_HEAP_SIZE;
    let primary_size = vm.get_heap_size().unwrap() - BACKUP_HEAP_SIZE;

    heap::init_primary(primary_start, primary_size);
    heap::init_backup(backup_start, backup_size);

    // omitted
}
}

Scheduler

Scheduler is a trait for the scheduler and defined in awkernel_async_lib/src/scheduler.rs as follows.

#![allow(unused)]
fn main() {
pub(crate) trait Scheduler {
    /// Enqueue an executable task.
    /// The enqueued task will be taken by `get_next()`.
    fn wake_task(&self, task: Arc<Task>);

    /// Get the next executable task.
    fn get_next(&self) -> Option<Arc<Task>>;

    /// Get the scheduler name.
    fn scheduler_name(&self) -> SchedulerType;

    #[allow(dead_code)] // TODO: to be removed
    fn priority(&self) -> u8;
}
}

There are several functions regarding the scheduler in awkernel_async_lib/src/scheduler.rs.

SchedulerType is an enum for the scheduler type and defined in awkernel_async_lib/src/scheduler.rs as follows.

#![allow(unused)]
fn main() {
pub enum SchedulerType {
    GEDF(u64), // relative deadline
    PrioritizedFIFO(u8),
    PrioritizedRR(u8),
    Panicked,
}
}

SleepingTasks

SleepingTasks is a struct for managing sleeping tasks and defined in awkernel_async_lib/src/scheduler.rs as follows.

#![allow(unused)]
fn main() {
struct SleepingTasks {
    delta_list: DeltaList<Box<dyn FnOnce() + Send>>,
    base_time: u64,
}
}

SleepingTasks struct has the following functions.

Scheduler Implementation

Some schedulers are implemented under the folder awkernel_async_lib/src/scheduler.

$ ls awkernel_async_lib/src/scheduler
> gedf.rs  panicked.rs  prioritized_fifo.rs  prioritized_rr.rs

A scheduler can be implemented by implementing Scheduler Trait. Each scheduler must be registered in the following three locations. fn get_next_task(), fn get_scheduler(sched_type: SchedulerType) and pub enum SchedulerType.

GEDF Scheduler

The Global Earliest Deadline First (GEDF) scheduler is implemented in gedf.rs. This scheduler implements a real-time scheduling algorithm that prioritizes tasks based on their absolute deadlines.

The scheduler maintains a BinaryHeap<GEDFTask> as its run queue, where tasks are ordered by their absolute deadlines. When a task is enqueued via wake_task(), the scheduler calculates the absolute deadline by adding the relative deadline (specified in SchedulerType::GEDF(relative_deadline)) to the current uptime. The task's priority is updated using MAX_TASK_PRIORITY - absolute_deadline to ensure proper inter-scheduler priority comparison.

The GEDFTask struct implements custom ordering where tasks are compared first by absolute deadline (earlier deadlines have higher priority), and then by wake time for tie-breaking. The scheduler supports preemption through the invoke_preemption() method, which sends IPIs to target CPUs when a task with an earlier deadline arrives and can preempt currently running tasks.

PrioritizedFIFO Scheduler

The PrioritizedFIFO scheduler is implemented in prioritized_fifo.rs. This scheduler provides fixed-priority scheduling where tasks are executed in First-In-First-Out order within each priority level.

The scheduler uses a PriorityQueue<PrioritizedFIFOTask> as its run queue. When a task is enqueued through wake_task(), the priority is extracted from SchedulerType::PrioritizedFIFO(priority) and used to insert the task into the priority queue. The get_next() method retrieves the task at the head of the highest-priority non-empty queue.

The scheduler implements preemption via invoke_preemption(), which evaluates all currently running tasks and determines if the newly awakened task should preempt any of them. If preemption is triggered, the scheduler sends an IPI to the target CPU and updates the preemption pending queue.

PrioritizedRR Scheduler

The PrioritizedRR (Prioritized Round Robin) scheduler is implemented in prioritized_rr.rs. This scheduler combines fixed-priority scheduling with time quantum enforcement to provide fair CPU time distribution.

The scheduler maintains a PriorityQueue<PrioritizedRRTask> similar to PrioritizedFIFO, but adds time quantum management with a default interval of 4ms (4,000 microseconds). The scheduler provides two preemption mechanisms: invoke_preemption_wake() for priority-based preemption when tasks are awakened, and invoke_preemption_tick() for time quantum-based preemption.

The invoke_preemption_tick() method is called periodically on primary CPU to check if the currently running task has exceeded its time quantum. It compares the elapsed execution time against the configured interval and triggers preemption by sending an IPI if the quantum is exceeded.

Panicked Scheduler

The Panicked scheduler is implemented in panicked.rs. This scheduler handles tasks that have entered a panicked state and provides them with the lowest scheduling priority in the system. The scheduler uses a simple VecDeque<Arc<Task>> as its run queue, implementing basic FIFO ordering without any priority considerations.

PCIe

This section explains the functions and structures defined in awkernel/awkernel_drivers/src/pcie.rs.

PCIeTree

PCIeTree is a structure for managing a hierarchical collection of PCIe buses.

#![allow(unused)]
fn main() {
struct PCIeTree {
    // - Key: Bus number
    // - Value: PCIeBus
    tree: BTreeMap<u8, Box<PCIeBus>>,
}
}

There are several methods regarding the PCIeTree structure.

PCIeTree structure implements the fmt::Display trait as follows.

#![allow(unused)]
fn main() {
impl fmt::Display for PCIeTree {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        for (_, bus) in self.tree.iter() {
            if !bus.devices.is_empty() {
                write!(f, "{bus}")?;
            }
        }

        Ok(())
    }
}
}

PCIeBus

PCIeBus is a structure that represents individual PCIe buses.

#![allow(unused)]
fn main() {
pub struct PCIeBus {
    segment_group: u16,
    bus_number: u8,
    base_address: Option<VirtAddr>,
    info: Option<PCIeInfo>,
    devices: Vec<ChildDevice>,
}
}

There are several methods regarding the PCIeBus structure.

PCIeBus structure implements the PCIeDevice trait in awkernel/awkernel_drivers/src/pcie.rs as follows.

#![allow(unused)]
fn main() {
impl PCIeDevice for PCIeBus {
    fn device_name(&self) -> Cow<'static, str> {
        if let Some(info) = self.info.as_ref() {
            let bfd = info.get_bfd();
            let name = format!("{bfd}: Bridge, Bus #{:02x}", self.bus_number);
            name.into()
        } else {
            let name = format!("Bus #{:02x}", self.bus_number);
            name.into()
        }
    }

    fn children(&self) -> Option<&Vec<ChildDevice>> {
        Some(&self.devices)
    }
}
}

PCIeBus structure implements the fmt::Display trait as follows.

#![allow(unused)]
fn main() {
impl fmt::Display for PCIeBus {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        print_pcie_devices(self, f, 0)
    }
}
}

PCIeInfo

PCIeInfo is a structure used to store the essential information required to initialize a PCIe device.

#![allow(unused)]
fn main() {
/// Information necessary for initializing the device
#[derive(Debug)]
pub struct PCIeInfo {
    pub(crate) config_space: ConfigSpace,
    segment_group: u16,
    bus_number: u8,
    device_number: u8,
    function_number: u8,
    id: u16,
    vendor: u16,
    revision_id: u8,
    interrupt_pin: u8,
    pcie_class: pcie_class::PCIeClass,
    device_name: Option<pcie_id::PCIeID>,
    pub(crate) header_type: u8,
    base_addresses: [BaseAddress; 6],
    msi: Option<capability::msi::Msi>,
    msix: Option<capability::msix::Msix>,
    pcie_cap: Option<capability::pcie_cap::PCIeCap>,

    // The bridge having this device.
    bridge_bus_number: Option<u8>,
    bridge_device_number: Option<u8>,
    bridge_function_number: Option<u8>,
}
}

There are several methods regarding PCIeInfo structure.

PCIeInfo structure implements the fmt::Display trait as follows.

#![allow(unused)]
fn main() {
impl fmt::Display for PCIeInfo {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(
            f,
            "{:04x}:{:02x}:{:02x}.{:01x}, Device ID = {:04x}, PCIe Class = {:?}",
            self.segment_group,
            self.bus_number,
            self.device_number,
            self.function_number,
            self.id,
            self.pcie_class,
        )
    }
}
}

ChildDevice

ChildDevice is an enum that represents different types of devices and their attachment states on a PCIe bus.

#![allow(unused)]
fn main() {
pub enum ChildDevice {
    Bus(Box<PCIeBus>),
    Attached(Arc<dyn PCIeDevice + Sync + Send>),
    Attaching,
    Unattached(Box<PCIeInfo>),
}
}

There are several methods regarding ChildDevice enum.

UnknownDevice

UnknownDevice is a structure that represents an attached PCIe device including its segment group, bus, device and function numbers.

#![allow(unused)]
fn main() {
struct UnknownDevice {
    segment_group: u16,
    bus_number: u8,
    device_number: u8,
    function_number: u8,
    vendor: u16,
    id: u16,
    pcie_class: pcie_class::PCIeClass,
}
}

UnknownDevice structure implements the PCIeDevice trait in awkernel/awkernel_drivers/src/pcie.rs as follows.

#![allow(unused)]
fn main() {
impl PCIeDevice for PCIeBus {
    fn device_name(&self) -> Cow<'static, str> {
        let bfd = format!(
            "{:04x}:{:02x}:{:02x}.{:01x}",
            self.segment_group, self.bus_number, self.device_number, self.function_number
        );

        let name = format!(
            "{bfd}: Vendor ID = {:04x}, Device ID = {:04x}, PCIe Class = {:?}",
            self.vendor, self.id, self.pcie_class,
        );
        name.into()
    }

    fn children(&self) -> Option<&Vec<ChildDevice>> {
        None
    }
}
}

PCIeDeviceErr

PCIeDeviceErr is an enum that represents various error types related to PCIe devices.

#![allow(unused)]
fn main() {
#[derive(Debug, Clone)]
pub enum PCIeDeviceErr {
    InitFailure,
    ReadFailure,
    PageTableFailure,
    CommandFailure,
    UnRecognizedDevice { bus: u8, device: u16, vendor: u16 },
    InvalidClass,
    Interrupt,
    NotImplemented,
    BARFailure,
}
}

PCIeDeviceErr structure implements the fmt::Display trait as follows.

#![allow(unused)]
fn main() {
impl fmt::Display for PCIeDeviceErr {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::InitFailure => {
                write!(f, "Failed to initialize the device driver.")
            }
            // omitted
        }
    }
}
}

Initialization

x86

For x86, the PCIe is initialized with ACPI or I/O port by init_with_acpi:awkernel/awkernel_drivers/src/pcie.rs or init_with_io:awkernel/awkernel_drivers/src/pcie.rs as follows.

#![allow(unused)]
fn main() {
/// Initialize the PCIe with ACPI.
#[cfg(feature = "x86")]
pub fn init_with_acpi(acpi: &AcpiTables<AcpiMapper>) -> Result<(), PCIeDeviceErr> {
    use awkernel_lib::{addr::phy_addr::PhyAddr, paging::Flags};

    const CONFIG_SPACE_SIZE: usize = 256 * 1024 * 1024; // 256 MiB

    let pcie_info = PciConfigRegions::new(acpi).or(Err(PCIeDeviceErr::InitFailure))?;
    for segment in pcie_info.iter() {
        let flags = Flags {
            write: true,
            execute: false,
            cache: false,
            write_through: false,
            device: true,
        };

        let mut config_start = segment.physical_address;
        let config_end = config_start + CONFIG_SPACE_SIZE;

        while config_start < config_end {
            let phy_addr = PhyAddr::new(config_start);
            let virt_addr = VirtAddr::new(config_start);

            unsafe {
                paging::map(virt_addr, phy_addr, flags).or(Err(PCIeDeviceErr::PageTableFailure))?
            };

            config_start += PAGESIZE;
        }

        let base_address = segment.physical_address;
        init_with_addr(segment.segment_group, VirtAddr::new(base_address), None);
    }

    Ok(())
}
}

#![allow(unused)]
fn main() {
/// Initialize the PCIe with IO port.
#[cfg(feature = "x86")]
pub fn init_with_io() {
    init(0, None, PCIeInfo::from_io, None);
}
}

Others

The PCIe is initialized with the base address by init_with_addr:awkernel/awkernel_drivers/src/pcie.rs or init:awkernel/awkernel_drivers/src/pcie.rs as follows.

#![allow(unused)]
fn main() {
/// If `ranges` is not None, the base address registers of the device will be initialized by using `ranges`.
pub fn init_with_addr(
    segment_group: u16,
    base_address: VirtAddr,
    ranges: Option<&mut [PCIeRange]>,
) {
    init(
        segment_group,
        Some(base_address),
        PCIeInfo::from_addr,
        ranges,
    );
}
}

#![allow(unused)]
fn main() {
fn init<F>(
    segment_group: u16,
    base_address: Option<VirtAddr>,
    f: F,
    ranges: Option<&mut [PCIeRange]>,
) where
    F: Fn(u16, u8, u8, u8, VirtAddr) -> Result<PCIeInfo, PCIeDeviceErr>,
{
    let mut visited = BTreeSet::new();

    let mut bus_tree = PCIeTree {
        tree: BTreeMap::new(),
    };

    let mut host_bridge_bus = 0;

    //omitted: Construct `PCIeTree` and create a tree structure.

    bus_tree.update_bridge_info(host_bridge_bus, 0, 0);

    if let Some(ranges) = ranges {
        bus_tree.init_base_address(ranges);
    }

    bus_tree.attach();

    log::info!("PCIe: segment_group = {segment_group:04x}\r\n{bus_tree}");

    let mut node = MCSNode::new();
    let mut pcie_trees = PCIE_TREES.lock(&mut node);
    pcie_trees.insert(segment_group, Arc::new(bus_tree));
}
}

Checking the PCI Buses

The following functions are used to check all devices on the PCIe bus.

check_bus

Check thirty-two devices on the PCIe bus.

#![allow(unused)]
fn main() {
#[inline]
fn check_bus<F>(bus: &mut PCIeBus, bus_tree: &mut PCIeTree, visited: &mut BTreeSet<u8>, f: &F)
where
    F: Fn(u16, u8, u8, u8, VirtAddr) -> Result<PCIeInfo, PCIeDeviceErr>,
{
    for device in 0..32 {
        check_device(bus, device, bus_tree, visited, f);
    }
}
}

check_device

Check eight functions on the device.

#![allow(unused)]
fn main() {
#[inline]
fn check_device<F>(
    bus: &mut PCIeBus,
    device: u8,
    bus_tree: &mut PCIeTree,
    visited: &mut BTreeSet<u8>,
    f: &F,
) where
    F: Fn(u16, u8, u8, u8, VirtAddr) -> Result<PCIeInfo, PCIeDeviceErr>,
{
    for function in 0..8 {
        check_function(bus, device, function, bus_tree, visited, f);
    }
}
}

check_function

Retrieve PCIe bus information and store it in the PCIeTree structure.

#![allow(unused)]
fn main() {
fn check_function<F>(
    bus: &mut PCIeBus,
    device: u8,
    function: u8,
    bus_tree: &mut PCIeTree,
    visited: &mut BTreeSet<u8>,
    f: &F,
) -> bool
where
    F: Fn(u16, u8, u8, u8, VirtAddr) -> Result<PCIeInfo, PCIeDeviceErr>,
{
    let offset =
        (bus.bus_number as usize) << 20 | (device as usize) << 15 | (function as usize) << 12;

    let addr = if let Some(base_address) = bus.base_address {
        base_address + offset
    } else {
        VirtAddr::new(0)
    };

    if let Ok(info) = f(bus.segment_group, bus.bus_number, device, function, addr) {

        // omitted: Push PCIe device to `PCIeTree`, considering if it is a bridge device

        true
    } else {
        false
    }
}

}

Read the base address

Read the base address specified by the offset from the PCIe device's configuration space.

read_bar

Read the base address.

#![allow(unused)]
fn main() {
/// Read the base address of `addr`.
fn read_bar(config_space: &ConfigSpace, offset: usize) -> BaseAddress {
    let bar = config_space.read_u32(offset);

    if (bar & BAR_IO) == 1 {

        // omitted: Read the base address for x86

} else {
        // Memory space

        let bar_type = bar & BAR_TYPE_MASK;
        if bar_type == BAR_TYPE_32 {

            // ommitted: Read the base address for 32bit target

        } else if bar_type == BAR_TYPE_64 {

            // ommitted: Read the base address for 64bit target

        } else {
            BaseAddress::None
        }
    }
}

}

Print the PCIe devices

Print the configuration of the devices on the PCIe bus.

print_pcie_devices

Print the configuration of PCIe devices, including device names, vendor ID, device ID, PCIe class and bridge information.

#![allow(unused)]
fn main() {
fn print_pcie_devices(device: &dyn PCIeDevice, f: &mut fmt::Formatter, indent: u8) -> fmt::Result {
    let indent_str = " ".repeat(indent as usize * 4);
    write!(f, "{}{}\r\n", indent_str, device.device_name())?;

    if let Some(children) = device.children() {
        for child in children.iter() {
            match child {
                ChildDevice::Attached(child) => {
                    print_pcie_devices(child.as_ref(), f, indent + 1)?;
                }
                ChildDevice::Unattached(info) => {
                    let name = format!(
                        "{}: Vendor ID = {:04x}, Device ID = {:04x}, PCIe Class = {:?}, bridge = {:?}-{:?}-{:?}",
                        info.get_bfd(),
                        info.vendor,
                        info.id,
                        info.pcie_class,
                        info.bridge_bus_number,
                        info.bridge_device_number,
                        info.bridge_function_number,
                    );

                    let indent_str = " ".repeat((indent as usize + 1) * 4);
                    write!(f, "{indent_str}{name}\r\n")?;
                }
                ChildDevice::Bus(bus) => {
                    print_pcie_devices(bus.as_ref(), f, indent + 1)?;
                }
                _ => (),
            }
        }
    }

    Ok(())
}
}

License

Awkernel License

See https://github.com/tier4/awkernel/blob/main/LICENSE.

OpenBSD License

Copyright (c) 1982, 1986, 1990, 1991, 1993
	The Regents of the University of California.  All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
 1. Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.
 2. Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.
 3. All advertising materials mentioning features or use of this software
    must display the following acknowledgement:
	This product includes software developed by the University of
	California, Berkeley and its contributors.
 4. Neither the name of the University nor the names of its contributors
    may be used to endorse or promote products derived from this software
    without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

smoltcp License

Copyright (C) smoltcp contributors

Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN
AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Futures License

Copyright (c) 2016 Alex Crichton
Copyright (c) 2017 The Tokio Authors

Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.

Ixgbe License

SPDX-License-Identifier: BSD-3-Clause

Copyright (c) 2001-2017, Intel Corporation
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

 1. Redistributions of source code must retain the above copyright notice,
    this list of conditions and the following disclaimer.

 2. Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.

 3. Neither the name of the Intel Corporation nor the names of its
    contributors may be used to endorse or promote products derived from
    this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

Igb License

Copyright (c) 2001-2003, Intel Corporation
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

 1. Redistributions of source code must retain the above copyright notice,
    this list of conditions and the following disclaimer.

 2. Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.

 3. Neither the name of the Intel Corporation nor the names of its
    contributors may be used to endorse or promote products derived from
    this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.

Genet License

Copyright (c) 2020 Jared McNeill <jmcneill@invisible.ca>
Copyright (c) 2020 Mark Kettenis <kettenis@openbsd.org>
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.