Booting, My Confusions and ChatGPT

Early boot is easier to understand as a ladder:

Power applied
    |
    v
CPU can fetch first instruction
    |
    v
Firmware can execute from ROM/flash
    |
    v
Tiny temporary writable storage exists
    |
    v
DRAM is initialized
    |
    v
Assembly prepares a C runtime
    |
    v
Memory types and caches are configured
    |
    v
First page tables create virtual memory
    |
    v
Larger firmware, bootloader, kernel, or OS code can run normally

Important rule:

• Every step earns one new assumption.
• Do not use an assumption before the boot path has created it.

Common false assumptions:

• “The CPU is executing, so RAM must work.”
• “There is a stack because C code has local variables.”
• “A virtual address exists because the program was linked with it.”
• “A physical address is enough, so cacheability does not matter.”
• “ROM execution means global variables are writable.”

Firmware is not the first thing that exists. Firmware is the first platform-visible instruction stream that software developers usually see. Below it, the CPU and platform already need enough hardwired behavior to reset, clock, fetch, decode, and begin architectural execution.

Layer view:

power-good / reset / clocks
      |
      v
hardwired reset sequencing and first-fetch control
      |
      v
built-in decode / micro-op / microcode machinery
      |
      v
architectural instruction execution
      |
      v
platform firmware fetched from ROM/flash

Pointwise:

• Power and reset signals bring the CPU into a defined reset state.
• Clocking and reset sequencing are below firmware.
• Address-decode logic must make the reset-vector fetch reach firmware storage.
• Some instruction fetch, decode, and execution machinery must already work before external firmware code can run.
• Microcode itself also needs lower-level hardware control and sequencing.
• Public manuals document the architectural reset state and programmer-visible behavior; exact internal sequencing is usually vendor-private.

Useful distinction:

hardwired/control logic:
  makes reset, first fetch, minimum decode, and low-level sequencing possible

built-in microcode/control store:
  helps implement complex architectural behavior inside the CPU

platform firmware:
  normal ISA-visible code fetched from ROM/flash

The caveat:

• Microcode is not “BIOS running earlier.”
• Firmware is software visible at the ISA level.
• Microcode is internal CPU implementation and update machinery below that level.
• Even microcode needs some lower-level signal-driven hardware to get control.

Reference:

• Intel documents the architectural reset state and programmer-visible execution model in the SDM, while internal sequencing remains implementation-specific: Intel SDM.

Before DRAM is initialized:

• Instruction bytes may come from ROM or flash.
• Writable memory may not exist yet.
• A stack may not exist yet.
• Global variables may not be usable.
• .bss is not guaranteed to be zero.
• .data may still be sitting in ROM, not RAM.

Early code may use:

• CPU registers;
• CPU or SoC scratch registers;
• tiny on-chip SRAM, if available;
• cache-as-RAM;
• temporary memory created by firmware.

Diagram:

Before DRAM init

Instruction fetch:
  CPU ---> firmware ROM/flash      works

Writable data:
  CPU ---> DRAM                    not safe yet
  CPU ---> stack in DRAM           not safe yet
  CPU ---> globals in DRAM         not safe yet

Practical consequence:

• Early code is often assembly.
• If C is used, it must be a restricted subset or must run only after startup code has created the required environment.

References:

• UEFI PI SEC phase describes temporary memory and cache-as-RAM style behavior: UEFI PI SEC phase.
• coreboot describes bootblock responsibilities such as cache-as-RAM, stack setup, BSS clearing, and loading the next stage: coreboot architecture.

XIP means execute-in-place.

Pointwise:

• Code runs directly from its storage location.
• In firmware, that storage is commonly ROM or flash.
• The instruction stream does not need to be copied into RAM first.
• XIP solves instruction fetch.
• XIP does not solve writable data.

Diagram:

XIP firmware image

+-----------------------+       fetch       +-----+
| ROM / flash           | ----------------> | CPU |
|                       |                   +-----+
| .text  executable     |
| rodata readable       |
| .data load image      | -- must copy --> RAM before normal C use
| .bss  no bytes stored | -- must zero --> RAM before normal C use
+-----------------------+

Caveats:

• ROM is not writable.
• Flash may be slow.
• Relocation-sensitive code may not work from arbitrary addresses.
• Writable global state still needs RAM or temporary storage.
• .data and .bss must be prepared before normal C code relies on them.

Reference:

• UEFI documents TE image loading behavior for XIP environments: UEFI TE Image Loading.

C code assumes a runtime environment.

Cache-As-RAM can make some C possible earlier:

• A CAR-backed stack can support calls and local variables.
• A tiny CAR-backed heap may support restricted allocation.
• Small temporary globals may work if the linker/startup code places them in the temporary memory model intentionally.
• Permanent C runtime state still needs DRAM migration, .data copying, and .bss clearing.

Before calling ordinary C, assembly startup code usually must:

• set a valid stack pointer;
• satisfy ABI alignment requirements;
• make constants and read-only data addressable;
• copy .data from its load address to its runtime address;
• zero .bss;
• apply relocations if the image is not running where it was linked to run;
• set up minimal exception or fault handling if the platform requires it;
• jump or call into the first C function.

Section layout:

Firmware image in ROM/flash                 Runtime layout in RAM

+----------------------+                    +----------------------+
| .text                | execute in place    | optional .text copy   |
| .rodata              | ----------------->  | optional .rodata copy |
| .data initial bytes  | ---- copy --------> | .data writable        |
| no .bss bytes        | ---- zero --------> | .bss zeroed           |
+----------------------+                    | stack                |
                                            | heap later, maybe    |
                                            +----------------------+

VMA and LMA:

• VMA means virtual memory address, or the address code expects at runtime.
• LMA means load memory address, or where bytes live in the image.
• .data often has an LMA in ROM and a VMA in RAM.
• .bss has a VMA in RAM but usually no stored bytes in the image.

Example invariant:

static int global_counter;

Pointwise:

• The C language says global_counter starts as zero.
• Hardware does not zero it automatically.
• Startup assembly must clear the .bss range.
• Until that happens, normal C assumptions are false.

Reference:

• GNU ld documents VMA/LMA, ROM-loaded .data, and zeroing .bss: GNU ld Output Section LMA.

Not every physical address has the same behavior.

Examples:

• normal DRAM;
• firmware ROM;
• MMIO registers;
• framebuffer memory;
• PCIe device windows;
• reserved platform ranges.

Common memory types:

• WB: write-back, usually normal RAM;
• UC: uncacheable, often MMIO;
• WT: write-through;
• WC: write-combining, often useful for framebuffer-style writes.

Diagram:

Physical address space

+----------------------+  cacheable WB    ordinary DRAM
| RAM                  |
+----------------------+  UC or WP        firmware ROM/flash window
| ROM / flash alias    |
+----------------------+  UC              device registers
| MMIO                 |
+----------------------+  WC or UC        framebuffer
| framebuffer          |
+----------------------+

MTRR:

• MTRRs describe memory types over physical ranges.
• Firmware may configure MTRRs early.
• MTRRs are coarse compared with page-level mappings.

PAT:

• PAT gives page-level memory type control.
• PAT interacts with MTRRs.
• The same physical page must not be mapped with conflicting cacheability.

Caveats:

• A correct physical address with the wrong memory type can still be wrong.
• MMIO must not be treated like ordinary cached RAM.
• A framebuffer may need WC rather than WB.
• Identity and higher-half aliases must agree on memory type.
• Memory type aliasing can cause behavior that looks impossible.

References:

• Linux MTRR documentation: Linux MTRR documentation.
• Linux PAT documentation: Linux PAT documentation.
• Intel documents MTRRs, PAT, and paging memory types in the SDM: Intel SDM.

Goal:

• Force this virtual range:

VA base: 0xffffffff81000000

• To resolve to this physical range:

PA base: 0x0000000000100000

• Then this access:

VA: 0xffffffff81001234

• Should reach:

PA: 0x0000000000101234

Assumptions:

• x86-64 4-level paging.
• 48-bit canonical virtual addresses.
• A 2 MiB page is used for the mapping.
• The virtual base is 2 MiB aligned.
• The physical base is 2 MiB aligned.
• The leaf entry is a page-directory entry with the large-page bit set.

The virtual address is split into indexes:

VA = 0xffffffff81001234

  63                  48 47      39 38      30 29      21 20             0
 +----------------------+----------+----------+----------+---------------+
 | sign extension       | PML4 idx | PDPT idx | PD idx   | 2 MiB offset  |
 +----------------------+----------+----------+----------+---------------+

PML4 index   = 511 = 0x1ff
PDPT index   = 510 = 0x1fe
PD index     =   8 = 0x008
2 MiB offset = 0x1234

The page-table walk:

CR3
 |
 v
PML4 table
  entry[511]  --> PDPT physical address
                    |
                    v
                  PDPT table
                    entry[510] --> PD physical address
                                      |
                                      v
                                    PD table
                                      entry[8] --> PA base 0x00100000
                                                   flags: present, writable,
                                                          executable, PS=1
                                                   page size: 2 MiB

Final address calculation:

physical address = leaf physical base + page offset
physical address = 0x00100000 + 0x1234
physical address = 0x00101234

So:

0xffffffff81001234  --->  0x0000000000101234

The important detail:

• The page-table entry does not store 0xffffffff81000000.
• It stores the physical base 0x00100000 plus flags.
• The virtual address selects which entries to walk.
• The leaf entry supplies the physical frame.
• The low offset bits are carried into the final physical address.

This is conceptual pseudocode, not drop-in firmware code.

map_2m(va_base, pa_base, flags):
    assert va_base is 2 MiB aligned
    assert pa_base is 2 MiB aligned

    pml4_i = bits(va_base, 47, 39)
    pdpt_i = bits(va_base, 38, 30)
    pd_i   = bits(va_base, 29, 21)

    pml4 = physical_address_from_cr3()

    pdpt = ensure_physical_page_table(pml4[pml4_i])
    pd   = ensure_physical_page_table(pdpt[pdpt_i])

    pd[pd_i] = pa_base
             | PRESENT
             | WRITABLE
             | PAGE_SIZE_2M
             | flags

    make_cpu_observe_page_table_update()

Important caveats:

• ensure_physical_page_table must allocate or use real physical memory.
• Page-table pointers stored in entries are physical addresses.
• If modifying active mappings, stale TLB entries must be invalidated.
• If enabling paging for the first time, the current code path must already be mapped.
• Cacheability flags must be consistent with MTRR/PAT memory type rules.

Normal logging usually arrives late.

Before that, debugging may use:

• POST codes;
• serial output after UART setup;
• firmware trace buffers after temporary memory exists;
• JTAG or hardware probes;
• emulator traces;
• QEMU logs;
• GPIO toggles;
• infinite loops at known milestones.

Capability-aware debugging:

No RAM yet       -> registers, POST codes, hardware trace
UART ready       -> serial bytes
Temp RAM ready   -> tiny trace buffer
DRAM ready       -> larger logs
Paging ready     -> virtual-address-aware diagnostics
OS ready         -> normal logging

Main debugging rule:

• Do not design the debug method around a capability that has not been created yet.

• Intel architecture, reset state, control registers, paging, MTRRs, PAT, TLBs, and MSRs: Intel SDM
• Intel runtime microcode update behavior: Runtime Microcode Update
• Linux early and late microcode loading: Linux x86 microcode loader
• UEFI SEC phase and temporary memory: UEFI PI SEC phase
• UEFI XIP image handling: UEFI TE Image Loading
• coreboot boot stages and cache-as-RAM: coreboot architecture
• Linker VMA/LMA, copying .data, and zeroing .bss: GNU ld Output Section LMA
• x86 MTRR behavior in Linux: Linux MTRR documentation
• x86 PAT behavior and memory type aliasing: Linux PAT documentation