Continuing from Part 1.
Let’s look at what this looks like in practice, and where QEMU’s handling of it actually falls over.
The latest good breakpoint was alloc_page_freelists+0x54 (I spent some time catching it):
[0]> alloc_page_freelists+0x54::bp
[0]> :c
kmdb: stop at alloc_page_freelists+0x54
kmdb: target stopped at:
alloc_page_freelists+0x54: stx %i0, [%l6 + %l7]
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
[0]> kmem64_szc/X
kmem64_szc:
kmem64_szc: 5 - <this is 256M page for this specific allocation >
[0]> kmem64_base/J
kmem64_base:
kmem64_base: 70000000000
[0]> kmem64_aligned_end/J
kmem64_aligned_end:
kmem64_aligned_end: 70010000000
[0]> 70000000000/J
0x70000000000: 0
[0]> 7000000F000/J
0x7000000f000: 0
[0]> 7000000FF00/J
0x7000000ff00: 0
[0]> 7000000FFF0/J
0x7000000fff0: 0
[0]> 7000000FFFE/J -<hang where we try to look the code out of 64K>
Here, kmem64_szc is 5, which corresponds directly to TTE256M from the table in Part 1 - confirming that the kernel is indeed trying to map the kmem64 segment using 256M pages, driven by the broken_md_flag logic, but it can’t do that.
The hang happens exactly at the boundary where we step outside the 64K range - right where QEMU would need to service a translation for the 256M-sized mapping. This lines up with the suspicion from Part 1: QEMU’s MMU emulation may not correctly handle the 256M page size that the guest kernel enables as a side effect of the missing domaining-enabled property. The guest isn’t doing anything unusual - it’s following the exact logic in niagara.c - but QEMU’s handling of that particular [sz] encoding looks like the likely point of failure, producing the endless stream of Hyperprivileged Trap Instruction / Data Access MMU Miss entries seen in the log at the start of this investigation. This remains a suspicion rather than a confirmed root cause.
When we tried to suppress the pagesize to 0x9 (8K+4M), like so:
[0]> ::bp alloc_kmem64
[0]> :c
[0]> mmu_exported_pagesize_mask/W b
the kernel booted successfully, but without 256MB pages.
I then started looking at the QEMU code. The TTE page-size field is defined in QEMU’s /target/sparc/cpu.h as:
#define TTE_PGSIZE(tte) (((tte) >> 61) & 3ULL)
and the sun4v_tte_to_sun4u() conversion inside QEMU’s target/sparc/ldst_helper.c only preserves 2 bits for the page size:
sun4u_tte |= (sun4v_tte & 3ULL) << 61; /* TTE_PGSIZE */
It seems the logic only preserves two bits (61/62), giving a value of 0-3, when the highest bit - bit 48 (szh) - is also supposed to be part of it, which makes the operation a bit more involved: we need to reconstruct it as
| szh[48] | szl[62] | szl[61] | |—|—|—|
to build the table described below in the UltraSPARC T1 Supplement draft:
to hold all the properties in our case.
Interestingly, when I tried to figure out when this extension was introduced into the CPU spec, I found the Panther/USIV CPU listed as the first to support it, then T1 supported the same TTE pagesize bits, but T2 changed the format:
| Document | Where it lands |
|---|---|
| JPS1 Commonality, Working Draft 1.0.5, TABLE F-1, p.441 | The state before the extension: 2-bit size, Data<49:47> “Reserved, read as 0” (bit 48 included). Joint Sun+Fujitsu, so it covers US-III and SPARC64 V. |
| Panther Impl. Supplement (UltraSPARC IV+), Preliminary Draft 17 Mar 2006, TABLE F-1-4, p.337 | The citation to use. States plainly: “Bit 48 is the most significant bit of the page size and is concatenated with bits <62:61>”, with the full 000-101 encoding. |
| UltraSPARC T1 Supplement to the UltraSPARC Architecture 2005, Draft D2.1, sec. 13.1.2, TABLE 13-1, p.182 | Same bit in T1’s sun4u-format TTE, named szl/szh. |
| UltraSPARC T2 Supplement, TABLE 12-4, p.107 | T2 dropped the sun4u format entirely: one contiguous 4-bit field, and only 4 sizes (512K and 32M Reserved). |
When I started prototyping the fix with Claude, it took five iterations to land on this patch and send it to the qemu-devel community. Each of the first four had something wrong: they used the wrong bitfields for storing the [szh] bit - the first time we built it on unused bit 42, which is out of UltraSparc spec :), and it even worked, sort of; or used suboptimal extra operations to store this bit in the TTE. All of these worked, but…
With it applied, the guest started to boot:
[0]> alloc_page_freelists+0x54::bp
[0]> :c
kmdb: stop at alloc_page_freelists+0x54
kmdb: target stopped at:
alloc_page_freelists+0x54: stx %i0, [%l6 + %l7]
[0]> kmem64_szc/X
kmem64_szc:
kmem64_szc: 5
[0]> kmem64_base/J
kmem64_base:
kmem64_base: 70000000000
[0]> kmem64_aligned_end/J
kmem64_aligned_end:
kmem64_aligned_end: 70010000000
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
[0]> 7000000FFFE/J <- it's survived!
0x7000000fffe: 0
[0]> 70000010000/J
0x70000010000: 0
[0]> 70000100000/J <- also survived
0x70000100000: 0
[0]> :c
kmdb: stop at alloc_page_freelists+0x54
kmdb: target stopped at:
alloc_page_freelists+0x54: stx %i0, [%l6 + %l7] - <then it stopped the second time
[0]> 70000100000/J
0x70000100000: 0
[0]> mmu_exported_pagesize_mask/X
mmu_exported_pagesize_mask:
mmu_exported_pagesize_mask: 2b
[0]> :c
SunOS Release 5.11 Version snv_151a 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
And here’s how it looks under the QEMU monitor:
(qemu) info tlb
(qemu) info tlb
MMU contexts: Primary: 0, Secondary: 0
DMMU Tag Access: 180a000, TSB Tag Target: 0
DMMU dump
[00] VA: 70000000000, PA: d0000000, 256M, priv, RW, unlocked, ie no, ctx 0 local <<<<<-- there's the first 256M chunk, visible to QEMU!
[01] VA: ffffffffffffc000, PA: 801ee000, 8k, priv, RW, unlocked, ie no, ctx 0 local
[02] VA: ffffffffff300000, PA: f1f0c000, 8k, priv, RW, unlocked, ie no, ctx 0 local
[03] VA: fffffffcc000a000, PA: f1394000, 8k, priv, RW, unlocked, ie no, ctx 0 local
[04] VA: 1930000, PA: ff530000, 8k, priv, RW, unlocked, ie no, ctx 0 local
