So, in Part 1 we decided that the problem comes from a wrongly formed Machine Description (MD) file: the mblock doesn’t have an ‘fwd’ directive to the root node.
And we need to patch the blob, or find a way to change all of this the regular way.
The original OpenSPARC T1 archive from 2006 contains the source code/toolchain for building it, named mdgen and mdlint; it’s located under the hypervisor/src/md directory inside the tar.bz2 file.
Also, legion/src/config/niagara contains the machine description blobs (and what’s important here: one is specially used by the QEMU uniprocessor 1up.*), and the proper conf files used for these blob generations. When we build it using the specific configuration, at the end we can get a 1up-md/1up-hv bin pair which is identical to the one shipped with the original T1 OpenSPARC archive from 2006 (and used by QEMU). The toolchain with the generated files located here it contain some byteswap modifications for building on Linux
Each pair is built from the lexical directives based on four files: two of them describe hypervisor properties (hdesc), and the other two describe the machine description (pdesc). Two of these are common — literally common — for the whole architecture, while the other two describe the specific system (i.e., 1up (uniprocessor) or 32-thread 1g32p).
so, we use pair
- 1up.hdesc/pdesc
- common.hdesc/pdesc
Now, let’s focus on the ‘lost’ memory in the MD, and let’s look at common.pdesc:
node root root {
<...>
max-#tsb-entries = 0x2;
reset-reason = 0; // power-on
fwd -> platform_data;
fwd -> vdev;
}
node platform platform_data {
We have a fwd directive for platform_data and vdev, but not for memory here But per 5.5.4 Memory node config on (page 7)[https://sun4v.github.io/ARChive/FWARC/2005/115/final.materials/md-bindings.pdf] At the same time we have pre-defined a mblock0 in the configuration:
node mblock p0_memlist0 {
base = 0x0000000080000000;
size = 256M;
}
The mblock directive is literally ‘hanged in the air,’ without any real references to any other pdesc/hdesc file(s). Let’s try to describe the memory node as the parent for the p0_memlist0 mblock node.
node mblock p0_memlist0 {
base = 0x0000000080000000;
size = 256M;
}
+node memory memory {
+ fwd -> p0_memlist0;
+ back -> root;
+}
And push the fwd directive for the memory node from root, as it is required by root per md binding
node root root {
<...>
fwd -> platform_data;
fwd -> vdev;
+ fwd -> memory;
}
We built the blob binaries, and we passed the mblock allocation in them!
Loaded modules: [ unix krtld genunix ]
[0]> ::bp md_alloc_scan_dag
[0]> ::bp panic
[0]> :c
kmdb: stop at md_alloc_scan_dag
kmdb: target stopped at:
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> <o2/s
0x10d1548: mblock
[0]> n_mblocks/X
n_mblocks:
n_mblocks: 0
[0]> :c
kmdb: stop at md_alloc_scan_dag
kmdb: target stopped at:
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> n_mblocks/X
n_mblocks:
n_mblocks: 1
… and then immidiately failing on cpu node
[0]> <o2/s
0x10ce3a8: cpu
[0]> :c
ERROR: Last Trap: Fast Data Access MMU Miss
So, we need to add the CPU node also, i.e:
add to common.pdesc
node cpus cpus {
fwd -> cpu0;
back -> root;
}
and adding the fwd directive after memory node too:
{
fwd -> vdev;
fwd -> memory;
+ fwd -> cpus;
}
Next, it passed the CPU node properties, and it fails on the ‘platform’ node.
md_alloc_scan_dag: save %sp, -0xb0, %sp
[0]> <o2/s
0x10ce848: platform
[0]> :c
ERROR: Last Trap: Fast Data Access MMU Miss
Next thing, we walk to the platform and set up an md_get_prop_val breakpoint, because we need to know the specific variable that leads to a panic in md_alloc_scan_dag:
[0]> <o2/s
0x10ce848: platform
[0]> ::bp md_alloc_scan_dag
[0]> ::bp md_get_prop_val
[0]> :c
kmdb: stop at md_get_prop_val
kmdb: target stopped at:
md_get_prop_val:save %sp, -0xb0, %sp
[0]> <o2/s
0x10ce9b0: domaining-enabled
[0]> :c
kmdb: stop at md_get_prop_val
kmdb: target stopped at:
md_get_prop_val:save %sp, -0xb0, %sp
[0]> <o2/s
0x10ce580: mmu-page-size-list
[0]>
it’s points us to usr/src/uts/sun4v/os/fillsysinfo.c
static uint64_t
get_cpu_pagesizes(md_t *mdp, mde_cookie_t cpu_node_cookie)
{
uint64_t mmu_page_size_list;
if (md_get_prop_val(mdp, cpu_node_cookie, "mmu-page-size-list",
&mmu_page_size_list))
mmu_page_size_list = 0;
if (mmu_page_size_list == 0 || mmu_page_size_list > MAX_PAGESIZE_MASK)
cmn_err(CE_PANIC, "Incorrect 0x%lx pagesize mask returned"
"by MD", mmu_page_size_list);
return (mmu_page_size_list);
}
In a similar way, mmu-#va-bits was added to the node (after a bit of walking through breakpoints) — the routine cyclically walks across memory/cpu/platform/cache/exec_unit, and then walks around different values across md_get_prop_val:
on common.pdesc - both parameters is cpu node properties
+ mmu-page-size-list = 0x9; \
+ mmu-#va-bits = 0x40; \
In between these two mmu- properties, due to the platform node’s unavailability, we had a cache node failure the same way as with mnode and cpu, and we need to declare it properly
And then, kernel initialized properly and “normally” paniced:
Loaded modules: [ unix krtld genunix ]
[0]> :c
panic - kernel: bop_alloc_chunk failed
Program terminated
at the end we have very simple directives for md nodes in 1up.conf:
--- a/1up.pdesc
+++ b/1up.pdesc
@@ -35,3 +35,17 @@
base = 0x0000000080000000;
size = 256M;
}
+
+node memory memory {
+ fwd -> p0_memlist0;
+ back -> root;
+}
+
+node cpus cpus {
+ fwd -> cpu0;
+ back -> root;
+}
+
+node cache cache {
+ back -> cpu0;
+}
and folliwing fwd/back directives +mmu-# bits on common.pdesc
--- a/common.pdesc
+++ b/common.pdesc
@@ -34,9 +34,12 @@
isalist = "sparcv9","sparcv8plus","sparcv8","sparcv8-fsmuld","sparcv7","sparc"; \
compatible = { "SUNW,UltraSPARC-T1", "SUNW,sun4v-cpu", "sun4v" }; \
max-#tsb-entries = 0x2; \
+ mmu-page-size-list = 0x9; \
+ mmu-#va-bits = 0x40; \
+ back -> cpus; \
+ fwd -> cache; \
}
-
node options XXoptions {
input-device = "ttya";
output-device = "ttya";
@@ -66,6 +69,8 @@
reset-reason = 0; // power-on
fwd -> platform_data;
fwd -> vdev;
+ fwd -> memory;
+ fwd -> cpus;
}
