OXYGENHUB
◆ main enhanced-x16-emulator / docs / code-map-width-propagation.md
18.1 KiB · Markdown · last changed chore(): initial commit 25 days ago renderedsource blame history raw
9120e874 Andy McCall 1
# code_map: where the 65C816 operand widths are still a guess
2
3
Status: **known limitation, deliberately not "fixed"**. This note exists so the
4
next person to look at it does not mistake it for an oversight, and so that the
5
tempting fix is on record as the wrong one.
6
7
## The gap
8
9
`cm_propagate()` in `src/code_map.c` predicts the effective processor status of
10
the next instruction from the current opcode. Exactly five instructions change
11
the 65C816 register widths:
12
13
| Instruction | Modelled? | Why |
14
|---|---|---|
15
| `REP #imm` | its status bits, exactly | the bits to clear are in the operand |
16
| `SEP #imm` | its status bits, exactly | the bits to set are in the operand |
17
| `XCE`      | only if C and E are right | swaps carry and the emulation flag |
18
| `PLP`      | **no** | restores a status byte pulled off the stack |
19
| `RTI`      | **no** | restores the status the interrupt pushed |
20
21
`XCE` is exact only when **both** of its inputs are right, and neither is
22
guaranteed.
23
24
*The carry* is not tracked through any data-dependent change. `CLC` and `SEC`
25
are modelled, and `REP`/`SEP`/`XCE` naturally update it as part of what they do,
26
but `ADC`, `SBC`, the compares, and the shifts and rotates all move the carry
27
without the estimate noticing. So an `LSR A` that really clears the carry,
28
followed by `XCE`, makes the model predict a switch into emulation mode that
29
does not happen, forcing the widths to 8-bit for every following line that has
30
no anchor of its own. That needs no `PLP` or `RTI` at all;
31
`tests/test_code_map.c` pins it under `KNOWN LIMITATION`.
32
33
*The emulation flag* is not recorded by anchors at all — they store the
34
effective status byte but **not** `E`. The running `E` is seeded from the live
35
`regs.e`, so landing on an anchor re-syncs that line's widths but not `E`, and
36
propagating past the anchor folds the stale `E` back in
37
(`if (e) status |= INDEX|MEMORY`) — which can mis-size the next unanchored line,
38
wherever the stale `E` and the real one imply different widths.
39
40
A bad `E` is not unrecoverable, though. `XCE` writes `E` from the estimate's
41
carry, which is exactly what the real instruction does, so an `XCE` decoded with
42
an *accurate* carry — one taken from an anchor, or set explicitly by a `CLC`/
43
`SEC` the estimate did track — puts `E` back in step. What is not promised is
44
that such a sequence appears: an ordinary anchor does not repair `E`, so nothing
45
guarantees recovery.
46
47
Most X16 software sets `E` once during startup and leaves it alone, which is why
48
this is latent rather than routine. What defeats that reassurance is not a
49
deliberate mode switch by the guest: it is enough that the walk *crosses* an
50
`XCE` — the one from startup, one on a path the program never takes, or a `$FB`
51
byte that is really data — with a carry the estimate did not track. `run_e` is
52
written in exactly one place, `case 0xFB`, so without an `XCE` in the walked
53
stream the estimate keeps the seeded `E`.
54
55
For `PLP` and `RTI` the file leaves the running estimate untouched. Operand
56
widths for lines decoded after one of them can therefore be wrong, and with
57
them the instruction boundaries that follow.
58
59
## Why it is not fixed
60
61
The value is not recoverable by this analysis, which sees only the instruction
62
stream. What `PLP` restores
63
depends on what was pushed, which depends on the path taken to get there; what
64
`RTI` restores is the status at the moment an interrupt was taken, which is not
65
in the code at all.
66
67
Two avenues were considered and rejected:
68
69
- **Walk back from `PLP` to the matching `PHP`.** Needs the routine boundaries
70
  and a guarantee that no other stack traffic intervened. Neither is available,
71
  and a wrong pairing produces a confident wrong width rather than an admitted
72
  unknown.
73
- **Record the status pushed at each interrupt and consult it for `RTI`.** The
74
  emulator does see every interrupt, and `src/cpu/irq_ctx.c` already tracks
75
  entries and exits — but it records the vector, the interrupted PC and the
76
  stack pointer, not the pushed status. Adding it is easy; *using* it is not.
77
  `cm_propagate()` runs over arbitrary addresses while drawing a disassembly
78
  window, and has no way to know which interrupt frame a particular `RTI` on
79
  screen will return through. The innermost live frame is not necessarily the
80
  right one. So the result would still be a guess, just a more expensive one.
81
82
Both need a notion of "known vs unknown" per status bit, so that a failed
83
attempt reports uncertainty instead of substituting another guess. That does not
84
exist, and adding it is a larger change than the defect warrants.
85
86
## Why the gap is bounded
87
88
It costs nothing unless *all* of these hold at once:
89
90
- the machine is a 65C816 (on the 65C02 operand widths never vary, so the
91
  estimate cannot change a length);
92
- the line being drawn has no currently believed anchor — usually because that
93
  code has never executed, but also if its anchor was evicted with its bank
94
  context (the cache is capped, see `CM_MAX_CONTEXTS`) or is no longer believed
95
  because the opcode byte under it changed;
96
- the status the estimate carries into that line is not the status the code
97
  there really runs with.
98
99
That third condition is the general one, and it is *not* only the unmodelled
100
instructions. It happens whenever:
101
102
- one of the unmodelled cases lies between the last anchor and the line — a
103
  `PLP`, an `RTI`, a stale carry feeding an `XCE`, or a stale `E`;
104
- **the walk crossed a branch or a call.** Propagation is address-linear: it
105
  applies whatever opcode sits at the next *address*. Given
106
  `BRA +2 / REP #$20 / LDA #`, execution jumps over the `REP` and the `LDA` is
107
  two bytes, while the walk applies the `REP` and sizes it three. The same goes
108
  for a `JSR` whose callee changes `M`/`X` and returns. `REP` and `SEP` are
109
  applied exactly *as instructions*; that does not make the resulting estimate
110
  right for a path the program never took;
111
- **the walk started somewhere the seed does not describe.**
112
  `code_map_disasm_forward` seeds from `regs.status`/`regs.e`, which is the
113
  state at the *current PC*, not at `start`. That is the DAP `disassemble`
114
  path, which takes an arbitrary address, so disassembling never-executed code
115
  that runs at a different width is wrong from the first line — with no anchor
116
  and no unmodelled instruction anywhere in sight.
117
118
Note what is **not** on that list: native mode. The real CPU in emulation mode
119
does force 8-bit widths and is always right — but the fold-in here
120
(`if (e) status |= INDEX|MEMORY`) uses the *estimated* `E`, not the machine's,
121
and no anchor repairs that estimate. `run_e` is seeded once per forward run and
122
thereafter written only by `XCE`, from the estimate's own carry; an anchor
123
overrides the status but never the `E`. Once the estimate diverges it will size
124
operands as though native while the machine is really in emulation, until an
125
`XCE` with an accurate carry happens to put it back. A test pins exactly that.
126
127
The mechanism that would recover the value is largely the same mechanism that
128
makes the gap irrelevant: code that has executed has anchors. The correspondence
129
is not perfect — a handler paused before its own, not-yet-executed `RTI` does
130
have a live interrupt frame — but associating a particular on-screen `RTI` with
131
a particular frame is exactly the part that is not solved.
132
133
Every line this can affect already reports `recorded == false`, so a UI can say
134
so. **The strength of the recovery differs by which input was wrong**, and none
135
of it is unconditional:
136
137
- a wrong **status** estimate can be corrected once an accurate, believed anchor
138
  is reached. Three caveats, all real: a same-opcode stale anchor is still
139
  believed and hands back its *old* status; there is no guarantee an accurate
140
  anchor exists ahead at all; and even an accurate one only fixes its **own**
141
  line — an anchor on a `PLP` or `RTI` supplies the status going *into* it,
142
  while the byte it restores is still unmodelled, so the next line can be wrong
143
  immediately;
144
- a wrong **E** is weaker again, since anchors do not store `E` at all. An
145
  anchor fixes the width of its own line, and propagation past it folds the
146
  stale `E` straight back in, which can mis-size the next unanchored line
147
  wherever the two imply different widths. It is recoverable in principle — an
148
  `XCE` decoded with an accurate carry writes a correct `E` — but no ordinary
149
  anchor does it, so nothing guarantees it happens.
150
151
In both cases the decode is not allowed to swallow a recorded start (unless it
152
is itself recorded — see the overlap policy in `cm_fill`), so a wrong guess
153
cannot silently run past hard evidence about *boundaries*, even where it keeps
154
getting *widths* wrong.
155
156
## What is pinned by tests
157
158
`tests/test_code_map.c` covers the bound rather than the missing value, for all
159
three cases:
160
161
- `PLP`: "reports a line decoded from a guessed width as unrecorded",
162
  "re-syncs onto the next anchor after a bad width guess", "recovers the real
163
  width from the anchor it landed on".
164
- `RTI`: "leaves the width estimate alone across RTI instead of guessing",
165
  "reports a line decoded across RTI as a guess, not as known", "reaches the
166
  anchor after RTI and recovers the real width".
167
- `XCE` with a stale carry: "KNOWN LIMITATION: a stale carry into XCE
168
  mispredicts the widths", "an anchor recovers the width of its own line after a
169
  bad XCE", and — pinning the *weaker* bound described above — "KNOWN
170
  LIMITATION: past that anchor a stale E mis-sizes again".
171
172
The first `RTI` check is deliberately written to fail if someone substitutes a
173
plausible width without also building the known/unknown machinery. It is a
174
tripwire, not a claim that the current behaviour is ideal — if the widths are
175
ever tracked properly, that check should be replaced, not deleted quietly.
176
177
---
178
179
# Related: a stale anchor that kept its opcode byte
180
181
**Known limitation, pinned by tests, not fixed.**
182
183
An anchor is believed while the opcode byte it recorded is still in memory. That
184
one-byte check catches overlay loads, a second program over the first, and
185
self-modifying code without needing a hook on every write — but replacement code
186
that happens to repeat the same opcode byte at the same address keeps the old
187
anchor, and with it the old recorded **status**.
188
189
On a 65C816 that stale status can imply a different operand width, so the line
190
decodes wider than the new code really is. Because a recorded line is exempt
191
from the interior-anchor clamp (see the overlap policy in `cm_fill`), it can
192
then swallow a genuinely fresh anchor inside it:
193
194
```
195
old: $8000  A9 xx xx   LDA #$xxxx   recorded with 16-bit A -> 3 bytes
196
new: $8000  A9 xx      LDA #$xx     8-bit A, and $8002 freshly executed
197
```
198
199
The opcode is still `$A9`, so the stale anchor survives and wins.
200
201
**It cuts both ways.** The example above decodes too *wide* and swallows a fresh
202
anchor, which is the loud failure. The opposite is quieter and just as
203
reachable: an anchor recorded when the accumulator was 8 bits sizes its `LDA #`
204
at two bytes, so if the replacement code is about to run with a 16-bit
205
accumulator the row stops a byte early and the *next* row starts inside the real
206
instruction's operand. Nothing is swallowed and the rows still tile perfectly,
207
so there is no visible seam — the disassembly is simply misaligned from there
208
until an accurate anchor is reached. Both directions are pinned by tests.
209
210
Why it is accepted rather than fixed:
211
212
- It needs the replacement code to repeat the same opcode byte at the same
213
  address, which the one-byte check then cannot see through.
214
- The damage is self-correcting *if* an accurate anchor is reached that this
215
  line — or another stale one — has not swallowed. That is not guaranteed: right
216
  after an overlay load the new code typically has no valid anchors at all, and
217
  the stale status then propagates to the end of the window. (An anchor whose
218
  opcode no longer matches re-syncs nothing; it is rejected by `cm_anchor_ok()`
219
  and simply stops being evidence.)
220
- The alternative — always clamping — costs the far more common case, the
221
  `.byte $2C` skip idiom, where two overlapping starts are both genuinely real
222
  and clamping renders a whole known instruction as a fragment.
223
224
**Scope: the wrong-*width* flavour is 65C816-only; the swallowing is not.** On a
225
65C02 operand widths never vary, so a stale anchor cannot decode wider than the
226
new code. But it does not have to: a stale anchor whose opcode byte survived is
227
still believed, is still exempt from the interior-anchor clamp, and can still
228
cover a fresh start inside it. An old `BIT abs` at `$8000` overlaid by new code
229
that also begins `$2C` will swallow a genuine instruction start at `$8001` on
230
any CPU. `tests/test_code_map.c` covers both flavours.
231
232
**The stale case is not distinguishable from a legitimate one.** The recorded
233
state above is byte-for-byte identical to a routine that genuinely executed at
234
two different accumulator widths — an anchor at `$8000` with a 16-bit status and
235
an anchor at `$8002` with an 8-bit one, over memory starting `$A9`. In that
236
benign case emitting the 3-byte `LDA` is exactly correct.
237
238
That rules out the obvious fix. A per-address recording *epoch* ("prefer the
239
newer anchor") does not resolve this: applied to the benign case it would clamp
240
a legitimately recorded 3-byte `LDA` back to 2 bytes purely because the other
241
width ran more recently, reintroducing the defect this branch fixed. It trades
242
one wrong answer for another.
243
244
Widening the staleness check — storing two or more bytes per anchor instead of
245
one — is also not the clean win it looks like. It would catch *this* example
246
(the operand bytes differ), but it is the same class of heuristic, so it only
247
narrows the window rather than closing it; replacement code with identical bytes
248
stays invisible either way. It also misfires on one-byte instructions, which are
249
extremely common (`NOP`, `INX`, `RTS`, …): for those the second byte compared is
250
not part of the instruction at all but the *next* instruction's opcode, so
251
changing the following instruction would invalidate a perfectly good anchor.
252
Storing the instruction's real length would avoid that — it does not need a full
253
`disasm()` call, an opcode length table plus the small M/X-dependent set would
254
do — but it is per-instruction classification plus another byte of storage per
255
address, and it still does not close the identical-bytes hole.
256
257
What does resolve it is invalidation keyed on the **write** that replaced the
258
code, because that is the event which actually differs between the two cases.
259
That does not necessarily mean a hook on every CPU write: the dominant source of
260
stale anchors is a host-side load, and those already funnel through the KERNAL
261
LOAD/MACPTR interception in `main.c` (the bytes themselves land in `ieee.c`), so
262
even a conservative `code_map_reset()` when an intercepted load completes would
263
be sound for that case and needs no range or bank bookkeeping. Targeted range
264
invalidation keeps more coverage but is more work. Self-modifying code and
265
in-guest decompressors would still need the general write path in `memory.c`.
266
Either way it is a change to the recording side rather than the disassembly
267
side.
268
269
(It would *not* help the PLP/RTI width gap described earlier in this document.
270
That one is about status bytes restored from the stack, which is unrelated to
271
code being replaced.)
272
273
`tests/test_code_map.c` pins the current behaviour under checks labelled
274
`KNOWN LIMITATION`, so it is visible rather than folklore. If it is ever fixed
275
those checks will fail — update them, do not delete them.
276
277
One caveat on that tripwire: it fires for a fix *inside* code_map (a wider
278
staleness check, a stored instruction length, an epoch), but **not** for the
279
write-invalidation fix recommended above. The unit test models the overlay with
280
its own `poke()`, which writes the harness's memory arrays directly, and
281
code_map exposes no write-notification entry point for it to call. Anyone
282
adding one should route `poke()` through it, or these checks will stay green
283
while this note quietly becomes a description of behaviour that no longer
284
exists.
285
286
---
287
288
# Related: a window-straddling instruction is decoded through one bank
289
290
**Not fixed. Not covered by a passing test.** Found while adding banking
291
coverage for `cm_x16bank_for()`.
292
293
`cm_fill()` reads the raw bytes of an instruction one at a time, each through
294
the window backing that byte's own address, so an instruction straddling the
295
`$BFFF`/`$C000` boundary shows the right bytes. `cm_decode()` does not: it hands
296
`disasm()` a *single* `x16Bank`, derived from the instruction's **start**
297
address, and `disasm()` uses that one bank for every operand read.
298
299
So for a `STA $1234` whose opcode is at `$BFFF` in RAM bank 5 and whose operand
300
lives at `$C000`/`$C001` in ROM bank 2, the operand is read as *ROM bank 5*.
301
With the test fixture that means `bytes[]` correctly reads `8D 34 12` while
302
`text` reads `sta $eaea` and `eff_addr` is `$EAEA`. This is a *debugger-side*
303
mis-read only: the CPU's own fetch goes through the real banking hardware and
304
is correct, so the program runs fine — it is the disassembly shown to the user
305
that is wrong. The same wrong bytes come back from the host-side read on a real
306
machine's memory image (`ROM[rambank * 16384 + …]`; for a bank number ≥ 32 it is
307
open bus, or the cartridge's bytes when one supplies that bank).
308
309
Note the width propagation had the *same* bug and **is** fixed: `cm_propagate()`
310
now reads a REP/SEP operand through the window backing the operand's own
311
address. That one is code_map's own read, so fixing it touched nothing shared.
312
313
The test in `tests/test_code_map.c` asserts the displayed **bytes** only, and
314
says so in a comment; it does not assert the text, because the text is wrong.
315
316
Fixing it properly means giving `disasm()` a per-byte, window-aware reader (or
317
having `cm_decode()` detect a straddle and re-render the operand from the bytes
318
`cm_fill()` already read correctly). `disasm()` is shared with the classic SDL
319
debugger and the DAP server, so that is a wider change than the code_map
320
defects this work covers.
321
322
Scope: one instruction per window boundary, and only when code is laid across
323
one. There are two such boundaries, not one — `$9FFF`/`$A000` as well as
324
`$BFFF`/`$C000` — since an opcode below `$A000` follows the live bank while its
325
operand in the RAM window needs the caller's RAM bank. Rare, but it renders as
326
confidently wrong text rather than as an admitted unknown, which is the worst
327
failure mode this file has.