Same Bug, Two Models: One Wasted a Day, One Found the Silicon Limit

A note before this gets going: this is my personal account of debugging two bugs that rode into upstream Zephyr and nRF Connect SDK, written in a personal capacity. It’s not a statement from Nordic Semiconductor. The credit, the mistakes, and the opinions are mine — and one of the best moves in the story was made by a colleague, not me.

The one-line version

I fixed two real Wi-Fi bugs on Nordic silicon with AI coding agents — a silent DMA corruption on nRF91/nRF54H, and a WPA3 connection failure on nRF54L that turned out to be a 10× crypto regression. “Test your AI’s fix on hardware” is not the lesson — I did, and in Case 1 a wrong fix passed on the board anyway, green boot and all. The lesson is sharper and less comfortable: which model you point at a hard bug decides whether you lose a day to confident fiction or find a silicon limit in minutes — and the thing that tells a real fix from a plausible one is never the build or the boot, it’s an orthogonal instrument: a git bisect, a .config diff, a sniffer capture, a profiler.

That’s the whole post, really. The interesting part is which parts of real firmware debugging an agent is genuinely good at, which parts it will lie to you about with total confidence, why the cheap model is the expensive one, and how you split the work so the lie gets caught in minutes instead of days.

I ran this across the tools I actually use day to day: Cursor (both its Auto model picker and Opus explicitly) and Claude Code on Opus 4.8. No custom MCP server, no Nordic-specific agent plumbing — just the agents, a shell, west, a J-Link, and a serial port. Everything below is public: the fixes are merged upstream and linked at the end.

Case 1: Boot_sig check failed … Actual: 0x17

The symptom was a Wi-Fi chip that wouldn’t boot:

[00:00:01.090] <err> wifi_nrf: nrf_wifi_hal_fw_chk_boot: Boot_sig check
  failed for RPU(0), Expected: 0x5A5A5A5A, Actual: 0x17

nRF9151 DK + nRF7002 shield. The nRF70’s firmware patch download was corrupting RPU memory, and the boot signature check only caught the damage after the fact. That word “signature” sends most people toward crypto, keys, a bad firmware image — all red herrings. Experience pointed me at the SPI bus almost immediately; the hard part was never which subsystem, it was proving the exact mechanism — a driver refactor, a DMA controller limit, a bounce region — across two SoCs that turned out to fail for two different reasons.

The agent’s confident, wrong first act

I started in Cursor on Auto — let the tool pick the model — and asked it to find the root cause. It did what a plausible senior engineer might do under time pressure: it built an elaborate, specific, and completely wrong theory. Three “regressions,” no less, all in pinctrl:

• a removed pinctrl_apply_state(..., PINCTRL_STATE_SLEEP) at driver init,
• nrfy_spim_enable()/disable() toggling on every GPIO chip-select edge,
• a dropped SCK CPOL idle write in the refactored configure() path.

It committed a fix. It even committed a workaround — CONFIG_SPI_NRFX_SPIM_LEGACY=y to fall back to the pre-refactor driver — and reported that it “worked on hardware.” This is the failure mode that matters, because it doesn’t look like failure. It looks like success. A detailed root cause, a patch, a green boot. The pinctrl story was internally coherent, well-written, and had nothing to do with the actual bug.

The bench clue that broke the spell

The thing that cracked it didn’t come from the model or from a better prompt. A colleague ran an experiment on the bench: drop MAX_PATCH_CHUNK_SIZE from 8192 to 4096 and the boot passes. He also pointed out the pinctrl of the working and failing SoCs was byte-for-byte identical.

That single empirical fact was incompatible with the entire pinctrl theory. Chunk size has nothing to do with pin muxing. The agent hadn’t been reasoning badly so much as reasoning in a sealed room, confidently, with no ground truth pushing back. The bench was the ground truth.

At that point I stopped letting Auto drive and switched Cursor to Opus explicitly, with one question: why does main need the smaller chunk size but v3.4 doesn’t?

The connection the strong model made that I hadn’t

Here’s the honest part, and it’s the reason I’m not writing a smug “AI is dumb” post. I knew about the EasyDMA MAXCNT limit — it’s not obscure. What I hadn’t done was connect it to this bug, under a misleading symptom, behind a driver refactor. That’s the more common and more useful kind of AI assist: not teaching you a fact you never had, but making the link you didn’t make under pressure. Fed the bench clue and a sharper question, Opus walked it straight out of the code and the devicetree:

• nRF91’s SPIM3 EasyDMA MAXCNT is 13 bits — 8191 bytes max per transfer (easydma-maxcnt-bits = <0xd> in the DTS). An 8192-byte patch chunk is one byte over the hardware limit.
• The old monolithic SPI driver capped each transfer to that limit and let the event handler walk the remainder. The refactored driver on main dropped that cap — it handed the full chunk to the common layer, which rejects anything over MAXCNT with -EINVAL.
• And the nRF70 firmware-write path was void end to end — it discarded the -EINVAL and marched on. So the patch download “succeeded,” the RPU got garbage, and the only evidence was a boot-signature mismatch three steps downstream.
• nRF5340 was never affected: its EasyDMA MAXCNT is 16 bits (65535), so an 8192 chunk always fit in one transfer. Same code, no symptom — which is exactly why it looked like a nRF91-specific mystery.

  nRF91 SPIM3 EasyDMA MAXCNT = 13 bits  ->  max 8191 bytes per transfer
  nRF70 patch chunk = 8192 bytes        ->  1 byte over the limit
  --------------------------------------------------------------------
  old driver (monolithic):  caps to 8191 and loops  ->  works (8191 + 1)
  new driver (refactored):  passes 8192 through     ->  -EINVAL
  --------------------------------------------------------------------
  -EINVAL swallowed by the void write path  ->  corrupt RPU  ->  Boot_sig 0x17

The fix is boring, which is the point: cap chunk_len to the controller limit and let the existing loop split oversized buffers. Seven lines. That’s zephyr#110621.

Same symptom, a second root cause

Then it got better. The nRF91 fix worked — and the nRF54H20 showed the exact same Boot_sig 0x17 with a completely different cause. Its EasyDMA MAXCNT is 15 bits (32767), so 8192 fits fine; the MAXCNT cap never engages. This time the culprit was the DMM bounce region: the SPIM routes transfers through a shared 4 KB cpuapp_dma_region, and a 4096-byte bounce buffer can’t be carved out of a 4 KB pool that the console UART is also using — so dmm_buffer_alloc() returns -ENOMEM, swallowed by the same void path.

The moment that made this fast: earlier we’d added one LOG_ERR line at the bus shim so a failed RPU write would stop being silent. The next boot said it out loud:

<err> wifi_nrf: zep_shim_qspi_cpy_to: write to 0x28c000 (4096 bytes) failed: -12

-12 is -ENOMEM. Diagnosis in one line of console output, because we’d made the silent error loud. Same symptom, two independent bugs, across two SoCs — and the thing that told them apart was the bench, not the model.

  Boot_sig 0x17  =  one symptom, two unrelated root causes
  ----------------------------------------------------------------
  nRF91     8192 > MAXCNT limit (8191)                ->  -EINVAL
  nRF54H20  8192 fits MAXCNT, but not the DMA bounce  ->  -ENOMEM
  ----------------------------------------------------------------
  both errors swallowed by the void write path  ->  the same symptom

Case 2: “Wrong password” — except the password was right

Different tool, same shape — this one in Claude Code on Opus 4.8, on an nRF54LM20 (Cortex-M33) running a Wi-Fi Alliance QuickTrack certification suite.

The symptom was a WPA3-SAE network that simply wouldn’t connect, and the logs were a cascade of confident, contradictory accusations:

<err> wpa_supp: WPA: Failed to generate EAPOL-Key version 1 key_mgmt 0x2 MIC
<inf> wpa_supp: WPA: 4-Way Handshake failed - pre-shared key may be incorrect
      Connection request failed (Wrong password/2)
<inf> wpa_supp: CTRL-EVENT-ASSOC-REJECT ... status_code=53

Every one of those lines is misleading. The password was correct and the key management was fine — these are generic supplicant failure messages that fire whenever the 4-way handshake doesn’t complete, whatever the real reason. So we did what you do: chase the symptoms one plausible layer at a time. This part is worth showing, because it’s what real protocol debugging looks like before the root cause shows up — four coherent theories, all wrong:

• a MIC-generation error that pointed at a legacy TKIP / ver=1 WPA-IE path we’d deliberately removed;
• an “MFP Required but network not MFP Capable” skip that looked like an access-point misconfiguration;
• an “invalid key_mgmt ‘SAE’” that looked like a supplicant config bug;
• association rejects (status 53) that looked like a PMKID-caching or SAE fast-reconnect problem.

None of the on-device logs named the real problem. An over-the-air sniffer capture did: the SAE exchange was completing successfully — just too late. In the capture you could watch the access point retry, run out of patience, and reject the association at almost the exact instant the DUT finished the math. This wasn’t a protocol bug at all. It was latency — SAE computation had regressed roughly 10× from one release to the next, and every downstream “auth failure” was a symptom of a handshake that no longer finished inside the AP’s timeout window.

  What the logs screamed              What was actually true
  ----------------------------------  ----------------------------------
  "Failed to generate ... MIC"        the MIC path was fine
  "4-Way HS failed: wrong key"        the password/PMK was correct
  ASSOC-REJECT  status_code=53        SAE completed... just too late
  ----------------------------------  ----------------------------------
  three accusations, one cause: SAE math regressed ~10x, so the AP
  gave up and rejected before the DUT could finish the handshake.

That reframed the whole thing from “why is the protocol failing” to “why did the crypto get 10× slower” — and that is where the agent earned its keep. The Mbed TLS 4.x migration had landed between the two releases (that’s its own story), so crypto was the suspect.

I brought my own wrong hypothesis to the party. My prompt literally asked whether I needed to enable a hardware crypto accelerator (CRACEN) — meld the two .config files, I said, and see what HW accelerators we’re missing. The agent killed my theory in its first reply:

CRACEN ECC acceleration is already enabled and byte-for-byte identical in both builds… The 450 ms → 5 s regression is a software mbedtls-ECP tuning regression introduced by the Mbed TLS 4.x migration.

(Why software crypto at all, on a part that has a hardware accelerator? This build ran SoftAP alongside station mode, and the PSA-backed path doesn’t support that combination yet — so WPA3 fell back to the supplicant’s internal SAE, which does its elliptic-curve math with Mbed TLS building blocks directly instead of going through the top-level PSA API. On that path the hardware accelerator never gets a look in, exactly as the agent said.)

It then isolated the real delta: two software Mbed TLS options — MBEDTLS_ECP_NIST_OPTIM and MBEDTLS_HAVE_ASM — were on in v3.3 and off in v3.4. Great. Root cause in one turn. It even wrote the fix: set those two Kconfig symbols back on.

The fix that built clean and changed nothing

I flipped the flags. Rebuilt. Flashed. Still 5 seconds.

This is the same trap as Case 1, wearing a different outfit. The fix was right in principle and inert in practice — and it took a human reality-check on hardware to expose that, because the build was perfectly happy. My message was three words of ground truth: “still 5s.” That single fact sent the agent somewhere far more interesting than its confident first answer:

The Kconfig symbols you flipped only exist in .config — they appear in no generated header. So the symbols you set are dead. Your test never actually exercised the optimization.

In this SDK’s build, the crypto library is generated from config templates, and those two performance symbols were never emitted as #defines into the generated crypto config header. Flipping the Kconfig did nothing because nothing downstream read it. And why were they off in the first place? Not the downstream SDK:

The downstream SDK never disabled these — Zephyr did, in the Kconfig conversion.

Upstream TF-PSA-Crypto defaults MBEDTLS_ECP_NIST_OPTIM on. When that config was converted to a Zephyr Kconfig symbol, it was given no default line at all — so it silently fell to n. MBEDTLS_HAVE_ASM had a blanket default y if !ARM, which turns the assembly multiply off on every ARM target, including the Cortex-M mainline cores where it’s perfectly usable. An upstream default, lost in translation, with no crash and no error — just a protocol that quietly got an order of magnitude slower.

The ground truth: numbers off the board

Static reasoning found the cause; only the hardware confirmed it. The agent added a small SAE-PROF timing helper around the SAE hot spots, I built and ran it on the nRF54L, and the numbers settled the argument:

NIST_OPTIM alone did most of the work — an order of magnitude on the dominant step — and the assembly multiply shaved off another chunk. “Both” lands at ~425 ms, right back at the old ~450 ms. That’s the receipt. The fixes are zephyr#111632 (default NIST_OPTIM on) and zephyr#111633 (enable HAVE_ASM on Cortex-M mainline).

What actually did the debugging

Strip out the AI and you’re left with the techniques that carried both cases. None of them are new; the agent’s value was running them fast and tirelessly, not replacing them.

Give the agent a real pass criterion and real instruments. The reason these sessions worked at all is that the agent wasn’t debugging in the abstract — it had a UART console, a west flash, and an unambiguous pass/fail signal (Boot_sig good, Scan request done, a millisecond count). An agent with a ground-truth oracle self-corrects; an agent reasoning in a sealed room writes beautiful fiction — Case 1’s whole first act was the sealed-room version. And some oracles the agent simply can’t run itself: the thing that named Case 2’s real problem was an over-the-air sniffer capture, and the thing that quantified it was an on-target profiler (a few SAE-PROF timing prints the agent wrote, that I built and ran). The model doesn’t hold the logic analyzer or the sniffer — you do, and those captures are exactly what break the “confident but wrong” spell.

git bisect — and its trap. Bisect is the first reflex, and the agent drove it well. But it also walked straight into the classic trap: it bisected a local refactor branch and landed on a commit that never shipped on the release lines at all. A bisect is only as honest as the tree you run it on. On release branches, the real divergence wasn’t in drivers/spi/ — it was one level up, in the west.yml manifest pin.

Zephyr’s config and devicetree are diffable — use that. The moves that cracked Case 1 were embarrassingly concrete:

• meld the two .config files — but know the difference between a symbol that changes behavior and Kconfig serialization noise (we burned real time on SOC_NORDIC_BSP_NAME, which turned out to be cosmetic).
• Diff the DTS — easydma-maxcnt-bits was the entire ballgame, and it’s right there in the board’s generated devicetree.
• Diff the west.yml manifest pins across branches. The “identical SPI driver” was a lie of omission: the two branches pinned different Zephyr revisions, and that’s where the driver diverged.
• Check which driver you actually compiled (wc -l on the source is a perfectly good start). “The code is the same” is a hypothesis, not a fact.

Make silent errors loud. Both bugs were only cryptic because a return code got swallowed — the void write path in Case 1, a dead Kconfig symbol in Case 2. One LOG_ERR at the right layer turned a 12-minute hardware hunt into a one-line answer. If your failure mode is “wrong result, no error,” the highest-leverage change is often to make the error exist.

Escalate the model on purpose. This is the one habit I’d hand every embedded engineer. Auto spent a day building a confident, wrong pinctrl theory. The EasyDMA root cause — the connection I hadn’t made — showed up minutes after I switched to Opus and asked a sharper question. Model choice is not a background setting on a hard bug. If the agent is generating plausible prose instead of converging, stop and escalate; don’t let the cheap model dig you a deeper hole.

What this says about agents on real silicon

Put the two cases side by side and the pattern is uncomfortable but useful:

Neither “AI can’t debug firmware” nor “AI debugs firmware for you” is true. A frontier model connected a hardware limit to a misleading symptom faster than I did — from public code, in minutes. The same class of tool, on autopilot, also burned a day defending a fiction. The dividing line wasn’t the model’s intelligence — it was whether a human kept a ground-truth oracle in the loop and escalated deliberately when the reasoning stopped converging.

So the useful mental model isn’t “assistant” or “autopilot.” It’s a tireless, fast, occasionally overconfident junior engineer who has read all the code and none of the datasheets you keep in your head — and who must never be allowed to grade its own homework. You own the bench. You own the pass criterion. You own the escalation. Do that, and it’s genuinely one of the best debugging accelerators I’ve used on embedded. Skip it, and it’ll hand you a board-passing fix built on a wrong theory and call it done.

Where it landed

Kept short — the substance is above; this is the receipt. All merged upstream:

• SPI EasyDMA MAXCNT chunking fix (nRF91): zephyr#110621.
• WPA3-SAE performance — restore the Mbed TLS defaults Zephyr’s Kconfig lost: zephyr#111632 (MBEDTLS_ECP_NIST_OPTIM) and zephyr#111633 (MBEDTLS_HAVE_ASM on Cortex-M mainline).

The takeaway

The pitch for AI coding is speed. The real lesson from these two bugs is less flattering and more useful. First: the model you pick is the difference between an afternoon and a wasted day — on Auto, the same tool spent a day defending a fiction; escalated to Opus, it found a hardware limit in minutes. Second: a fix passing on the board is not the same as a fix being right — a cheap model will happily hand you a green-booting workaround built on a wrong theory, and the only thing that exposes it is an orthogonal instrument (a sniffer, a profiler, a bisect on the right tree), never the build and never the boot. Used that way — strong model, real instruments, and the judgment to distrust a confident answer nobody has measured — an agent is one of the best debugging accelerators I’ve used on embedded. Used on autopilot, it’s a tireless, confident way to be wrong on schedule.

PS: This post was drafted with Claude Code from my own session transcripts and notes, and every PR linked was checked against the public GitHub history for merge state and content before being cited. The debugging, the wrong turns, the colleague’s bench experiment, and the opinions are as they happened.

Dotstar Systems builds and debugs wireless systems where firmware meets silicon — Wi-Fi drivers, DMA paths, crypto stacks, on Zephyr, Linux, and bare metal. If you’re pulling AI agents into embedded work and want them pointed at real bugs instead of confident fiction — or you just have a Boot_sig 0x17 that won’t quit — let’s talk.