Inspecting Code That's Running Somewhere Else

6 min

If you’ve read my earlier post on cross compilation, you’ll know that embedded development involves a separation that mobile development abstracts away: your development machine and your target device are fundamentally different systems. This separation doesn’t just affect how you build your code. It also changes how you debug it.

The Debugging Experience I Was Used To

Coming from mobile development, I had a particular mental model of debugging. I’d set a breakpoint in Xcode or Android Studio, run my app, and when execution hit that breakpoint, I could inspect variables, step through code, and examine the call stack. The debugger and the code being debugged felt like they existed in the same world, even when the app was running on a connected phone rather than a simulator.

What I didn’t fully appreciate was how much infrastructure was working behind the scenes to create that experience. Mobile platforms provide debugging services built into the operating system. The IDE communicates with these services over USB, and it all just works.

Embedded systems, particularly bare-metal microcontrollers with no operating system, don’t have any of this infrastructure. There’s no debug service waiting for your IDE to connect. The microcontroller is just executing instructions, with no awareness that you might want to pause and inspect what it’s doing.

How Cross Debugging Works

Cross debugging solves this problem by splitting the debugger into two parts that communicate across the boundary between your host machine and the target device.

On your development machine, you run a debugger frontend. This is typically GDB, the GNU Debugger, or a graphical interface built on top of it. The frontend is where you set breakpoints, issue commands, and view your source code. It runs natively on your laptop and understands your keyboard, your display, and your project files.

On the target side, you need some way to actually halt the processor, read its registers, and inspect memory. This is where debug probes come in. Devices like the ST-Link or SEGGER J-Link connect to your microcontroller through special debug interfaces, most commonly JTAG or SWD. These interfaces are built into the silicon itself, providing hardware-level access to the processor’s internal state.

The debug probe runs a small piece of software called a GDB server. This server translates between the GDB protocol that your frontend speaks and the low-level debug commands that control the target hardware. When you click “pause” in your IDE, that request travels from the frontend, across USB to the debug probe, and down into the chip’s debug hardware, which halts the processor mid-instruction.

Nothing Comes for Free

Something I hadn’t initially considered is that this debug capability has costs. The microcontroller has to dedicate some of its limited resources to supporting these debug interfaces.

At the silicon level, the debug hardware takes up physical space on the chip. There are dedicated pins for JTAG or SWD that can’t be used for anything else while debugging. Some microcontrollers let you reclaim these pins for GPIO in production builds, but during development they’re unavailable. On a chip with only twenty or thirty pins, losing two or four to debugging is a real trade-off.

There’s also internal circuitry dedicated to the debug interface: logic for halting the core, reading registers, and managing breakpoints. This circuitry consumes a small amount of power even when you’re not actively debugging. On some devices, particularly those designed for low-power applications, you can disable the debug interface entirely in production to save power, but then you lose the ability to diagnose issues in the field.

Some debug features consume RAM too. If you’re using features like real-time tracing or certain advanced breakpoint modes, the chip may need to buffer data internally before sending it to the debug probe. On a microcontroller with only a few kilobytes of RAM, even a small debug buffer can represent a notable percentage of your available memory.

The Printf Alternative

Given these costs and the complexity of setting up proper debug hardware, some developers reach for a more familiar approach: printf debugging. The idea is simple. You write diagnostic messages to an unused serial port, connect a USB-to-serial adapter to your development machine, and watch the output in a terminal. It feels comfortable, especially coming from higher-level development where console logging is standard practice.

The problem is that embedded systems often operate under timing constraints that make printf debugging problematic.

Sending characters over a serial port takes time. At 115200 baud, transmitting a single character takes roughly 87 microseconds. A modest debug message of 50 characters might take over 4 milliseconds to transmit. If your system is running a control loop that needs to execute every millisecond, inserting printf statements can break your timing.

This leads to a common frustration in embedded debugging: bugs that disappear when you try to observe them. Adding printf statements changes the timing of your system enough that race conditions, buffer overflows, or interrupt-related bugs no longer manifest. You add logging, the problem vanishes, you remove logging, the problem returns. You might also introduce new bugs that only exist because of the timing changes from your debug output.

There are ways to mitigate this. You can buffer your debug output and transmit it during idle time, or use a faster interface like SWO (Serial Wire Output) that’s designed for debug tracing with less overhead. But these approaches add complexity and still don’t fully eliminate the observer effect.

Halting Hardware

What took me a while to get used to is how literal the cross debugging process is. When you hit a breakpoint during cross debugging, the microcontroller’s processor actually stops. The clock is still ticking, but the CPU isn’t fetching or executing instructions. You’re halting the hardware in place.

This has consequences that don’t arise in mobile debugging. If your embedded system is controlling something in the real world, pausing execution might cause problems. A motor might keep spinning, a heater might stay on, or communication with another device might time out.

Similarly, inspecting variables works differently than you might expect. There’s no runtime providing nice abstractions. When you ask to see a variable’s value, the debugger is reading raw bytes from specific memory addresses and interpreting them based on your source code’s type information. If that memory happens to be a hardware register that changes when read, simply inspecting it might alter your system’s behaviour.

What Mobile Development Hides

Once again, mobile platforms abstract away this complexity. When you debug an iOS or Android app, you’re debugging a process running under an operating system that mediates everything. The OS provides memory protection, process isolation, and debugging APIs. You’re never directly touching hardware registers or halting a physical processor. And console logging on a modern smartphone has negligible timing impact.

Embedded cross debugging puts you closer to the hardware. You’re not debugging through an operating system’s abstraction layer; you’re interacting with the silicon directly.

The Practical Reality

If you’re coming from mobile development and setting up cross debugging for the first time, the good news is that the day-to-day experience isn’t dramatically different. You’ll still set breakpoints, step through code, and inspect variables. The tools have matured to the point where the underlying complexity is largely invisible. Modern embedded IDEs like STM32CubeIDE, PlatformIO, or VS Code with the right extensions hide much of this complexity behind familiar interfaces. You click a button, your code uploads, and breakpoints work roughly as you’d expect. But underneath, the machinery is different from what you’re used to.

But when something goes wrong: when breakpoints don’t trigger, when your timing-sensitive code behaves differently under the debugger, when you can’t figure out why your debug session won’t connect, understanding the architecture helps. Knowing there’s a GDB server running on your probe, that your chip has dedicated debug hardware with its own limitations, and that halting the processor means genuinely halting it gives you a mental model for troubleshooting.

As always, thanks for joining me on this journey into embedded development!