Ep.09 - S2|E2: Writing, Debugging & Cross-Compiling My First Linux Character Driver

In this new episode of Build, Create & Learn – A Maker’s Journey, I take the next step into the world of Embedded Linux: writing, debugging, and cross-compiling my very first character driver.

 

After my initial “Hello Kernel” experiment, I wanted to go deeper — to understand how real drivers work, how they register with the kernel, and how data travels between user space and kernel space. Along the way, I learned the hard way how to debug without crashing the system, and how to cross-compile a kernel module from my Ubuntu machine to a Raspberry Pi.

 

This episode is about more than just code — it’s about exploring the invisible layers that make Linux so powerful, and why knowing what happens beneath the surface gives every maker a new kind of confidence.

 

Thanks for listening, and don’t forget to check out the companion article at https://herndlbauer.com/blog/a-makers-journey-podcast-episode-9/ — it includes the full source code and more details from my journey into the kernel.

Episode Transcript

(00:01):
Hello everybody, welcome back to Build, Create & Learn A Maker’s Journey.
Last time we wrapped up with the first kernel Hello World module.
It was small but symbolic.
The moment I realized I could actually inject my own code into the kernel and do something.

(00:25):
So what’s next after this first step?
Most resources like books, blogs and videos recommend as a next milestone to write your own character driver.
So I started looking into that.

(00:48):
Character drivers are like the training wheels of kernel programming.
They are simple but powerful and you can use them to control almost any kind of hardware that deals with streams of bytes.
Think serial ports, sensors or old school terminals.
They are also where you first meet one of Linux’s favorite philosophies.

(01:13):
Everything is a file.
On Linux you don’t talk directly to hardware.
You open, read and write to files on the /dev.
Those files are actually the interfaces to the drivers.
If you are cat /dev/random you literally read bytes from a random number generator inside the kernel.

(01:35):
So when you write your own character driver you are not just writing C code.
You are extending the file system itself.
You are adding your own voice to /dev.
I didn’t want to reinvent UART or SPI.

(01:58):
Those already exist written by people way smarter than me.
My goal was to understand how the things plug into the kernel.
Because I am not aiming to becoming a low level firmware engineer.
I am a systems builder.
I build drones, sensors and prototypes that combine hardware and software.

(02:21):
Knowing what happened at the driver level gives you X-ray vision.
Suddenly when something doesn’t work you can look beneath the surface, interrupts, buffers, syscalls.
All these hidden layers start to make sense.

(02:45):
Kernel programming feels familiar until it suddenly doesn’t.
You are still in C but half of the libraries and language features you rely on are gone.
There is no printf, no malloc, no sleep.
The kernel has its own versions.
With printk, kmalloc and msleep are designed for a special world of the kernel space.

(03:08):
And the world is dangerous in the user space.
If you mess up your program crashes.
In the kernel space if you mess up everything crashes.
The kernel is the heart of the operating system and when it panics the machine freezes or reboots.
That’s why kernel programming feels a bit like performing surgery while the patient is awake.

(03:31):
You have incredible access but one wrong move can cause chaos.
This separation between the kernel and the user space exists for a reason.
Stability and security.
User programs run in isolated memory.
If they misbehave the kernel just kills them.
But kernel code runs with full privileges.

(03:54):
That’s the trade of ultimate power and ultimate responsibility.
Debugging in user space is like walking with a flashlight.
You can stop, inspect and rerun as often as you like.

(04:15):
In the kernel space it’s like crawling through a cave with only the glow in the dark breadcrumbs.
You can just attach the gdp to the running kernel module.
Instead you use logging and tracing tools.
The simplest one is printk.
It’s the kernel’s version of the printf and it writes messages to the system log which you can read using dmessage.

(04:42):
But sometimes you need more visibility.
That’s where the tracing framework under /sys/kernel/debug/tracing comes in.
You can use an echo command to enable specific function tracing then cat trace to see the live feed of what the driver is doing.

(05:03):
Which function gets called, how long they take and in what order.
It feels primitive compared to modern ide’s but once you get used to it it’s surprisingly powerful.
You start thinking differently, more methodical and cautious.
Each logline is like a breadcrumb back to sanity.

(05:24):
And yes, I crashed my system a few times.
A missing pointer check, a faulty copy operation, boom, kernel panic.
At first it’s terrifying but when you learn to deal with it you reboot, fix the code and move on a little wiser.

(05:49):
My character driver followed a fairly classic pattern.
I defined two core functions.
One for reading and one for writing and a small buffer to hold the data.
When the user space writes to the /dev/char driver the kernel calls my write function which safely copies into the buffer using the copy_from_user method.

(06:13):
When the user space reads from it my read function calls copy_to_user handling data back.
These functions exist to keep the kernel and user memory isolated.
You never ever dereference user pointers directly in kernel code.
Then comes the registration step.

(06:33):
Every character driver needs a major and a minor number.
The major number identifies the driver and the minor number represents the instances of it.
For my test I manually registered the major number 100 and created the device node myself.
Once I did that I could finally ecotext into /dev/char driver and read it back with cat.

(06:57):
That sounds small but that moment seeing data flow through my kernel code feels pretty good.
It was proof that the plumbing was working.
There were a few gotchas along the way.

(07:18):
For instance the register_chrdev_region() call reserves a range of device numbers but if you forget to release them in your cleanup function then the next build will fail until you reboot.
Another subtle bug came from the file offset position used in the read function.
If you don’t update it correctly the driver will always return the same data or never indicate the end of file.

(07:46):
If you don’t update it correctly your driver will always return the same data or never indicate the end of file.
It’s these tiny details so trivial in the user space that suddenly matters a lot when you are deep inside the kernel.
I also learned to always include a proper cleanup function.

(08:07):
Unload a buggy driver without cleaning up allocated memory or device numbers can leave the kernel in a weird and half broken state.
If you would like to see the full source code of my character driver I’ve published a companion article on my blog.
You will find the link in the show notes and the episode description.
There I’ve added the complete code listing, the make file setup and some other details.

(08:34):
It’s a great way to follow along and if you want to try to build your own driver step by step.
After getting the driver running on my Ubuntu machine I wanted to test it also on a Raspberry Pi.
It’s my go-to single board computer for quick experiments.

(08:57):
Of course you can just build it directly on the Pi but if you have ever compiled the kernel on a Pi directly you know it takes forever.
Cross-compiling on a faster desktop machine saves you time and lets you use your favorite tools.
I followed the official documentation on the Raspberry Pi page to set up the cross-compiler.

(09:21):
The key is to build the same kernel version your target device runs and install the matching headers.
Otherwise your module won’t load.
The kernel checks version compatibility very strictly.
You make some adjustments to your make file and then run make producing a.co file for your target ARM architecture.

(09:46):
I transferred the file to the Raspberry Pi and run sudo insmod chardriver.co and it worked on the first try.
That felt orderly satisfying. Like swapping an engine piston between two different cars and watching both still run.
It proved that my code wasn’t just tied to one environment.

(10:08):
It could live in another processor entirely.
Understanding how cross-compilation works gives you the freedom to target whatever hardware makes sense for the project.
It’s one of these meta skills that multiplies your options as a maker.
Kernel programming trains a certain mindset, patience, attention to detail and respect for complexity.

(10:39):
It forces you to think about how the data actually travels from the user space through the system calls into the kernel structure and finally to the hardware.
You begin to see why operating systems are layered this way and how those abstractions keep everything stable.
It also lets you rethink what you develop.

(11:02):
When you load a buggy module and freezes your system you realize just how fragile software can be.
But when it works you feel the quiet pride that only comes from controlling something low level.
If you want to try it yourself the classic book Linux Device Drivers the third edition is still a jam.
It’s a bit old but it’s free and open source and the concepts are still timeless.

(11:27):
And if you prefer video learning the LinkedIn Learning course Linux Device Drivers gives you a modern overview.
Once you got the basics explore the tracing tools or experiment with /sys and /proc, and maybe even peek into how interrupts are handled.
Each layer opens up a new world.
Now that I have built and tested my own real driver I’m planning to move deeper into embedded Linux integration.

(12:03):
Things like build route and the Yocto project and how drivers fit into a complete image.
That’s where the real fun begins.
Building whole systems where the kernel, the driver and the user application all play together.
But I will definitely circle back to the kernel space again especially for projects where the single board computer collaborates with a microcontroller.

(12:27):
In those setups writing a lean custom driver can mean faster communication and less latency.
Working inside the Linux kernel feels like exploring an engine room of a massive ship.
It’s loud, complex and full of moving parts.
You don’t need to live there permanently but once you visit it you never see the surface the same way again.

(12:50):
For me this journey wasn’t about becoming a kernel developer.
It’s about curiosity.
Peeling back another layer of the system I everyday use.
So if you are tempted to peek under the hood so do it.
You will crash a few times, sure, but you will also understand the machine in a way few people ever do.

(13:14):
If you’re listening on YouTube or Spotify leave a comment or a subscribe which really helps others to discover this show.
If you enjoy this first step into the kernel model make sure to subscribe also to the makers logbook.
My weekly newsletter where I share behind the scene notes, project updates and extra resources.
Thanks for tuning in until next time.

(13:36):
Let’s keep building, creating and learning together.