In this module we will focus on replacing the kernel image currently used by the BeagleBoard Black with one generated by ourselves. We will start with a completely clean copy of the Linux source code, configure it for the hardware we are using, and finally build the kernel. We will also get a first glimpse at modules and the role they play in a Linux-based system.

Configuring and building the kernel

The process we will follow to obtain a kernel image built by ourselves should feel familiar at this point, since it follows the usual make-based “configure-and-then-build” approach. Although in principle it is the same as, say, configuring and building U-Boot, for the kernel we want to also build a few more components that are required by the kernel itself, namely device trees and modules.

  • Download a copy of the Linux source code from the official website: https://kernel.org.
    • At the time of writing this document, the latest release is 6.11.2. If that is not the case by the time you are reading this, you can still use whatever the latest one is at that point. The approach we will follow should remain the same regardless of the version.
    • It is not recommended to clone the entire repository, since it is quite big. It is enough to simply get a compressed archive, e.g.
$ wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.11.2.tar.xz.
  • Uncompress the archive and navigate to it:
$ tar -xf linux-6.11.2.tar.xz
$ cd linux-6.11.2
  • Take some time to look around. You should see the now-familiar arch directory. As expected, it contains a plethora of subdirectories that correspond to the architectures supported by Linux.
$ ls arch
alpha  arm    csky     Kconfig    m68k        mips   openrisc  powerpc  s390  sparc  x86
arc    arm64  hexagon  loongarch  microblaze  nios2  parisc    riscv    sh    um     xtensa

For the BeagleBone Black, we are interested in arm as the architecture. The default configurations for each architecture are stored in a directory named configs that is in turn inside the architecture-specific directory. In our case, that would be arch/arm/configs. As you can see, this directory contains quite a few files whose names follow the pattern *_defconfig. If you examine any of them, you will see that they contain a sequence of key=value entries. This, again, should feel familiar from the previous module, where we configured and built U-Boot. In fact, U-Boot “copied” this build system from the Linux project.

This system can be thought of as two closely related components: Kconfig and Kbuild. The main source of documentation is in the Linux source tree, specifically in the Documentation/kbuild directory. You are highly encouraged to read it.

We briefly touched on the basics of this system during the last module, but given its relevance we will take some time to go through it in more detail.

The Linux project uses make as its underlying build system, and most Makefiles in the source tree conform to the Kbuild syntax rules. These files can be name either Makefile or Kbuild. If files of both names exist in a directory, Kbuild will be used.

The essential trick used in Kbuild files combines a feature of make with a convention of the configuration file .config which is brought into the Kbuild system.

  • The trick itself is simple: a Makefile variable may have its definition extended (i.e., have something appended to it) with the operator +=. For example:
    obj-y += foo.o
  • The convention in the .config file is that (most) configuration variables will be given the value y or m depending on whether the feature is configured to be linked into the kernel or build to be a dynamically loaded module. These configuration variables are named following the pattern CONFIG_<name>.
    CONFIG_SPI_MASTER=y
    CONFIG_SPI_ATMEL=m
    • If a feature is not configured, a comment indicates its absence from the active configuration:
      # CONFIG_SPI_CH341 is not set
  • A combination of these two features is used to create or extend variables with the names obj-y, obj-m and (for features not configured) obj-. For example, from drivers/spi/Makefile we have:
    obj-$(CONFIG_SPI_MASTER) += spi.o
    obj-$(CONFIG_SPI_ATMEL) += spi-atmel.o
    obj-$(CONFIG_SPI_CH341) += spi-ch341.o
    Based on the previous configuration, variable obj-y will have the value spi.o, variable obj-m the value spi-atmel.o, and the discarded (in the end) variable obj- will have the value spi-ch341.o.
  • The variables obj-y and obj-m are known to the Kbuild system as goal variables. For each directory, the system will compile all of the objects in obj-y, and then link them into a consolidated file in the directory named built-in.o. (This step is known as partial linking, since the resulting object file still contains unresolved references to names that will be linked from other object files.)
  • All the objects in the goal variable obj-m will be built as dynamically loadable modules.

The configuration file we will use for the BeagleBone Black is arch/arm/configs/multi_v7_defconfig. This is a “general” configuration that should be applicable to any system based on ARMv7, which is the ISA implemented by the CPU on the board. This means that, even though it should “just work”, the default configuration will also probably include many things that are probably not usable by the BeagleBone Black. Be that as it may, this will not be an issue for us.

  • Use the previous configuration file as the basis for the kernel to be built.
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- multi_v7_defconfig

Since we are going to be cross compiling a lot from now on, it would be tiresome to specify values for ARCH and CROSS_COMPILE every time we want to do so. One “trick” is to define an alias for make that includes the appropriate arguments. For ARM-based systems, I like to call it armmake.

Add the following to the shell configuration profile (if you are using Bash then the file would be ~/.bashrc):

export PATH=$HOME/cross-toolchain/arm-unknown-linux-gnueabi/bin:$PATH
alias armmake='make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi-'
This will allow us to simply type armmake followed by the build target. You need to reopen a shell emulator in order for these changes to take effect.

(Please make sure to understand what we did here, do not just blindly copy things!)

At this point we have a valid configuration stored in the .config file. Feel free to examine it:

$ armmake menuconfig

We will not change anything for now, but it is a good idea to become familiar with the configuration options that the kernel provides.

  • Build the kernel based on the current configuration:
$ armmake menuconfig
    • The zImage build target represents a compressed kernel image (hence the z prefix).
    • This will take a while. It is definitely recommended that you instruct make to use more than one core by specifying a number after the -j option. In this case we used the output from the nproc utility.

Once this finishes you can find the generated kernel image in arch/arm/boot/zImage. Please make a note of the current time as indicated by your host platform. When the kernel image is generated, a timestamp is added to it. When we boot this image in a few minutes, we will check this timestamp and make sure that it is what we expect.

If you recall from the previous module, when booting we had to load the kernel and a file which we have been referring to as a device tree. For now, this looks quite arbitrary since we have not spent time talking about what it actually is. We will certainly do so, but in a future module, so for the moment being we will proceed to generate a device tree that corresponds to this kernel.

  • In order to know all the build targets available, you can execute $ armmake help. The output is somewhat verbose, so we will not include it here, but note that there is one target named dtbs. This stands for device tree blobs. Its description reads Build device tree blobs for enabled boards.
  • Build this target:
$ armmake dtbs -j $(nproc)
    • This should not take as much time as building the kernel.

After this step finishes, you can find the relevant device tree in arch/arm/boot/dts/ti/omap/am335x-boneblack.dtb.

Almost there! The last thing we need to build are the modules, that is, the drivers that are going to be dynamically loaded, as opposed to those that were built directly into the kernel.

  • The relevant build target is called modules.
$ armmake modules -j $(nproc)
    • This will also take a bit of time to complete, unfortunately.

The resulting modules are… well, all over the place. Do not worry about this for now, we will soon “collect” them all.

At this point we are ready to replace the kernel we have been using when booting from the uSD card with the one we just generated.

  • Insert the uSD card into the host platform and make sure that it is accessible from the virtual machine (in case you are using one).
    • If the uSD card still contains the image that we were using in the previous module, the host operating system should automatically mount the two partitions. If for some reason you had to erase the contents of the uSD card, first flash it by executing the prepare-card.sh script you were given access to.
  • If you recall, the uSD card contains two partitions. The kernel and device tree are stored in the first one, so replace them with the new ones:
$ cp arch/arm/boot/zImage /media/training/boot/vmlinuz
$ cp arch/arm/boot/dts/ti/omap/am335x-boneblack.dtb /media/training/boot
  • Copy the modules previously built to the root filesystem on the uSD card, which is located in the second partition. There is a build target named modules_install that does precisely this, but be sure to specify a value for the environment variable INSTALL_MOD_PATH:
$ armmake INSTALL_MOD_PATH=/media/training/root modules_install
    • You will notice that we modules are being copied to $INSTALL_MOD_PATH/lib/modules/6.11.2. The version will of course match that of the kernel.
  • Correctly eject the uSD card from the host platform:
$ sync
$ sudo umount /media/training/boot
$ sudo umount /media/training/root
$ sudo eject /dev/sdb
  • Insert the uSD card into the appropriate slot in the BeagleBone Black, and boot from it.
  • Interrupt the autoboot process.
  • If you have a U-Boot environment variable that contains the sequence to boot from the uSD card, go ahead and run it.
    • Otherwise create one and be sure to save it:
=> setenv my_boot 'fatload mmc 0:1 $loadaddr /vmlinuz; fatload mmc 0:1 $fdtaddr /am335x-boneblack.dtb; setenv bootargs console=${console} root=/dev/mmcblk0p2 ro rootfstype=${mmcrootfstype}; bootz $loadaddr - $fdtaddr'
=> saveenv
Saving Environment to FAT... OK
=> run my_boot

If everything goes well, you should boot using the kernel image you just built a few minutes ago. To make sure that this is the case, run uname -a and check the timestamp. It should match the time when the kernel was generated.

Alternatively, you can also check the contents of /proc/version.

Bonus: Booting over Ethernet

This section is optional. It is possible to continue booting from the uSD card throughout the rest of the training.

If an embedded system has networking capabilities, it is possible to load the kernel via TFTP (Trivial File Transport Protocol) and then access the root filesystem over NFS (Network File System). This could be used during development to reduce the time it takes to transfer files between the host platform and the embedded device.

The overall idea is as follows:

  1. We will start booting from the uSD card, which must contain a U-Boot image.
  2. We will instruct U-Boot to load the kernel and device tree from the network (as opposed to the uSD card like before) onto their respective addresses.
  3. We will set a slightly different sequence of arguments to be passed to the kernel, this time specifying a different root device that corresponds to the NFS server on the host platform.
  4. We will then tell U-Boot to boot from the loaded kernel using the familiar bootz command.
  5. The kernel will boot as usual but instead of mounting the root filesystem from the uSD card, it will do so through the network.

To start setting things up, connect an Ethernet cable from the BeagleBone Black to the host computer.

If you are using a virtual machine, it is necessary to create a network adapter that corresponds to the new connection. The following steps are specific to Virtual Box.

  • Power off the VM.
  • Select it from the list of VMs and go to Settings.
  • Go to Network and then Adapter 2 (Adapter 1 should probably be already in use).
  • Enable it. Attach it to a Bridge Adapter and select the appropriate device under Name. (This last step will vary depending on the hardware you are using, make sure to select the one that corresponds to the Ethernet connection. If you are using an Ethernet-to-USB converter, select that.)
  • Under Advanced/Promiscuous Mode, select Allow All.
Hit OK to save the changes. You can power on the VM now.

You should see a new connection:

The name may differ from the one shown here (enp0s8). If you are connecting the Ethernet cable directly to the host platform, it is possible that the connection is called eth1, for example. Make sure you identify the correct name; one approach can be to look at the output of $ ip link before and after connecting the cable.

From now on, for simplicity, every time we want to refer to this connection we will use the name $IFACE. You can create an environment variable with it if you want, but it is not necessary.

We also need to setup a fixed IP address for this connection. The reason for this is that we will have to refer to its associated IP address from U-Boot, and if the address is dynamic, we would have to modify the boot command every time it changes.

The Linux distribution used by your host platform probably uses Network Manager to manage network interfaces. There are many ways to do this, so we will focus on one of them, namely nmtui, which is a curses-based GUI which is part of Network Manager.

  • Execute nmtui.
$ nmuti
  • Choose Edit a connection.
  • Choose the appropriate Ethernet interface (most likely Wired connection 2, but could be 1 as well).
  • Change its Profile name to beagleboneblack (or whatever you prefer, this is merely to distinguish it from others).
  • Select Show for IPv4 CONFIGURATION.
  • Change the IPv4 CONFIGURATION to Manual (the default value will most likely be Automatic).
  • Add an Address with value 192.168.111.1/24.
  • Set Gateway to 0.0.0.0.
  • Enable Never use this network for default route.
  • Make sure Automatically connect is enabled.
  • Choose OK at the bottom.
  • Chose Quit to exit.

See if you can now ping this address from U-Boot!

  • Power on the BeagleBoard Black and stop the autoboot process.
  • Create an environment variable named ipaddr and assign it the value 192.168.111.5.
  • ping the host platform.

We now need to have a TFTP server on the host platform that can be used to serve files to U-Boot (in particular, the kernel and device tree).

  • Install the tftpd-hpa package.
$ sudo apt install tftpd-hpa
  • Edit the file /etc/default/tftpd-hpa (as superuser) and change the value of TFTP_DIRECTORY to /srv/beagle/tftp.
    • Know that this is not strictly necessary, the default value of /srv/tftp is just fine. This is only to be specific about the fact that this directory is going to be used to serve files that are related to the BeagleBone Black.
  • Restart the associated service to take into account the new changes.
$ sudo systemctl restart tftpd-hpa.service
  • Create the /srv/beagle/tftp directory and copy the kernel and device tree.
$ sudo mkdir /srv/beagle/tftp
$ sudo cp /path/to/linux-source/arch/arm/boot/zImage /srv/beagle/tftp/vmlinuz
$ sudo cp /path/to/linux-source/arch/arm/boot/dts/ti/omap/am335x-boneblack.dtb /srv/beagle/tftp
  • Check that the TFTP server is working by making U-Boot load the kernel and device tree from the host platform.
    • Create a U-Boot environment variable named serverip with value 192.168.111.1.
      => setenv serverip 192.168.111.1
    • Use the tftp command to load the kernel and device tree into their respective addresses.
      => tftp ${loadaddr} vmlinuz
      [... output omitted ...]
      => tftp ${fdtaddr} am335x-boneblack.dtb

Note that in principle this is very similar to the approach we had to follow when loading the kernel and device tree from the uSD card, only that in this case we are using the network.

Note that you can use this kernel and device tree to continue with the booting process as if they had been loaded from the uSD card.
=> setenv bootargs console=${console} root=/dev/mmcblk0p2 ro rootfstype=${mmcrootfstype}
=> bootz ${loadaddr} - ${fdtaddr}
Kernel image @ 0x82000000 [ 0x000000 - 0xad2200 ]
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 8ffeb000, end 8ffff2a3 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 6.11.2 (training@modularmx) (arm-unknown-linux-gnueabi-gcc (crosstool-NG 1.26.0) 13.2.0, GNU ld (crosstool-NG 1.26.0) 2.40) #1 SMP Thu Oct 10 11:36:00 JST 2024
[    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: TI AM335x BeagleBone Black
...
In this case we are using the root filesystem that is on the uSD card.

In order to serve the root filesystem from the host platform to the BeagleBone Black, we need to configure a suitable NFS server.

  • Create the directory from which we will serve the root filesystem to the board.
$ sudo mkdir -p /srv/beagle/nfs
  • Install the nfs-kernel-server package.
$ sudo apt install nfs-kernel-server
  • Edit the file /etc/exports (as superuser) and add the following entry:
    /srv/beagle/nfs *(rw,insecure,sync,no_subtree_check,no_root_squash)
  • Restart the NFS service.
$ sudo systemctl restart nfs-server.service
  • Make sure that the new configuration settings are loaded.
$ sudo exportfs -a
$ sudo showmount -e
Export list for modularmx
/srv/beagle/nfs *

With this we are now, in principle at least, ready to serve the root filesystem through NFS.

To show that we really are using the one on the host platform (as opposed to the one stored on the uSD card) we will create a minimalistic root filesystem and completely boot the system. The word minimalistic is actually quite literal here, because this root filesystem will only have one file, namely, an executable that will start running as soon as the kernel is done initializing the system.

Please note that this is really only intended as a fun exercise. In practice you would want a more complete root filesystem!

The first program executed is usually /sbin/init. The plan is then to create a simple “Hello, world!”-type of executable, serve it to the BeagleBone Black through the NFS protocol, and have Linux run it as the process with PID 0. Given that this is the only file that will be available on the root filesystem, it cannot have any dependencies, so that rules out writing a program that depends on the C standard library (for example, by calling printf). A simple alternative is then to write a small program in assembly that uses the Linux system call interface to write a message to the screen.

It is not required that you know assembly, though! You can simply follow the sequence of instructions that will be provided next, and everything should just work. If you want to geek out a bit about what is going on with the assembly code, feel free to ask the instructor.

  • Edit a file named hello.asm with the following contents:
.section .text

.global _start
_start:
  mov r0, #1
  ldr r1, =message
  ldr r2, =length
  mov r7, #4
  swi 0

loop:
  b loop

.section .data
message:
  .asciz "Hello from ARM assembly!\n"
length = .-message
  • Create an executable out of it.
$ arm-unknown-linux-gnueabi-as hello.asm -o hello.o
$ arm-unknown-linux-gnueabi-ld hello.o -o hello
  • Copy the executable to the appropriate location in the root filesystem to be served to the BeagleBone Black
$ sudo mkdir /srv/beagle/nfs/sbin
$ sudo cp hello /srv/beagle/nfs/sbin/init
  • Power up the board and have U-Boot run the following sequence of commands (remember that you can save them as a variable in the environment):
=> setenv ipaddr 192.168.111.5
=> setenv serverip 192.168.111.1
=> tftp ${loadaddr} vmlinuz
=> tftp ${fdtaddr} am335x-boneblack.dtb
=> setenv bootargs console=${console} rootwait ipv6.disable=1 root=/dev/nfs nfsroot=${serverip}:/srv/beagle/nfs,vers=3 rw ip=${ipaddr}:${serverip}:::::dhcp
=> bootz ${loadaddr} - ${fdtaddr}
    • You should start seeing the now familiar boot logs from the kernel. Eventually, assuming everything went well, you will see this:
    • Take a moment to bask in the joy of having done something cool

Labs

  • Copy the root filesystem that is on the second partition of the uSD to /srv/beagle/nfs and reboot the board. You may notice that the system gets stuck after systemd tries to configure the network interface on the BeagleBoard Black. Try and solve this!
    • Hint: the system is trying to use DHCP to configure the network interface, but there is no DHCP server to get it from. You could install one on the host platform and have it issue an IP address in the range being used by our setup. You could also try to setup a static IP address for the network interface instead.
  • If you found our small assembly program interesting, how about extending it?
    • Write a simple version of the guess-the-number game, where the program asks the user repeatedly to “guess” a number.
      • If the user guesses the number, the program ends. If not, the program indicates to the user whether the guess is less or greater than the number and repeats the process.
      • You can perform I/O by using system calls. In the previous example we used write, which on ARM32 has the number 4. You can check this link for a detailed reference; figure out how to use read.
      • For an introduction to ARM assembly, you can go here.
    • Can you think of anything else?