smolBSD

build your own minimal BSD UNIX system

What is smolBSD?

smolBSD helps you create a minimal NetBSD 🚩 based BSD UNIX virtual machine that's able to boot and start a service in a couple milliseconds.

• No prior NetBSD installation is required, a microvm can be created and started from any NetBSD, GNU/Linux, macOS system and probably more.
• PVH boot and various optimizations enable NetBSD/amd64 and NetBSD/i386 to directly boot QEMU or Firecracker in about 10 milliseconds on 2025 mid-end x86 CPUs.

microvm typical boot process

Usage

Requirements

• A GNU/Linux, NetBSD or macOS operating system (might work on more systems, but not CPU accelerated)
• The following tools installed
  • curl
  • git
  • bmake if running on Linux or macOS, make on NetBSD
  • qemu-system-x86_64, qemu-system-i386 or qemu-system-aarch64 depending on destination architecture
  • sudo or doas
  • nm (not used / functional on macOS)
  • bsdtar on Linux (install with libarchive-tools on Debian and derivatives, libarchive on Arch)
• A x86 VT-capable, or ARM64 CPU is recommended

Project structure

• mkimg.sh creates a root filesystem image, usually called by [b]make
• startnb.sh starts a NetBSD virtual machine using qemu-system-x86_64 or qemu-system-aarch64
• sets contains NetBSD "sets" by architecture, i.e. amd64/base.tgz, evbarm-aarch64/rescue.tgz...
• pkgs holds optional packages to add to a microvm, it has the same format as sets.

A service is the base unit of a smolBSD microvm, it holds the necesary pieces to build a BSD system from scratch.

• service structure:

service
├── base
│   ├── etc
│   │   └── rc
│   ├── postinst
│   │   └── dostuff.sh
│   ├── options.mk       # Service-specific defaults
│   └── own.mk           # User-specific overrides (not in git)
├── common
│   └── basicrc
└── rescue
    └── etc
        └── rc

A microvm is seen as a "service", for each one:

• There COULD be a postinst/anything.sh which will be executed by mkimg.sh at the end of root basic filesystem preparation. This is executed by the build host at build time
• If standard NetBSD init(8) is used, there MUST be an etc/rc file, which defines what is started at vm's boot. This is executed by the microvm.
• Image specifics COULD be added in make(1) format in options.mk, i.e.

$ cat service/nbakery/options.mk
# size of resulting inage in megabytes
IMGSIZE=1024
# as of 202510, there's no NetBSD 11 packages for !amd64
.if defined(ARCH) && ${ARCH} != "amd64"
PKGVERS=10.1
.endif

• User-specific overrides COULD be added in own.mk for personal development settings (not committed to repository)

In the service directory, common/ contains scripts that will be bundled in the /etc/include directory of the microvm, this would be a perfect place to have something like:

$ cat common/basicrc
export HOME=/
export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/pkg/bin:/usr/pkg/sbin
umask 022

mount -a

if ifconfig vioif0 >/dev/null 2>&1; then
        # default qemu addresses and routing
        ifconfig vioif0 10.0.2.15/24
        route add default 10.0.2.2
        echo "nameserver 10.0.2.3" > /etc/resolv.conf
fi

ifconfig lo0 127.0.0.1 up

export TERM=dumb

And then add this to your rc(8):

. /etc/include/basicrc

Build considerations

You can build a system using 2 methods:

• using the base Makefile target, this will build directly on the host, it will use ext2 as the filesystem if building on a Linux host, and ffs if building on a NetBSD host
• using the recommended build Makefile target, this will fire up a NetBSD builder microvm that will use its own root filesystem and will format the image using ffs.

Warning

When using the base target, bmake(1) will directly use your host to build images, and as postinst operations are run as root in the build host: only use relative paths in order not to impair your host's filesystem.

Again, it is highly recommended to use the build target for any other service than the builder image.

Note

You can use the ARCH variable to specify an architecture to build your image for, the default is to build for the current architecture.

Image building

Note

In the following examples, replace bmake by make if you are using NetBSD as the host.

• Either create the builder image if you are running GNU/Linux or NetBSD

$ bmake buildimg

• Or by simply fetch it if you are running systems that do not support ext2 or ffs such as macOS

$ bmake fetchimg

Both methods will create an images/build-<arch>.img disk image that you'll be able to use to build services.
To create a service image using the builder, execute the following:

$ bmake SERVICE=nitro build

This will spawn a microvm running the build image, and will in turn build the requested service.

Example of a very minimal (10MB) virtual machine

Create a rescue-amd64.img file for use with an amd64 kernel.

$ bmake SERVICE=rescue build

Create a rescue-amd64.img file but with read-only root filesystem so the VM can be stopped without graceful shutdown. Note this is the default for rescue as set in service/rescue/options.mk

$ bmake SERVICE=rescue MOUNTRO=y build

Create a rescue-i386.img file for use with an i386 kernel.

$ bmake SERVICE=rescue ARCH=i386 build

Create a rescue-evbarm-aarch64.img file for use with an aarch64 kernel.

$ bmake SERVICE=rescue ARCH=evbarm-aarch64 build

Start the microvm

$ ./startnb.sh -k kernels/netbsd-SMOL -i images/rescue-amd64.img

Example of an image filled with the base set on an x86_64 CPU

$ bmake SERVICE=base build
$ ./startnb.sh -k kernels/netbsd-SMOL -i images/base-amd64.img

Example of an image running the bozohttpd web server on an aarch64 CPU

Service name is specified with the SERVICE make(1) variable.

$ make ARCH=evbarm-aarch64 SERVICE=bozohttpd build
$ ./startnb.sh -k kernels/netbsd-GENERIC64.img -i images/bozohttpd-evbarm-aarch64.img -p ::8080-:80
[   1.0000000] NetBSD/evbarm (fdt) booting ...
[   1.0000000] NetBSD 10.99.11 (GENERIC64)     Notice: this software is protected by copyright
[   1.0000000] Detecting hardware...[   1.0000040] entropy: ready
[   1.0000040]  done.
Created tmpfs /dev (1359872 byte, 2624 inodes)
add net default: gateway 10.0.2.2
started in daemon mode as `' port `http' root `/var/www'
got request ``HEAD / HTTP/1.1'' from host 10.0.2.2 to port 80

Try it from the host

$ curl -I localhost:8080
HTTP/1.1 200 OK
Date: Wed, 10 Jul 2024 05:25:04 GMT
Server: bozohttpd/20220517
Accept-Ranges: bytes
Last-Modified: Wed, 10 Jul 2024 05:24:51 GMT
Content-Type: text/html
Content-Length: 30
Connection: close

Example of starting a VM with bi-directionnal socket to host

$ bmake SERVICE=mport MOUNTRO=y build
$ ./startnb.sh -n 1 -i images/mport-amd64.img 
host socket 1: s885f756bp1.sock

On the guest, the corresponding socket is /dev/ttyVI0<port number>, here /dev/ttyVI01

guest$ echo "hello there!" >/dev/ttyVI01

host$ socat ./s885f756bp1.sock -
hello there!

Example of a full fledge NetBSD Operating System

$ bmake live # or make ARCH=evbarm-aarch64 live
$ ./startnb.sh -f etc/live.conf

This will fetch a directly bootable kernel and a NetBSD "live", ready-to-use, disk image. Login with root and no password. To extend the size of the image to 4 more GB, simply do:

$ dd if=/dev/zero bs=1M count=4000 >> NetBSD-amd64-live.img

And reboot.

Customization

The following Makefile variables change mkimg.sh behavior:

• ADDPKGS will fetch and untar the packages paths listed in the variable, this is done in postinst stage, on the build host, where pkgin, NetBSD's package manager, might not be available
• ADDSETS will add the sets paths listed in the variable
• MOUNTRO if set to y, the microvm will mount its root filesystem as read-only
• MINIMIZE if set to y, will invoke sailor in order to minimize the produced image
• By default, services are build on top of the base set, fetched in sets/<arch>/base.tar.xz, this can be overriden with the SETS make(1) variable.

The following environment variables change startnb.sh behavior:

• QEMU will use custom qemu instead of the one in user's $PATH

Basic frontend

A simple virtual machine manager is available in the app/ directory, it is a python/Flask application and needs the following requirements:

• Flask
• psutil

Start it in the app/ directory like this: python3 app.py and a GUI like the following should be available at http://localhost:5000:

Final notes

Kernel

As of June 2025, most of the features needed for smolBSD fast boot are integrated in NetBSD's current kernel, and NetBSD 11 releases those still pending are available in my NetBSD development branch.

Pre-built 64 bits kernel at https://smolbsd.org/assets/netbsd-SMOL and a 32 bits kernel at https://smolbsd.org/assets/netbsd-SMOL386

aarch64 netbsd-GENERIC64 kernels are able to boot directly to the kernel with no modification

In any case, the bmake kernfetch will take care of downloading the correct kernel.