Every boot writes a record of what actually happened to
/run/arctic/boot.log, copied to /var/log/boot.log
once the boot finishes so it survives past that session:
cat /run/arctic/boot.log cat /var/log/boot.log # from a previous boot
This did not always exist. Service failures used to be sent straight
to /dev/null by rc.boot, on the theory that a
quiet boot should stay quiet — but the actual effect was that a service
which failed to start left no trace anywhere: the console
showed a plain failed with no detail, there was nothing in
any log, and diagnosing why meant re-running the exact failing command
by hand and hoping the failure reproduced identically outside of boot
context. The log now records, for every service: that it was
considered, whether it started, and if not, its actual stderr output —
which is what made it possible to find the three separate causes behind
the iwd/dbus failure below in the first place, rather than continuing to
guess.
This looked like a single dbus problem and was actually three unrelated ones, found one at a time by adding boot logging and reading what it said instead of guessing further:
1. /var/run did not exist. dbus itself
listens on /run/dbus/system_bus_socket and was running
correctly the entire time — confirmed directly with
dbus-send --system ... org.freedesktop.DBus.ListNames,
which succeeded. But iwd is built on ell, a separate D-Bus client
library, and ell's default system bus lookup checks
/var/run/dbus specifically, not /run/dbus.
On a system where /var/run is not a symlink to
/run (the traditional FHS arrangement, and one this rootfs
build had simply never set up), that path does not exist, and iwd
cannot find a bus it is sitting right next to. Fixed by adding
/var/run -> ../run and /var/lock -> ../run/lock
as two more of the merged-filesystem symlinks created during the rootfs
build.
2. The netdev group did not exist.
iwd ships its own D-Bus policy file
(/usr/share/dbus-1/system.d/iwd-dbus.conf) granting members
of a group called netdev access to
net.connman.iwd. dbus-daemon logs
Unknown group "netdev" in message bus configuration file
and — this is the part that made it hard to spot — does not fail to
start over this. It logs the warning, discards that one policy file, and
continues running with everything else intact, so dbus itself looked
completely healthy while iwd was quietly unable to claim its own bus
name because the specific policy permitting it had been silently
dropped. Fixed by adding netdev as a real system group,
both on the live ISO and by the installer for a freshly installed
system.
3. No machine ID had ever been generated.
dbus-daemon --system refuses to start at all without
/etc/machine-id (or the older
/var/lib/dbus/machine-id location) being present, and
nothing on Arctic ever generated one. Combined with
rc.boot sending service output to /dev/null at
the time, this specific failure was completely invisible — dbus simply
never started, with nothing on screen or in any log to say why. Fixed
by generating one on /etc/rc.d/dbus's first start, via
dbus-uuidgen if present or a random-hex fallback otherwise,
and never regenerating it again once one exists.
To check the state of all three directly on a running system:
service status dbus service status iwd ls -l /etc/machine-id readlink /var/run grep netdev /etc/group
Check the boot log first — as of the fix above, a service that is
enabled but has no executable script at /etc/rc.d/<name>
is now reported explicitly (no executable /etc/rc.d/name)
rather than silently skipped the way it used to be. If the script does
exist and is executable, run it by hand to see the real error:
/etc/rc.d/some-service start
If it is a translated definition under a non-busybox init (OpenRC,
runit, and so on), regenerate it explicitly in case the translation is
stale relative to /etc/rc.d:
arctic-init-setup openrc # or whichever init is actually running
/etc/fstab is generated once, at install time, directly
from /proc/mounts at the moment the installer finishes —
not reconstructed from the config afterward, and not regenerated by
arctic-rebuild. If you change the disk layout after
installing (add a partition, move something), fstab does not update
itself; edit it by hand, the same as on any other distribution past the
install step.
arctic rollback
goes back exactly one generation, undoing the most recent rebuild. If the system will not boot at all, choose the previous "Arctic Linux (generation N)" entry directly from the boot menu instead — full detail on both paths in generations and rollback.
This is the sandbox working as intended, not a bug — see
building from source for exactly why
everything builds under a read-only host by design. Check the recipe for
a missing DESTDIR/pkgdir in its
package() step before assuming anything else is wrong.
If a recipe was recently regenerated from
ports/manifest.tsv and a hand fix to it has disappeared,
check for a missing recipe.local marker next to it — see
packages. This has happened more than once
to real recipes in this tree and is always the same root cause: a
hand-written recipe with no marker gets silently regenerated back to the
template on the next manifest-driven pass.
More: arctic linux docs · install · declarative configuration · generations and rollback · init systems · kernels · packages (alpm) · desktops and audio · networking · ephemeral shells · musl · building from source · install.conf reference