Email: firstname.lastname@example.org • Twitter: @benhutchingsuk • Debian: benh • Gitweb: git.decadent.org.uk • Github: github.com/bwhacks
Here's the second chunk of notes I took at Linux Plumbers Conference earlier this month. Part 1 covered the Distribution kernels track.
Moderators: George Wilson and Serapheim Dimitropoulos from Delphix; Omar Sandoval from Facebook
Problem: ability to easily anlyse failures in production (live system) or post-mortem (crash dump).
Debuggers need to:
Most people present use crash; one mentioned crash-python (aka pycrash) and one uses kgdb.
crash-python is a Python layer on top of a gdb fork. Uses libkdumpfile to decode compressed crash-dumps.
drgn (aka Dragon) is a debugger-as-a-library. Excels in introspectiion of live systems and crash-dumps, and covers both kernel and user-space. It can be extended through Python. As a library it can be imported and used from the Python REPL.
sdb is Deplhix's front-end to drgn, providing a more shell-like interactive interface. Example of syntax:
> modules | filter obj.refcnt.counter > 10 | member name
Currently it doesn't always have good type information for memory. A raw virtual address can be typed using the "cast" command in a pipeline. Hoping that BTF will allow doing better.
Allows defining pretty-print functions, though it appears these have to be explciitly invoked.
Answering tough questions:
Some discussion around the fact that drgn has a lot of code that's dependent on kernel version, as internal structures change. How can it be kept in sync with the kernel? Could some of that code be moved into the kernel tree?
Omar (I think) said that his approach was to make drgn support multiple versions of structure definitions.
Q: How does this scale to the many different kernel branches that are used in different distributions and different hardware platforms?
A: drgn will pick up BTF structure definitions. When BTF is available the code only needs to handle addition/removal of members it accesses.
Brendan Gregg made a plea to distro maintainers to enable BTF.
Moderator: Hans de Goede of Red Hat
Pain points and missing pieces with Wayland, or specifically GNOME Shell:
ssh -X. Pipewire goes some way to the solution. The whole desktop can be remoted over RDP which can be tunnelled over SSH.
Speaker: Peter Robinson of Red Hat
Can now use u-boot with UEFI support on most Arm hardware. Much easier to use a common kernel on multiple hardware platforms, and UEFI boot can be assumed.
"Enterprise" and "industrial" IoT is not a Raspberry Pi. Problems result from a lot of user-space assuming the world is an RPi.
Is bluez still maintained? No user-space releases for 15 months! Upstream not convinced this is a problem, but distributions now out of synch as they have to choose between last release and arbitrary git snapshot.
Wi-fi and Bluetooth firmware fixes (including security fixes) missing
linux-firmware.git. RPi Foundation has improved Bluetooth
firmware for the chip they use but no-one else can redistribute it.
Lots of user-space uses
/sys/class/gpio, which is now deprecated and
can be disabled in kconfig. libgpiod would abstract this, but has poor
documentation. Most other GPIO libraries don't work with new GPIO
Similar issues with IIO - a lot of user-space doesn't use it but uses user-space drivers banging GPIOs etc. libiio exists but again has poor documentation.
For some drivers, even newly added drivers, the firmware has not
been added to
linux-firmware.git. Isn't there a policy that it
should be? It seems to be an unwritten rule at present.
Speaker: Kees Cook of Google
LWN article: https://lwn.net/Articles/798913/
Speaker: Dodji Seketeli of Red Hat
Speakers: Maciej Rozycki of WDC
LWN article: https://lwn.net/Articles/799331/