Better living through software

Here's the second chunk of notes I took at Linux Plumbers Conference earlier this month. Part 1 covered the Distribution kernels track.

Kernel Debugging Tools BoF

Moderators: George Wilson and Serapheim Dimitropoulos from Delphix; Omar Sandoval from Facebook

Details: https://linuxplumbersconf.org/event/4/contributions/539/

Problem: ability to easily anlyse failures in production (live system) or post-mortem (crash dump).

Debuggers need to:

Get consistent stack traces
Traverse and pretty-print memory structures
Easily introduce, extend. combine commands

Most people present use crash; one mentioned crash-python (aka pycrash) and one uses kgdb.

Pain points:

Tools not keeping up with kernel changes
Poor scripting support in crash

crash-python is a Python layer on top of a gdb fork. Uses libkdumpfile to decode compressed crash-dumps.

drgn (aka Dragon) is a debugger-as-a-library. Excels in introspectiion of live systems and crash-dumps, and covers both kernel and user-space. It can be extended through Python. As a library it can be imported and used from the Python REPL.

sdb is Deplhix's front-end to drgn, providing a more shell-like interactive interface. Example of syntax:

> modules | filter obj.refcnt.counter > 10 | member name

Currently it doesn't always have good type information for memory. A raw virtual address can be typed using the "cast" command in a pipeline. Hoping that BTF will allow doing better.

Allows defining pretty-print functions, though it appears these have to be explciitly invoked.

Answering tough questions:

Can I see any stacks with a specific function in? (bpftrace can do that on a live system, but there's no similar facility for crash dumps.)
What I/O is currently being issued?
Which files are currently being written?

Some discussion around the fact that drgn has a lot of code that's dependent on kernel version, as internal structures change. How can it be kept in sync with the kernel? Could some of that code be moved into the kernel tree?

Omar (I think) said that his approach was to make drgn support multiple versions of structure definitions.

Q: How does this scale to the many different kernel branches that are used in different distributions and different hardware platforms?

A: drgn will pick up BTF structure definitions. When BTF is available the code only needs to handle addition/removal of members it accesses.

Brendan Gregg made a plea to distro maintainers to enable BTF. (CONFIG_DEBUG_INFO_BTF).

Wayland BoF

Moderator: Hans de Goede of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/533/

Pain points and missing pieces with Wayland, or specifically GNOME Shell:

GNOME Shell is slower
Synergy doesn't work(?) - needs to be in the compositor
With Nvidia proprietary driver, mutter and native Wayland clients get GPU acceleration but X clients don't
No equivalent to ssh -X. Pipewire goes some way to the solution. The whole desktop can be remoted over RDP which can be tunnelled over SSH.
No remote login protocol like XDMCP
No Xvfb equivalent
Various X utilities that grab hot-keys don't have equivalents for Wayland
Not sure if all X's video acceleration features are implemented. Colour format conversion and hardware scaling are implemented.
Pointer movement becomes sluggish after a while (maybe related to GC in GNOME Shell?)
Performance, in general. GNOME Shell currently has to work as both a Wayland server and an X compositor, which limits the ability to optimise for Wayland.

IoT from the point of view of view of a generic and enterprise distribution

Speaker: Peter Robinson of Red Hat

Details: https://linuxplumbersconf.org/event/4/contributions/439/

The good

Can now use u-boot with UEFI support on most Arm hardware. Much easier to use a common kernel on multiple hardware platforms, and UEFI boot can be assumed.

The bad

"Enterprise" and "industrial" IoT is not a Raspberry Pi. Problems result from a lot of user-space assuming the world is an RPi.

Is bluez still maintained? No user-space releases for 15 months! Upstream not convinced this is a problem, but distributions now out of synch as they have to choose between last release and arbitrary git snapshot.

Wi-fi and Bluetooth firmware fixes (including security fixes) missing from linux-firmware.git. RPi Foundation has improved Bluetooth firmware for the chip they use but no-one else can redistribute it.

Lots of user-space uses /sys/class/gpio, which is now deprecated and can be disabled in kconfig. libgpiod would abstract this, but has poor documentation. Most other GPIO libraries don't work with new GPIO UAPI.

Similar issues with IIO - a lot of user-space doesn't use it but uses user-space drivers banging GPIOs etc. libiio exists but again has poor documentation.

For some drivers, even newly added drivers, the firmware has not been added to linux-firmware.git. Isn't there a policy that it should be? It seems to be an unwritten rule at present.