Systemd

conf file management

The main recommendation is to respect the conffile hierarchy, have the minimum changes needed to effect customisations, using drop-ins if they are found to work enough.

It is preferred to do changes by adding files instead of modifying files placed by upstreams package manager, this comes into its own when there are updates to apply, unless the changes are being done in system packages like .deb

/etc/systemd/system.conf.d/override.conf

The itco watchdog timeout of 30 is only a linux suggestion, it supports up to 614 seconds according to linux source code, "0x3ff", so set it to that to make it less likely that systemd gets watchdogged, as it is really a last resort, even though systemd also helps the machine recover quickly.

I set the watchdog to maximum time so even if systemd takes a minute or so to process an instruction under heavy load like mdadm raid scrubbing work, the watchdog still waits:

  1. [Manager]
  2. RuntimeWatchdogSec=614s

wdctl can on some newer systems check the time left whilst pid1 has the watchdog open so if for example a daemon-reload command appears to block, the seconds left til the watchdog resets the system can be seen.

service units

control groups can be set to carve up the prioritisation of system resources, it is quality of service for processor and memory.

Socket activation

Socket activation allows for services to get started by socket activity, and possibly exit whenever there are no more sockets open from accept(), or even while all these sockets are idle, which the service uploads to systemd using the FDSTORE= functionality, then exits.

Lots of services do not have any support for this, so for efficiency a preloadable can modify their behaviour, which is preferred over use of a proxyd as the service will work with the "real" file descriptor.

TCP services tend to start up by calling linux socket(), setting some options, calling bind(), then at some point entering a loop calling accept() and adding these fd's to a set monitored by select() or various poll(), and then close().

To dup2() or not.

I want to make the service behave more closely to native, so use of bind() to be a wrapper around close() and dup2() might lead to surprise behaviour.

This leaves how to make socket() calls return the already open systemd socket, taking advantage that these usually take reconfiguration of a service to move around.

I decided to count the calls, the FileDescriptorName= is used to pass a hint to the activated process, where the matching socket() call returns the descriptor, it only requires to strace the service to work out which socket calls should get substituted.

By doing this, the passed file descriptors are kept in neat and tidy between 3 and SD_LISTEN_FDS, and by passing the hint, it is not dependent of the exact order of the fds which is a systemd internal anyway, and also used to throw away redundant bind() as second calls to this often fail, and possibly close() calls by the process, wheras “extra” ioctl and setsockopt calls setting the same thing again and again are usually ok.

Extra descriptors opened by the application then follow the end of SD_LISTEN_FDS, such as the results of calling accept() or local sockets used to upload such descriptors to systemd via the FDSTORE= system prior to exiting, so it can get restarted whenever activity happens.

As experimental software, best work only with converting unimportant and rarely used internal services to start on demand, to save server's precious memory, use mosquitto as a testcase:

It has also been trialed with slapd, so included code to stop slapd unlink() its own local listening socket.

Inject preloadable into /etc/systemd/system/mosquitto.service.d/override.conf

  1. [Service]
  2. Environment=LD_PRELOAD=/usr/local/lib/sockethack/sockethack.so

Define several mosquitto_testnet.socket, one for each binding. The socket(2,1,6)=0 means to match the zeroth call to socket() with the parameters 2,1,6. For example 2 refers to IPv4 and 10 to IPv6

  1. [Unit]
  2. Description=Attach test net
  3. [Socket]
  4. ListenStream=192.0.2.1:1883
  5. #FileDescriptorName=socket(10,1,6)=0
  6. FileDescriptorName=socket(2,1,6)=0
  7. Service=mosquitto.service
  8. FreeBind=true
  9. [Install]
  10. WantedBy=sockets.target

The final step is to stop and disable mosquitto.service, enable and start the socket units, then if all goes well connecting to any of the the sockets triggers the service to start and pass all the descriptors.