The systemd fallacy

(…) So, get yourself a copy of The Linux Programming Interface, ignore everything it says about POSIX compatibility and hack away your amazing Linux software. It’s quite relieving!

Lennart Poettering @ fosdem.org

systemd is wrong on so many levels I hardly know where to start. Perhaps its single most important design fault is that it was conceived with a blatant disregard for servers. The author’s manifesto and the “systemd for admins” series provide good insight into his motives for designing systemd.

He goes on and on about how you can save 3 seconds here and 5 seconds there by parallel and delayed service startup – systemd actually has a feature to measure system boot time. The question is: who cares? Desktop users, yes. Embedded users, maybe. Server users? Nope. It doesn’t matter if a server comes up in 96.3 seconds instead of 33.1. What counts is if it stays up and is not too cumbersome to maintain.

So how are systemd’s goals achieved? Basically, by throwing well-proven Unix paradigms out the window and clearly admitting it. Yes, Unix was designed 42 years ago. And no, it’s not broken. I’m not a die-hard traditionalist nor am I even that reluctant to adopt new solutions, but Unix stays the single most successful server OS design in the world for a reason – still used today in various forms after those 42 years. It’s simple, elegant and it works. The mainstay of its design is simplicity and modularity. One program for one task; easy interconnection between programs. Yet we are expected to ditch all that for something new and shiny.

One of the design goals of systemd is to get rid of shell scripts in the boot process and… rewrite everything in C, as the author doesn’t seem to be very fond of grep being called 77 times and awk 92 times during his system boot. Now, why do we have shell scripts in the boot process? They’re simple. They’re easy to read. Every single competent un*x admin knows at least the basics of shell scripting. There is almost complete control over the entire boot process and anything can be changed in a few seconds. Of course, one can argue it’s almost as easy to change systemd’s C code, recompile and reinstall. I’ll let you on a little secret: when do you usually need to change something in the boot process? When something doesn’t work right. No matter if you’re comfortable at your desk with your triple 30″ screens or in the trenches data center after an all-nighter gone horribly wrong – you need to fix the problem pronto. The last thing you want to worry about is instrumenting, debugging and rebuilding C code at the core of your OS.

The second design goal seems to be incredible and unwarranted intentional complexity. The single most important process in the userland is supposed to be clean, small and efficient. Let’s take a look what systemd is supposed to supervise:

  • restarting processes after they crash. sysvinit doesn’t do that and we don’t have restartd or a thousand other programs for it. Oh, wait…

  • collecting information on daemon crashes. Nowadays most daemons have their own crash report formats, logging to syslog, stderr, directly to text log files, to binary dumps, etc. Good luck making the authors conform to a single standard. And good luck with all the corner cases.

  • keeping control (via cgroups) over processes detached from their parents. But for that we already have, well… cgroups?

  • delayed/on-demand service startup. “on most machines where sshd might be listening somebody connects to it every other month or so.” says the author. On a workstation – maybe. How much RAM are you going to save by delaying the startup of a few daemons? If they’re unused, they’ll be swapped out anyway. To support on-demand startup of network services, yet another functionality already available elsewhere had to be implemented within systemd: inetd.

  • dependency based service management. To the author, dependency based management is redundant. The problem is, every boot process is dependency based. Think services, not processes. Your services depend on their filesystems having been mounted, the filesystems depend on the underlying devices having been initialized, and so forth. We’ve already had rudimentary dependency based service management in System V! S31fancyd and S31foobar depended on S30whatsit for setup. At teardown, only with K10foobar and K10fancyd down could the system proceed with K20whatsit. Servers are unlike desktops in that server boot time counts from the moment you press the big red button to the moment that server actually starts providing all its services. Or in other words: if you’re waiting, who cares if it’s in parallel or in series? It doesn’t matter if e.g. ftpd is allowed to start before /home/ftp is mounted and files can be served. Besides, an administrator may choose to stop S30whatsit without stopping S31fancyd – and he or she probably knows what they’re doing. It’s much harder to force service actions with systemd: you end up constantly fighting its decisions.

  • systemd creates autofs mount points and starts daemons before their filesystems are available (obviously, fs operations will block until then). Sounds horrible, right? This is going to be an administration nightmare. There is no way to do autofs right. If anything goes wrong with the underlying I/O or autofs itself, you’re left with an unusable system. Even on Solaris, which arguably has the most reliable automounter implementation available. Incorporating autofs into PID 1 and the boot process (and hanging services off it) guarantees problems.

  • listening to hardware changes introduces potential stability and security issues – and there already are [more or less] working facilities acting on hardware events.

  • communication via D-Bus. D-Bus is _very_ desktop-oriented. It’s not called Desktop Bus for nothing. It’s designed for portability – not speed, reliability or simplicity. There are dozens of simple, robust message passing and IPC protocols, but this is by far one of the most complicated, perhaps second only to CORBA/IIOP. Instead of letting this abomination die, or at least stay confined to the desktop, it’s actually going to be incorporated into the boot process. Daemon developers are encouraged to use it. Let’s put it in the kernel while we’re at it.

systemd is overcomplicated and bloated with unnecessary features, almost as if someone was trying to implement a second kernel in userland. It looks like it was designed by someone who never saw anything else than their own workstation. It’s a nice exercise in self-managing systems, with its kitchen sink approach it’s certainly worth a look by desktop/embedded vendors as an alternative to sysvinit or in-house inits, but that’s it.

Linux userland boot process should be reviewed and cleaned up in most major distributions, perhaps even standardized (not necessarily, though – we currently have about four major userland boot systems, all in major distros – some diversity here is actually welcome).

However, systemd is not the way to go. It would set us back a decade. Let’s hope it doesn’t catch on – just like upstart or the first implementation of devfs back in 2000. It’s hardly surprising how many people are drinking the kool-aid – systemd offers a lot of lofty promises. With Red Hat’s financial backing and all the propaganda (I can’t even call it PR), it’s going to be an arduous fight, but remember: after all upstart made it into Ubuntu; devfs even made it into the kernel. Not all hope is lost.

This entry was posted in IT and tagged , . Bookmark the permalink.

17 Responses to The systemd fallacy

  1. 0xdeafbeef says:

    I recommend watching the recording of this “talk” about other fallacies. If you don’t know it already, it’s worth watching, even if you don’t get a lot new information from it.

    Talk homepage:
    https://events.ccc.de/congress/2010/Fahrplan/events/4017.en.html

    Video:
    http://www.youtube.com/watch?v=ZTdUmlGxVo0

  2. oiaohm says:

    Even to server start up time can make a difference to how many +1 servers have to be running. Due to the speed a +1 server can be started.

    systemd gets quite a few things right.

    “collecting information on daemon crashes” this is not just logs but can be like core dumps. Information that will not be in the normal error logs. So what systemd is doing here is an enhancement. Basically there is more information that could be collected in case of daemon crashes that sys v init and upstart lets slip threw fingers. Kernel generated stuff. Items you don’t need deamon coders to play ball with. Yep it crashed what did the kernel tell me about it record done.

    “delayed/on-demand service startup.” Of course you miss that systemd style of this is expanded from inetd. /var/run/mysqld/mysqld.sock Can be hooked under systemd that inetd does not handle. Next is inetd does not handle services that should be left running after they have been started on demand. So in this regard systemd is more enhanced than inetd is. So at worst it should remain as the replacement to inetd.

    cgroup features have been disregarded by init systems. So its nice to see one that has bothered to integrate it.

    Of course you have presumed coder is insane. “dependency based service management” http://0pointer.de/public/systemd-man/systemd.unit.html Please read before and after features of systemd. Yes their is a dependency based system in systemd you can choose to use

    Systemd throws away the ideas of the numbered dependency based system. If I made everything I was running set as after=”thediskmounts” Then everything would only start after the disks were all mounted. Now if you want speed you can disregard this. Goes over to a direct assigned dependency system. Ie I depend on X don’t start me before it. Big issue with system v init was if you got your magic numbers wrong. No magic numbers nice improvement.

    “systemd creates autofs mount points” optional due to the the dependency system you missed in systemd design.

    “listening to hardware changes” Something has to. Also this segment of systemd can be disabled if you don’t want systemd responding to it. So secuirty concerns can be addressed.

    “communication via D-Bus” Of course you miss that systemd is its own dbus server. When you are running a dbus server for desktop users you might has well talk by that IPC as use another one.

    “restarting processes” I left this one to last because I do have a issue here. systemd has a method that attempts to hide the fact that a service did crash and has been restarted. By caching the communication. This is something I do see as a possible source of problems. Of course their is an option to disable this feature.

    Sorry to say out of your 8 points not one stacks up really all are addressable. Yes systemd is different. Done different. Lot of systemd should be taken onboard.

    system V needs to die just because of the idea of S1 to what ever and K1 to what ever instead of appache had to its config file added after mysql so when apache comes up everything php website needs inside is their. This is a far saner way of handling dependencies.

    Now a more detailed run through of the systemd dependency system.

    Nice is that the after=mysql instruction if you just directly say start apache mysql will not be forced started as well. But if both are starting at the same time. apache will be started after mysql. There are options in systemd to cause it to be force started as well.

    Now apache with systemd is configured.
    Requires=mysql
    after=mysql
    You try to manually start apache if mysql is not running. mysql will be started first. If mysql cannot be started apache is not attempted to be started. Once mysql is up then apache starts.

    Now of course Requires can be too strict. Since this setup if mysql dies apache will be taken down as well.

    RequiresOverridable=mysql instead of Requires=mysql will attempt to start mysql if it fails still start apache and will not shut apache down if mysql fails.

    Requisite=, RequisiteOverridable= Now this is even more evilish. If Requisite=mysql or RequisiteOverridable=mysql is used instead of Requires. apache will not even attempt to start at all if mysql is not already running. The start will be aborted.

    Nice simple tight dependency system with many dependency defines that suit what a system admin should need to happen. Ok what happened with good old system V. with a S lower than apache on mysql. mysql crashed on startup. system v keeps on going along starts apache now website does not work right.

    Hang on what is not a dependency based system. In fact system V init is not a dependancy based. Systemd has a well designed dependency based system. If you choose not to use it. Systemd will try to sort out your errors for you. Yes systemd is many time simpler to get the init system to make the correct selection starting stuff up.

    Shutting the system down on the other hand that could use some work.

    So will you not join me calling for system V init death as fast as we can.

    Also your ftpd example is counted by ConditionPathExists=/home/ftp So that while /home/ftp does not appear as a existing path. ftpd will not be started.

    Systemd allows giving very exact instructions. Lets say 1 drive fails to start up. Everything without a dependency on that drive will get started.

    Yes systemd requires to work perfectly more exact information about what stuff requires. Advantage if everything is provided their is no reason not to run.

  3. anonymous says:

    Why?
    Is it supposed to tell us not to mess with Poettering?

  4. aniou says:

    @oiaohm

    “on-demand service startup”? It isn’t expansion of inetd’s features. It is baroque ornament, completely unnecessary on server. If service is rarely used then should be called from inetd. If service is used continuously then should be started as daemon. I don’t see any benefit from delayed initialization – in addition, on my servers, exemplary MySQL will start soon, after call from monitoring system.

    By the way – how “one demand MySQL startup” (for example) mixes with replication? I see possibility that first call to slave service (unmonitored – unlikely, but possible) hits totally unsynchronized database, two days after master…

    We talk about inetd… You know, there are widely used alternatives. xinetd, for example. And I pretty sure that xinetd – in his specialized niche – has more features than systemd.

    You talk about respawning services. But… init does that. When you need more sophisticated solution you always can write simple script (some of mine works for years, really). And if You need even more then You can use another specialized tool, monit.

    Ah, monit. Handy tool that can handle service re-spawning, dependencies (from other processes, files, directories, local and remote services… to long to talk about all). systemd can’t do even half of that what monit does.

    But if I don’t like monit and I want conditional start, If remote NFS service is mounted? Here’s my little secret, look: if [ -f "/mountpoint/remote_file" ]; then echo “mountpoint not mounted”; (actions here); fi. OK, I’m joking. It isn’t secret. I wasn’t secret anytime. It’s sysadmin’s kindergarten.

    “dependency based-system”. I have it. In places where is necessary (monit, again). Ubuntu has this feature everywhere (upstart). And I saw Ubuntu in action: locked system because of mistype in IP addr. Locked system because separate weird problems with portmapper service and NFS mount. Look into their bugzilla, there are many, many other cases and ubuntu folks works hardly on them.

    And so on, and so on… Summarizing: systemd do many, many things and all things they do worse than smaller, optional, specialized tools. It is big, complicated daemon (I say nothing about D-BUS, but it is worth separate rant), with many internal dependencies that forces people to act in unnatural, inconvenient ways. Short example: someone wants simple ‘When condition then do_something ELSE to_something_else’ – ‘valid’ solution from systemd-devel: ‘You should create two separated pseudo-services with following dependencies..’ What next? Manager for visualizing gazilion dependencies services and pseudo-services?

  5. oiaohm says:

    xinetd is weaker than systemd majorally. There is one very particular feature. http://0pointer.de/blog/projects/socket-activation.html

    xinetd responding application when started has to stop when completed doing what woke it. A systemd socket-activation application does not have to stop. The start on demand is like Windows Service start on demand and OS X start on demard in systemd. The service after being activate on demand remains running until it has had no activity for long enough then is asked to shut back down cleanly.

    So yes mysql postgresql and other items can be start on demand on windows and OS X but are not on Linux due to weakness in inetd and init script design. This include xinetd.

    Socket activated where systemd passes the socket over to the started service addresses a weakness. xinetd has less features than systemd. Systemd can take every process xinetd can plus also proper start services on demand. So yes at a min xinetd should go and systemd remain in its place on Linux systems. Yes systemd can be used this way.

    Really xinetd and dbus are basically the same idea. Systemd fuses both of them and expands on.

    Simple fact is dependency order. You can run a script in a before mysql that is required if it fails mysql does not start. Or you can set mysql to only start when a particular file exists. Even better put both as one. So check program runs before mysql waiting for the master server. Master server running systemd has got a port bump from the slave so start the master server. The checking script on slave create a file to say it started exits and mysql slave starts.

    Really this is not required. Because you can set a mysql slave server either not to be serving data until the master server turns up or to serve data in read only until master server turns up. This is prity much standard in database servers. Yes unsynchronized database does happen in real world even running monit. Reason why I normally set my database servers to flatly refuse to serve data if the master server cannot be contacted. Slave up but not sending data master is down or unable to be connected to by slave. Yes I learnt my lesson on this from a switch malfunction. Where half the slaves were disconnected from the master and I did not notice. Better for site to fail than display wrong prices. Having monit tell my they were not synced was not a tail saver. Yes the worst part was monit messaging was going back threw the same failed network section and database replication. So monit was saying the servers were down. When they were up and still serving incorrect data to the public. After that mistake you learn a slave should never be able to send data if its master is missing.

    Thing is systemd your added script is an independent entry in systemd. Not going to get mangled just because the system updated. And multi things can be dependent on the same test. So you don’t have to run the same test 50 + times. Everything that needs that test is made dependent on that test. This is the difference. Systemd does not forbid scripts. Does not use them everywhere. And when you do use them they are done in a effect to reuse way.

    monit can be used in combination with systemd.

    “if [ -f "/mountpoint/remote_file" ]; then echo “mountpoint not mounted”; (actions here); fi”
    Except for the fact that is not what systemd does.

    I have a service waiting for a directory to turn up. Systemd knows its on the mount list and has not mounted yet. So systemd waits with the service marked as not started and waiting on a condition. Even better I can check on systemd what condition is not being met. Yours I have to scroll back through the log to see what in heck is going on. Not just go straight to the service I am interested in and pull the information.

    I find systemd works better with monit. Since it simpler to get messages out of systemd that a particular service has not started for a particular reason into monit. monit is only a useful as how clean the information on system state it can extract is. systemd is better this way.

    In fact shock horror systemd is many small programs. Not a single huge deamon. That can be disabled 1 by 1. If systemd does not need to do a particular role that can be removed from it.

    dbus is only systemd ipc because systemd is also a dbus server so is launchd on OS X by the way. Lot of items go by by due to systemd design.

    consolekit gone dbus server gone and the list goes on of independent items that systemd nukes. These are not items that really need be that different. What consolekit sets on users is handy to set on services as well. So systemd uses that functionally twice from a small program it contains. systemd is about the true Unix idea. Many small programs that each do their job well used in every local that job is required.

    dependency based-system does not prevent all forms of lock up correct. Also system v based system does not prevent all forms of lock ups. Every system can be pushed the wrong way.

    Really you need to read this page. http://0pointer.de/blog/projects/why.html Not all that upstart is missing. Upstart insanely dumb. Just like system v init is.

    dependency based systems to prevent data disasters at times. Due to the fact not starting something when all its required resources are not available.

    Lot of the worse issues of Upstart are preventable in systemd. Like trying to start 2 service using the same port. Then wondering why system dies. Upstart cannot check what ports are already in use and stop duplicate services from running into each other. Neither can system v init or monit simply. systemd can deal with this issue proper. You add to service information that it needs X port if X port is taken don’t attempt to start or kill what is in way. Yes you can attempt todo this with script. Systemd conflit kill is better than script because you script would have to find and kill what is on that port. systemd can do this very well.

    Bad ip address can send even system v init systems into major troubles.

    systemd also tries to have the least amount of dependency. If there is not a real issue between two items you don’t create a dependency. Of course their are areas where you must have dependencies. Not avoidable.

    There is a reason why systemd was designed to be able to start with no dependency data. I do mean no dependency data. This means if something is attempted to be started in the wrong order systemd will attempt to correct instead of fail.

    Basically systemd will have success on worse configured system than what system v init or upstart will. This is handy.

    In fact the closet thing out their to systemd is launchd on the OS X. So if you want to pull up errors from a init system that do truly relate go through the OS X bug list. Surprise there are very few bugs effecting launchd.

  6. Joost says:

    Fedora 15 forces systemd down our throats while it is barely functional for servers. It is supposed to be backwards compatible with upstart but there are many issues with that.
    My server does not have a graphical desktop. It does not have a monitor or even a graphics card installed, so the graphical services configuration wont work.

    I am used to chkconfig and ntsysv, but there are no equivalents for systemd. Yes, you can enable/disable/start/stop a single service with initctl, but how do I get a list of all services and what runlevel they start in? How do I change that with a few keystrokes?

    I use several services that have no native systemd support. Systemd provides backwards compatibility, but that does not work for all services, for example clamd wont run.

    I added initd=/sbin/upstart to the kernel commandline to revert to upstart. Then I discovered that some developers apply the ‘scorched earth’ policy and removed the initscript for basic services like cron and smartd!

    Then I tried systemd again. I have configured my system for runlevel 3, but initctl insists on runlevel 5 for the legacy initscripts. You cannot change this behaviour because it is hardcoded in the C source like engraved in stone, seriously! This is exactly why I prefer scripts.

    Snippet from systemctl.c:

    log_info("%s is not a native service, redirecting to /sbin/chkconfig.", i->name);

    argv[1] = file_name_from_path(sysv);
    argv[2] =
    streq(verb, "enable") ? "on" :
    streq(verb, "disable") ? "off" : "--level=5";

    log_info("Executing %s %s %s", argv[0], argv[1], strempty(argv[2]));

    That pretty much proves the “blatant disregard for servers”.

    I am running some PXE boot clients and maintain the boot image on the server. Previously I could just chroot the loop-mounted root filesystem and chkconfig the services. But now everything uses sockets and that wont work on a chrooted system.

    * The simplicity of sysvinit is its power, not its weakness! *

  7. Trevor Rollins says:

    The problem here is that you are just utterly clueless. The init process has a very well defined role. 40+ years of Unix has shown exactly the range of things it needs to do well. systemd does them all, very well, both in theory and in practice.

    Seriously, if you have a real-world, specific, practical issue with systemd, just say it. Otherwise, you are just a kneejerk idiot trying to have your say and make your mark. Yawn. No one is listening.

    • Chris S. says:

      Trevor Rollins – next time, try less abuse and more argument.

  8. Trevor Rollins says:

    @Joost, you are yet another knee jerk cretin. The problem you are facing can be narrowed down to somewhere between your ears. “Blatent disregard for servers”? Are you having a joke?

  9. Joost says:

    Dear Trevor Rollins.
    Being rude does not prove your superiority, neither does your lack of arguments.

    I’ll try to explain again and be more clear.

    Servers generally use runlevel 3, which is meant for console use. Runlevel 5 usually starts the graphical desktop. If service configuration is hardcoded for runlevel 5 (desktop use), then it disregards runlevel 3 (servers). If that is an issue between my ears, please correct me.

    So I was running runlevel 3:
    [code]
    $ runlevel
    N 3
    $ readlink /etc/systemd/system/default.target
    /lib/systemd/system/runlevel3.target
    [/code]

    Some legacy sysv service was disabled for runlevel 3 but still started on boot. Then I disabled it for runlevel 5 and yet again it started next boot, because it was enabled for runlevel 4. It appears that services get started unless they are disabled for all runlevels. Please tell me why this is not weird or confusing.
    Is it too difficult to properly map systemd runlevel3.target to sysv runlevel 3? Is this a bug or a feature? I honestly dont know.

  10. Erik says:

    Woah, I’m a bit disturbed that I actually sat down and read this random angry-nerd bashing.

    Just give it “five minutes”, man. Try to not be so biased and give it a sincere shot.

    I’m not really in favor for either way of doing booting. I know SYS-V is a pain in the butt, since I’ve tried writing my own initscripts..

    I can tolerate the long waits for a system to boot. But to be honest with you? I’d rather have it boot up 15x faster, with the same functionality – rather than “the true way”.

    • tgr says:

      Have you actually bothered to read the post? I don’t think so. Of course I used systemd. How could I possibly write a post about it without having given it “a sincere shot”?

      But I can see beyond my laptop and beyond those “five minutes” to recognize the inherent design issues.

      Read the post, read my answer to Kurt’s comment below and you’ll know what I’m talking about.

      Since you’ve tried writing your own init scripts (and I can’t see why that would be so hard, most sysv init scripts are as straightforward as shell scripts get), you must also know how flexible they are, especially compared to systemd’s ini-style unit files.

      As for boot time, I fully agree that faster is usually better, especially on desktops – but I’ve already said in the post that systemd is a viable alternative for desktop/embedded systems. On servers though, or even in the so called “cloud”, if your availability depends on booting in 30 seconds instead of 90, you’re most certainly doing something wrong.

      And it’s not “faster with the same functionality”. It’s faster, but without the flexibility.

  11. Kurt Aind says:

    I’ve been a Linux admin for 10 years, and I don’t know how anyone can claim that System V init is fine, or the different shell scripts are aproachable and manageable. The are not. And they are fragile.

    Not all of the features of SystemD are needed on servers. But what it has is perfect knowledge of the state of the services and processes. That’s enough for me for a server init system. Nothing else does it.

    • tgr says:

      I’d say one must be comfortable with shell scripting to work as anything other than a [very] junior sysadmin. I’m not saying scripts are the be all and end all of system management, but an important part of it.

      The init scripts in major distros are actually quite easy to read, understand and modify if necessary. I think you’re missing the point: what’s actually fragile is the startup process (as in: procedure) of some daemons. And systemd is not going to fix that (though it may – at some point in time – force the developers of such daemons to review their startup/shutdown code).

      Once again: what these scripts give you that systemd doesn’t is flexibility to modify the whole boot process, or the startup/shutdown process of any daemon on the spot.

      systemd is easy to use if you look at it – and that statefulness you mentioned is very seductive. When you give it a spin, it looks perfectly useful. But that’s the difference between installing software on your laptop to play around with it and actually using it in a large production deployment: when something goes wrong and you need to tweak, you’re going to wish for that intrinsic flexibility of sysv-style init scripts.

      Do I think that the current sysv init shell scripts are perfect? Far from it. Can they be refactored, standardized and improved? Sure.

      Not that it matters – in a few years, we’ll probably be using systemd anyway, as major distros seem hell bent on adopting it.

      BTW: I’ve seen some really bizarre claims around, e.g. that I had written in this post that it was impossible to launch shell scripts from systemd or that you had to change the code and recompile to configure services.

      Obviously, those who make such ridiculous claims haven’t even read the post, but let me point it out anyway: yes, you can give a shell script as an argument to ExecStart= in a systemd unit file. But that’s not the recommended way of launching services and it was never the goal of systemd to be a simple shell script launcher. systemd is designed to be monolithic and take over as much of the service startup process as possible.

  12. Pi says:

    I’m really rather fond of Runit. runsvdir‘s stage 2 is the init for my FreeBSD jails.

  13. Pingback: systemadm – simpler way to manage the service files for systemd | Linux and Photography Blog

  14. tgr says:

    (comments disabled after one year)

Comments are closed.