Systemd conditional restart / reload


Comments [0]

A company I contract for recently upgraded their servers to Centos 7, which means we’re now in the world of Systemd instead of SysVinit.

It turns out that Systemd doesn’t have facilities to handle conditional restarts of a service. Reloads can be made to work with multiple ExecReload commands.

I did a decent amount of digging and in all the threads I found one maintainer, in particular, was staunchly against adding the capability to Systemd. I can only posit he doesn’t do work in the real world where this is a necessity and not just a nice to have. But seriously, would it be that hard to ExecRestartPre?

A solution was needed and this is what I came up with. I’m putting it out there in case anyone else might find it useful.

Our needs were fairly simple, we run mod_perl on our web servers and we can test our preloader to make sure it compiles before moving forward with the web server restart. I tried to keep things generic enough that you can use any command line command as a test. One or more test commands can be defined for each service.

It has two components a (JSON based) config file and a script to replace systemctl.

Config file:

#------------------------------------------------------------------------------
# Config file used for replacing systemctl command so that we can try to do
# conditional reload/restarts for certain services.
#
# You can use %%TAGNAME%% in scalar vars and that will be replaced by the root
# level config item of the same name when the config is parsed.
#------------------------------------------------------------------------------
{
  "systemctl" : "/usr/bin/systemctl.dist",
  "perl"      : "/usr/bin/perl",
  "services" : {
    "httpd" : {
      "configTests" : [
        "%%perl%% -cw /web/virtual-servers/myserver.com/etc/preloader",
      ],
    },
  },
}

systemctl replacement script:

#!/usr/bin/perl

use JSON ();
use Data::Dumper;
$Data::Dumper::Indent = 1;

my $cfg = readConfig('configFile' => '/location/of/config/file/systemctl.conf');

#------------------------------------------------------------------------------
# Match service identifier argument with what's in the config.  We'll use the
# abbreviated form in the config file 'httpd' as opposed to 'httpd.service',
# but we need to identiy either as both are valid.
#------------------------------------------------------------------------------
my $service = matchService($cfg, $ARGV[1]);

#------------------------------------------------------------------------------
# If we are not dealing with reload/restart and config defined service then 
# just run the command like normal.
#------------------------------------------------------------------------------
if (
    ($ARGV[0] ne 'reload' && $ARGV[0] ne 'restart') || !$service
) {
    my $rv = system($cfg->{'systemctl'}, @ARGV);
    exit($rv >> 8);
}

#------------------------------------------------------------------------------
# Go through the list of configTests and build an array of commands we need
# to run before attempting a restart.  We'll join all of these using && and
# then execute.  If a command fails then none of the commands that come after
# the failed one in the chain will be executed.
#------------------------------------------------------------------------------
my $cmds = [];
foreach my $pl (@{$cfg->{'services'}{$service}{'configTests'}}) {
    push(@$cmds, $pl);
}
push(@$cmds, $cfg->{'systemctl'} . ' ' . join(' ', @ARGV));

my $cmd = join(' && ', @$cmds);
my $rv  = system($cmd);

exit($rv >> 8);

sub matchService {
    my $cfg     = shift;
    my $service = shift;

    foreach my $k (keys %{$cfg->{'services'}}) {
        if ($service eq $k || $service =~ /^$k/) {
            return($k);
        }
    }

    return(undef);
}

sub readConfig {
    my $args = {
        'configFile' => '',
        @_
    };

    my $fh;
    if (!open($fh, "$args->{'configFile'}")) {
        die "Can't open config file ($args->{'configFile'}) ($!)\n";
    }

    local $/ = "";
    my $json = <$fh>;
    close($fh);

    my $cfg = {};
    eval {
        $cfg = JSON::from_json($json, { 'relaxed' => 1 });
    };

    if ($@) {
        die "Parsing config file failed ($@)\n";
    }

    translateConfig($cfg, $cfg);

    return($cfg);
}

sub translateConfig {
    my $cfg    = shift;
    my $branch = shift;

    if (ref $branch eq 'ARRAY') {
        foreach my $item (@$branch) {
            if (!ref $item) {
                translateConfig($cfg, \$item);
            } else {
                translateConfig($cfg, $item);
            }
        }
    } elsif (ref $branch eq 'HASH') {
        foreach my $item (keys %$branch) {
            if (!ref $branch->{$item}) {
                translateConfig($cfg, \$branch->{$item});
            } else {
                translateConfig($cfg, $branch->{$item});
            }
        }
    } else {
        if ($$branch =~ /%%([^%]+)%%/) {
            my $tag = $1;
            if (exists $cfg->{$tag}) {
                $$branch =~ s/(%%[^%]+%%)/$cfg->{$tag}/;
            } else {
                print STDERR "Found translatable item without matching top ",
                    "level config var $tag / $$branch\n";
            }
        }
    }
}

To use this, I renamed /usr/bin/systemctl -> /usr/bin/systemctl.dist and symlinked the PERL script to /usr/bin/systemctl. Now it will catch reloads/restarts of our web server and perform the conditional tests before executing the actual systemctl command. If there is a failure it will print any output from the test command and exit.

I expect we’ll have to watch out for our sysmlink disappearing if Yum ever updates Systemd/systemctl, but that won’t be hard to deal with.

 



Comments are closed.