Quadlet: Running Podman containers under systemd

Quadlet lets you run your Podman containers as systemd services. This is especially useful for running containers in the background and automatically starting them after a server reboot.

Running Podman containers under systemd isn't new. Indeed, this was supported by Podman for a long time with the command podman generate systemd. But this command now shows a deprecation warning to migrate to Quadlet.

For some months, I was too lazy to do that migration on my home server. Why even touch a working system? But now that I finally found time for it, I really appreciate Quadlet! I think that Podman finally has a Docker Compose alternative which is even more flexible and powerful!

In this blog post, I will explain how to use Quadlet with rootless Podman and migrate from the old method with podman generate systemd.

Landscape mode recommended on mobile devices

The deprecated method

Let's see how the old method worked before comparing it to Quadlet. You can skip to the Quadlet section though.

First, you had to create a container. In an older post about Containerized PostgreSQL with rootless Podman, I created a container with a command similar to the following:

podman create \
  --name test-db \
  -p 5432:5432 \
  -v ~/volumes/test-db:/var/lib/postgresql/data:Z \
  -e POSTGRES_PASSWORD=CHANGE_ME \
  --label "io.containers.autoupdate=registry" \
  docker.io/library/postgres:16

The command details are explained in the older post. The only option that isn't explained there is --label "io.containers.autoupdate=registry". This option enables updating the container image using podman auto-update which will be explained later in this post.

After creating the container, you can run the following command:

podman generate systemd test-db -fn --new

It creates the systemd service file container-test-db.service in your current path. The options of the command aren't relevant for now, but they are also explained in the older post.

To use this generated service file, you had to place it in the directory ~/.config/systemd/user. To enable and start it, you had to run the following command:

systemctl --user enable --now container-test-db

The problem

The problem with the old method is that it required you to run commands to…

1. create a container
2. generate a service file
3. move the service file if not already in the mentioned directory
4. enable the service

Especially the command for creating the container is often lengthy. This means that you had to create a shell script with these commands if you wanted to be able to rerun them later.

To reduce duplication, I created the following fish function to be called in my fish scripts that create the containers:

function podman-default-create
    set -l container_name $argv[1]

    podman create \
        --name $container_name \
        --replace \
        $argv[2..]; or return 1

    podman generate systemd --no-header --new --name $container_name >~/.config/systemd/user/container-$container_name.service; or return 1

    systemctl --user enable --now container-$container_name
end

You don't have to understand the details of the function above. What I want to demonstrate with it is that the old method was too hacky and involved the usage of redundant commands.

There must be an easier way, you might think. Especially if you experienced the convenience that Docker Compose provides. But this is not the only problem. The old method is very inflexible!

If you want to cutomize the service file and use all systemd features, you need to manually edit it after each generation!

Quadlet

Let's take a look at the new method with Quadlet.

First, you create the directory ~/.config/containers/systemd. Then, you place a .container file inside it. For example, here is the test-db.container file:

[Container]
Image=docker.io/library/postgres:16
AutoUpdate=registry
PublishPort=5432:5432
Volume=%h/volumes/test-db:/var/lib/postgresql/data:Z
Environment=POSTGRES_PASSWORD=CHANGE_ME

[Service]
Restart=always

[Install]
WantedBy=default.target

It is a normal systemd service file but with the special section [Container]. This section has many documented options. Almost all these options map to command line options that can be used to create a container with Podman (podman create). The ones that we are interested in for the example are the following:

• Image specifies the image (with tag) to use
• AutoUpdate=registry maps to --label "io.containers.autoupdate=registry" (explained later in this post)
• PublishPort maps to -p
• Volume maps to -v
• Environment maps to -e

It is important to use the systemd specifier %h instead of ~ for the user home directory.

In the [Service] section, we use the Restart option and set it to always to always restart the container (unless stopped manually).

To automatically start the container on boot, we set the WantedBy option in the [Install] section to default.target.

Since we use user services for systemd, we have to enable the linger for our user to start the containers without the user being logged in:

loginctl enable-linger

For systemd to discover the new service file, run systemctl --user daemon-reload. Now, you can start the container with systemctl --user start test-db.

You can check the status of the container service by running systemctl --user status test-db. You can also verify that the Podman container is running by running podman ps. You should find the container systemd-test-db.

The container has the name of the service file (test-db.container without the .container extension) prefixed by systemd- to avoid collisions with containers not managed by systemd. But you can manually set the name of the container using the ContainerName option in the [Container] section.

Is it any better?

My first impression was: "Well, now I have to map all the podman create options to their equivalents in the [Container] section. Where is the benefit?".

But after migrating all containers, I found the following benefits:

• You have only one file (the container service file) instead of a script that generates a service file.
• You can use all options possible in systemd. All options that you know about in the [Unit] and [Service] sections are supported. For example, you can specify a command to run before starting the container by setting the StartExecPre option. No more manual editing of generated files anymore!
• Rather subjective: Writing configuration files is easier than writing and debugging shell scripts.
• It is easier to handle dependencies as we will see in the next section.

Dependencies

Let's assume that we have an app container that depends on the database container that we created.

You want the database container to be automatically started when the app container is started. You also want to make sure that the app container is started after the database container. Otherwise, the app container might fail to start.

How can we express this dependency?

Let's use OxiTraffic as an example (shameless plug 😅).

Here is the container service file oxitraffic.container that should be placed in ~/.config/containers/systemd:

[Container]
Image=docker.io/mo8it/oxitraffic:0.9.2
AutoUpdate=registry
Volume=%h/volumes/oxitraffic/config.toml:/volumes/config.toml:Z,ro
Volume=%h/volumes/oxitraffic/logs:/var/log/oxitraffic:Z

[Unit]
Requires=test-db.service
After=test-db.service

[Service]
Restart=always

[Install]
WantedBy=default.target

The new section is [Unit]. We set the Requires option to test-db.service to only start the app when the database is started. We also set the After option to make sure that both containers aren't started in parallel.

Note that we use test-db.service when referencing this container service and not test-db.container.

• test-db.container is the file name.
• test-db.service is the service name.
• systemd-test-db is the default container name.

For the app to communicate with the database, a network should be added to both containers with the Network option in the [Container] section, but networking is out of the scope of this post.

Too many files?

In our example, we created two files, one for the app container and one for the database container. Does this mean that multi-container apps are more complex with Quadlet because you can't just put them in one file like with Docker Compose?

It depends on how you define complexity in this context. Does splitting content over multiple files always result in a higher complexity?

For me, it is more complex to have everything in the same file. I had to maintain Docker Compose files with hundreds of lines and dozens of containers… That wasn't fun! Having each container in its own file has less mental overhead for me because I just have to think about this single container when I am in its file. Of course, you need to specify its dependencies on other containers, but you don't need to think about the details of these other containers.

The Docker Compose file of Mailcow is a terrifying example of huge Docker Compose files.

So we need multiple files. But we should still group related ones together! Quadlet supports placing unit files inside directories in the ~/.config/containers/systemd directory. For our example, you would create an oxitraffic directory and place both files inside it.

Updating images

Now, we have containers running in the background and automatically started after a server reboot. Wouldn't it be nice to have an easy method to update the images of these containers without running podman pull for every container and then restarting the updated ones?

For example, if a new image is uploaded for PostgreSQL 16 (with the image tag 16 that we used), then the image should be updated and the container should be restarted.

With Docker, you would need something like Watchtower. But Podman provides a tool out of the box!

If you set AutoUpdate=registry, you can just run podman auto-update and Podman will check if the registry has a newer image which is compatible with the used tag. In that case, the image will be pulled and the container will be restarted. It is that easy 😍

Of course, this could be dangerous if you use a tag like latest for OxiTraffic instead of a concrete version like 0.9.2. Because the next version pushed to the latest tag might include a breaking change! It is even more dramatic if you use the latest tag for the PostgreSQL image because manual migrations are always required when upgrading PostgreSQL to a new major version.

Therefore, always use a tag that can't lead to a breaking change! Trust me, this is not only a problem with Podman updates. I learned this the hard way when trying to deploy Docker containers a while ago that used the latest tag.

Personally, I run podman auto-update manually on the server every couple of days to see what has been updated and make sure that the containers are still healthy afterwards.

What about podman-compose?

There is the Python script podman-compose which runs Compose files with Podman. But I don't consider it a long-term alternative to Docker Compose for multiple reasons:

• podman-compose is just a translation layer between the Compose spec and Podman with systemd. Why yet another layer of abstraction that sacrifices flexibility? It doesn't let you use all features of systemd.
• I consider Python scripts a prototype because of Python's interpreted nature. Official Podman projects are written in a compiled language like Rust or Go.
• It is not actively maintained. The last commit was 5 months ago.

Quadlet aligns much better with the rootless, daemonless design of Podman.

If you are stuck with Compose files and would like to try out Quadlet, check out podlet that can help you during the migration.

Further resources

Take this post as an introduction. I highly recommend reading the man page podman-systemd.unit to get a deeper understanding of Quadlet. You don't need to read the documentation of every supported option though.

Quadlet doesn't only work with containers. It can also manage pods, networks and volumes (see man page).

If you are new to writing systemd unit files (like me), I also recommend checking out the man pages systemd.unit and systemd.service.

podlet is a wonderful Rust tool that can help you during the migration. It can create Quadlet files out of Podman commands or even (Docker) Compose files.

Check out the similar blog post on blog.while-true-do for another perspective and a second example. It is my favorite blog related to Linux 🥰

Finally, if you want to see my migration as an example, then you can compare before and after.