Ops Journal · Practical notes on running software in production

What actually survives a container rebuild: volumes, mounts and the data people lose

Published 2026-09-29 · 8 min read

The first time a container rebuild destroys data, it is usually a surprise, because the files were right there in the running container. Understanding why they vanished comes down to one distinction: what is stored in the container's writable layer, and what is stored outside it.

The writable layer is disposable, by design

Everything written inside a container that is not on a volume or bind mount lives in the container's own writable layer. That layer is tied to the container's identity. Remove the container and the layer goes with it. Recreate it from the same image and you get a fresh, empty layer.

This is not a bug. It is what makes containers replaceable. The consequence is that any state you care about must live somewhere that outlives the container.

There are three places it can live:

Kind Survives docker rm Survives docker compose up --build Managed by
Writable layer no no the container
Named volume yes yes Docker
Bind mount yes yes you
Anonymous volume yes, if not pruned yes Docker, unnamed

Checking what a running container actually uses

Before rebuilding anything, look at how the container is wired:

docker inspect --format '{{json .Mounts}}' myapp | python3 -m json.tool

The Mounts array lists every mount with its Type (volume or bind), its Source on the host, and its Destination in the container. If the path your application writes to does not appear in that list, its data is in the writable layer and will not survive the rebuild.

A quicker first pass, which is enough to spot the common case of a container with no mounts at all:

docker inspect --format '{{.Name}}: {{len .Mounts}} mounts' $(docker ps -q)

Named volumes are the safe default

A named volume is created and owned by Docker, and it persists until you explicitly remove it:

docker volume ls
docker volume inspect myapp_data

docker volume inspect prints the host path under Mountpoint. On Linux that is typically /var/lib/docker/volumes/myapp_data/_data, and being able to see it is useful for the next step — confirming the data is actually there before you tear anything down.

Two commands remove volumes, and both are destructive:

docker rm -v myapp            # removes the container and its anonymous volumes
docker compose down -v        # removes the project's volumes

The -v flag is the one that surprises people in compose. docker compose down on its own leaves named volumes alone; add -v and the database volume is gone. It is common to find -v in a script written by someone debugging, and it is worth grepping your scripts for it:

grep -rn "compose down" --include='*.sh' .

Bind mounts are explicit but easy to get subtly wrong

A bind mount maps a host path into the container:

services:
  app:
    volumes:
      - ./data:/app/data

The data lives at ./data relative to the compose file, so it survives everything. The subtle failure is the path: ./data is relative to the compose file's directory, not the directory you ran the command from. Run compose with -f /srv/app/docker-compose.yml from elsewhere and the relative path still resolves correctly, but a path typed by hand into a docker run -v does not — docker run -v data:/app/data creates a named volume called data, not a bind mount to a directory called data.

The tell is in docker inspect: a bind mount shows an absolute Source path, a named volume shows a name.

Anonymous volumes, and why they accumulate

A VOLUME instruction in a Dockerfile, or a -v /app/data with no host side, creates an anonymous volume with a random name. It survives container removal, which is deliberate — the intent is that docker run --rm on the same image reuses it. But nothing tracks which container it belonged to once the container is gone, so they accumulate:

docker volume ls -qf dangling=true

Those are volumes no container references. Pruning them is where data loss actually happens:

# destroys every unreferenced volume -- read the list first
docker volume prune

Always run the -qf dangling=true listing first and read it. A volume that looks dangling may be the one a stopped container needs, and docker volume prune does not ask twice.

Proving the data is where you think before a rebuild

The habit that prevents the incident: before any rebuild, confirm the data is present at the mount's host path.

# find the mountpoint of a named volume
MP=$(docker volume inspect myapp_data --format '{{.Mountpoint}}')
sudo ls -la "$MP" | head

If that directory is empty and your application claims to have written data, the writes are going into the writable layer and the next rebuild will lose them.

Backing up a volume properly

Copying files out of a volume is best done with a throwaway container that mounts it, rather than reaching into /var/lib/docker directly — the latter bypasses Docker's locking and can capture a torn view of a database volume:

docker run --rm \
  -v myapp_data:/data:ro \
  -v "$PWD":/backup \
  alpine tar -czf /backup/myapp_data.tar.gz -C /data .

The :ro matters: mounting read-only means the backup cannot corrupt the source, and it makes the intent explicit to anyone reading the command later.

The short version

Check docker inspect for the mount before rebuilding, never run compose down -v or volume prune without reading the list first, and put anything you cannot afford to lose on a named volume or a bind mount with an absolute path. The rebuild itself is safe; it is the cleanup commands around it that destroy data.