Stop Losing Data When Your Container Restarts

The upload folder that empties itself

Your application lets users upload profile photos. They're stored at /app/uploads inside the container. Users upload photos, they appear immediately, everything looks fine. Then the container restarts due to an OOM event, a deploy, or a node recycle. Every uploaded photo is gone. Users see broken images. The data was never persisted outside the container filesystem.

This is the most common data loss pattern in containerized applications, and it happens because "the container is running" feels like a stable state. It isn't. A container can be stopped and restarted at any time, and without volumes, anything written to the container filesystem during runtime is lost.

What actually persists across a container restart vs removal

There's an important distinction between restart and removal:

Container restart (docker restart, OOM restart, restart: unless-stopped policy): The container process restarts but the container's writable layer is preserved. Data written to the container filesystem survives a restart. This is why you might not notice the problem immediately — restarts look fine, but a container removal and recreation wipes everything.

Container removal and recreation (docker compose down && docker compose up, a deploy that replaces the container, a Kubernetes pod being rescheduled to a different node): The old container is gone, a new container is created from the image. The new container starts fresh — the writable layer from the old container is discarded.

In Kubernetes, pods are ephemeral by definition. A node going down, a node being drained for maintenance, a deployment rollout — all of these create new pods. If your application relies on the container filesystem for persistent data, it will lose data when pods are rescheduled.

Find every place your app writes to the filesystem

Before adding volumes, audit where your application writes data:

Log files: Does your application write log files anywhere in the container? Many frameworks default to writing logs to a file in the working directory or /var/log/. In containers, logs should go to stdout/stderr — let the container runtime collect them.

Uploaded files: User uploads, generated files, exports. These must be on a volume or external storage (S3, GCS, Azure Blob).

Caches: Application caches written to disk. If they're safe to recreate, this isn't a data loss concern — but a large cache rebuild on every pod start can cause performance issues. Consider whether the cache is better stored in Redis.

SQLite or embedded databases: If your application uses SQLite or an embedded H2 database for anything other than ephemeral test data, put the database file on a volume.

JVM artifacts: Thread dumps, heap dumps (-XX:+HeapDumpOnOutOfMemoryError defaults to writing in the working directory), GC logs. These are diagnostic, not business data, but losing them makes production incident investigation harder. Mount a volume for them.

Session data: Application sessions stored to disk. Use Redis or a database for session storage in containerized applications.

# Quick audit: what does your application write at runtime?
docker exec -it your-container inotifywait -m -r /app 2>/dev/null &
# Exercise the application for a few minutes, then check output

Mounting volumes for common write paths

User uploads:

services:
  app:
    volumes:
      - uploads:/app/uploads

volumes:
  uploads:

For production, prefer external object storage over a local volume. Local volumes don't survive node failures and can't be shared between multiple replicas. Use S3/GCS/Azure Blob and generate pre-signed URLs for uploads and downloads — your container becomes stateless.

Log files (when you can't switch to stdout):

services:
  app:
    volumes:
      - app_logs:/app/logs

volumes:
  app_logs:

Configure a log rotation mechanism alongside this, or your logs will fill the disk. Docker has built-in log rotation for stdout/stderr logs (--log-opt max-size=10m --log-opt max-file=3) — another reason to prefer stdout.

JVM diagnostic output:

services:
  app:
    environment:
      JAVA_OPTS: >-
        -XX:+HeapDumpOnOutOfMemoryError
        -XX:HeapDumpPath=/app/dumps
        -Xlog:gc*:file=/app/gc.log:time:filecount=5,filesize=20m
    volumes:
      - jvm_dumps:/app/dumps

volumes:
  jvm_dumps:

SQLite database:

services:
  app:
    volumes:
      - db_data:/app/data

volumes:
  db_data:

The read-only filesystem test

If you can run your container with a read-only filesystem, you've confirmed that no data is being written to paths that would be lost on recreation:

docker run --read-only --tmpfs /tmp your-image:tag

--tmpfs /tmp provides a writable in-memory tmpfs for /tmp, which is typically needed for application temp files. If the container starts successfully and operates correctly under --read-only, your persistent data is all on volumes.

In Kubernetes:

securityContext:
  readOnlyRootFilesystem: true
volumeMounts:
  - name: tmp
    mountPath: /tmp
  - name: uploads
    mountPath: /app/uploads
volumes:
  - name: tmp
    emptyDir: {}
  - name: uploads
    persistentVolumeClaim:
      claimName: uploads-pvc

Making the root filesystem read-only is also a security hardening measure — it prevents an attacker with code execution from writing malicious files to the container filesystem.

Backup strategy for volumes

A named volume that isn't backed up is not persistent storage — it's resilient-until-it-isn't storage. For production data:

#!/bin/bash
# Simple volume backup script
VOLUME_NAME="pg_data"
BACKUP_DIR="/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

docker run --rm \
  -v "${VOLUME_NAME}:/source:ro" \
  -v "${BACKUP_DIR}:/backup" \
  alpine \
  tar czf "/backup/${VOLUME_NAME}_${TIMESTAMP}.tar.gz" -C /source .

# Keep only last 7 backups
ls -t "${BACKUP_DIR}/${VOLUME_NAME}_"*.tar.gz | tail -n +8 | xargs -r rm

For PostgreSQL specifically, pg_dump inside the running container is better than a raw filesystem backup — it produces a consistent snapshot without requiring the database to be stopped:

docker exec postgres-container \
  pg_dump -U appuser mydb | gzip > "backup_$(date +%Y%m%d).sql.gz"

The operational checklist

For every service in your Compose or Kubernetes config, answer these questions:

1. Does this service write files to the container filesystem at runtime?
2. If yes, is that write path mounted to a named volume or external storage?
3. If the container is removed and recreated right now, what data is lost?
4. Is that data backed up?

Most teams find at least one answer that surprises them. Fix it before the next unplanned container restart finds it for you.