Understanding Docker's Modern Directory Architecture

In contemporary containerized infrastructure, Docker has evolved significantly from a monolithic tool into a sophisticated multi-layered architecture. On Linux systems, Docker's core data is strategically distributed across two primary directories, each serving distinct but complementary purposes in the container ecosystem.

The Docker Engine Management Directory: /var/lib/docker

This directory serves as the high-level management space for the Docker Daemon, orchestrating container lifecycle operations, configuration management, and persistent storage solutions. Let's explore its key components in detail:

Containers Metadata and Logs
This subdirectory houses container configuration files (such as config.v2.json), runtime state information, and standard output logs from containers (typically stored in JSON format as stdout/stderr). These files are crucial for debugging, monitoring, and understanding container behavior over time.

Volumes for Persistent Data
By default, user-created named volumes store their data within the volumes/ subdirectory. This represents the core area for container persistent data, ensuring that critical application data survives container restarts and recreations. Understanding volume management is essential for production deployments where data persistence is non-negotiable.

Image Metadata Storage
This component stores image manifests, layer relationship indices, and local tag information. Importantly, it does not directly store the binary filesystem layers of images themselves—those reside elsewhere in the architecture. This separation of metadata from actual content is a key design principle that enables efficient image management.

BuildKit Build Cache
When using Docker BuildKit for building images, intermediate cache data is stored here. This cache significantly speeds up subsequent builds by reusing unchanged layers, making it an important consideration for CI/CD pipelines and development workflows.

Network Configuration Management
This area manages container virtual network configurations and states, including Bridge, Overlay, and Macvlan networks. Proper network configuration is fundamental to container communication and service discovery in distributed systems.

The Containerd Runtime Directory: /var/lib/containerd

Containerd represents an industry-standard container runtime that Docker now employs as its core underlying component. This directory handles the most fundamental storage and execution logic, operating at a lower level than the Docker Engine:

Content Storage for Image Layers
This is where the actual compressed image layer files (Blobs) are stored after being pulled from registries. This represents the true physical location of image data, making it one of the most storage-intensive components of the Docker ecosystem.

Snapshotter and Filesystem Management
This is typically the largest space-consuming component (particularly when using overlayfs). The snapshotter is responsible for decompressing and stacking compressed image layers to form the root filesystem required for container execution. It also manages the writable layer for each container, enabling the copy-on-write semantics that make containers efficient.

Metadata Database
Containerd maintains a BoltDB or similar database that records associations between image content, snapshots, and namespaces. This database is critical for maintaining consistency and tracking relationships between various container components.

Input/Output Stream Management
This component handles the input and output streams for container processes, enabling proper communication between containers and the host system or other containers.

Why Migrate Both Directories Simultaneously?

Understanding the architectural evolution of Docker is crucial for effective migration:

Historical Context
In earlier Docker versions, image storage was located directly at /var/lib/docker/overlay2. However, in modern architectures, Docker has delegated底层 storage and runtime responsibilities to containerd, creating a clear separation of concerns.

Clear Division of Responsibilities
The /var/lib/docker directory handles upper-level business logic such as volume management and network orchestration, while /var/lib/containerd manages large-scale binary file storage at the底层 level. This separation enables each component to optimize for its specific use case.

Storage Distribution Reality
The majority of disk space consumption comes from containerd's image snapshots and layer data. Migrating only the docker directory often fails to resolve root partition exhaustion issues, as the bulk of storage remains untouched in the containerd directory.

Ensuring Consistency
By migrating both directories simultaneously to a data disk using symbolic links or mount points, you ensure the integrity of the container environment. This approach prevents startup failures or configuration conflicts that could arise from path inconsistencies.

Step-by-Step Migration Procedure for Linux Systems

Many Nvidia LLM frameworks provide NGC images that require substantial storage space. When using GPU rental platforms with Docker, you may encounter insufficient system disk space. The following comprehensive procedure ensures safe and effective migration:

Step 1: Gracefully Stop Docker Services

Before any migration, all Docker-related services must be stopped to prevent data corruption:

sudo systemctl stop docker
sudo systemctl stop docker.socket
sudo systemctl stop containerd

Stopping services in this order ensures that dependent services are terminated properly, preventing potential file locking issues during migration.

Step 2: Migrate Data Using rsync

We use rsync instead of mv for several important reasons:

  • Permission Preservation: rsync maintains file permissions, ownership, and hierarchical structure
  • Recovery Capability: If the migration is interrupted, original data remains intact
  • Verification: rsync provides checksums and progress information for validation

Execute the following commands:

# Create the migration target directory
sudo mkdir -p /mnt/data/docker-data

# Migrate the Docker directory
sudo rsync -avz /var/lib/docker/ /mnt/data/docker-data/docker/

# Migrate the containerd directory
sudo rsync -avz /var/lib/containerd/ /mnt/data/docker-data/containerd/

The -avz flags enable archive mode (preserving permissions), verbose output, and compression during transfer, optimizing both safety and performance.

Step 3: Clean Original Directories and Create Symbolic Links

After successful data migration, remove the original directories and create symbolic links:

# Remove original directories
sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd

# Create symbolic links pointing to new locations
sudo ln -s /mnt/data/docker-data/docker /var/lib/docker
sudo ln -s /mnt/data/docker-data/containerd /var/lib/containerd

Symbolic links provide transparency to applications—they continue accessing the standard paths while data physically resides on the larger storage volume.

Step 4: Verify Symbolic Link Status

Before proceeding, verify that symbolic links were created correctly:

ls -ld /var/lib/docker /var/lib/containerd

Expected output should show entries similar to:

lrwxrwxrwx 1 root root 26 Apr 9 14:30 /var/lib/containerd -> /mnt/data/docker-data/containerd
lrwxrwxrwx 1 root root 22 Apr 9 14:30 /var/lib/docker -> /mnt/data/docker-data/docker

Note the 'l' at the beginning of each line (indicating a symbolic link) and the arrow (->) showing the target path.

Step 5: Restore Docker Services

With migration complete and verified, restart all Docker services:

sudo systemctl start docker
sudo systemctl start containerd

Verify that services started successfully:

systemctl status docker
systemctl status containerd

Post-Migration Verification and Best Practices

After completing the migration, perform these verification steps:

Container Operation Test
Run a test container to ensure Docker functions correctly:

docker run --rm hello-world

Storage Verification
Confirm that new containers use the migrated storage:

docker system df
df -h /var/lib/docker
df -h /var/lib/containerd

Performance Monitoring
Monitor I/O performance on the new storage location to ensure it meets your workload requirements. Consider using tools like iostat or iotop for detailed analysis.

Additional Considerations for Production Environments

Backup Strategy
Before any migration, create a complete backup of both directories. This provides a recovery point if issues arise during or after migration.

Storage Selection
Choose appropriate storage for your workload:

  • SSD storage for development environments requiring fast I/O
  • High-capacity HDD arrays for production environments with many images
  • Network-attached storage (NAS) for shared container environments

Monitoring Setup
Implement monitoring for disk space usage on the new location to prevent future exhaustion issues. Set up alerts at 80% and 90% capacity thresholds.

Documentation
Document the migration procedure and new storage locations for team members and future reference. Include rollback procedures in case migration needs to be reversed.

Conclusion

Migrating Docker directories using symbolic links is a reliable, reversible method for addressing storage constraints on Linux systems. By understanding Docker's modern architecture and following the systematic procedure outlined above, administrators can safely relocate container data to larger storage volumes while maintaining full functionality and compatibility.

This approach is particularly valuable for GPU-accelerated workloads, CI/CD systems with many concurrent builds, and development environments where multiple large images are frequently pulled and tested. The symbolic link method provides transparency to applications while enabling flexible storage management strategies.