Skip to main content

Registry Maintenance

Guide for maintaining the private Docker registry, including tag cleanup and disk space management.

Overview​

Our private Docker registry (192.168.18.20:5000) stores all Docker images for fetcher and scheduler services. Over time, old tags accumulate and consume disk space.

Retention Policy​

We maintain the following retention policy:

  • Version Tags: Keep latest 3 versions (e.g., 2.0.3, 2.0.4, 2.0.5, v2.0.5)
  • Branch Tags: Always keep latest, main, sandbox
  • SHA Tags: Delete all 40-character SHA tags (redundant with version tags)
  • Other Tags: Keep for manual review

Automated Cleanup​

Quick Start​

# From infra/ directory - Dry run first (recommended)
cd infra
make docker-cleanup-tags
# Press Y or Enter when prompted

# Bypass prompt for actual cleanup
make docker-cleanup-tags DRY_RUN=false

# Alternative: Press 'n' when prompted during standard execution
make docker-cleanup-tags

Direct Script Usage​

# From infra/ directory

# Dry run (safe, shows what would be deleted)
DRY_RUN=true ./cleanup-docker-tags.sh

# Actual cleanup
./cleanup-docker-tags.sh

# Custom configuration
KEEP_LATEST_VERSIONS=5 DRY_RUN=false ./cleanup-docker-tags.sh

Configuration Options​

VariableDefaultDescription
REGISTRY192.168.18.20:5000Registry URL
KEEP_LATEST_VERSIONS3Number of version tags to keep
DRY_RUNtrueIf true, only shows what would be deleted

Example Output​

=========================================
  Docker Registry Tag Cleanup
=========================================

Registry: 192.168.18.20:5000
Keep latest versions: 3

=== Processing: faiht-fetcher-worker ===

Tag Statistics:
  Version tags: 8
  SHA tags: 19
  Protected tags: 2

Deleting SHA-based tags...
✓ Deleted: faiht-fetcher-worker:2c8c4e606f17...

Keeping latest 3 version tags:
  ✓ v2.0.5
  ✓ 2.0.4
  ✓ 2.0.3

Deleting old version tags...
✓ Deleted: faiht-fetcher-worker:2.0.2
✓ Deleted: faiht-fetcher-worker:2.0.1

Garbage Collection​

After deleting tags, you must run garbage collection to free disk space:

# SSH to the registry server
ssh faiht@lxc-docker-01

# Find the registry container
docker ps | grep registry

# Run garbage collection
docker exec <registry-container-name> bin/registry garbage-collect /etc/docker/registry/config.yml
Important

Tags are marked for deletion but disk space is not freed until garbage collection runs.

Automation (Cron)​

To run cleanup automatically:

# Edit crontab on lxc-docker-01
crontab -e

# Run weekly on Sunday at 2 AM
0 2 * * 0 cd /home/faiht/faiht2-mono/infra && ./cleanup-docker-tags.sh >> /var/log/docker-cleanup.log 2>&1

Manual Tag Management​

List all tags​

curl -s http://192.168.18.20:5000/v2/faiht-fetcher-worker/tags/list | jq -r '.tags[]'

Get tag digest​

curl -s -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  http://192.168.18.20:5000/v2/faiht-fetcher-worker/manifests/2.0.5 | \
  grep -i Docker-Content-Digest

Delete specific tag​

# Get the digest first
DIGEST="sha256:..."

# Delete the manifest
curl -X DELETE http://192.168.18.20:5000/v2/faiht-fetcher-worker/manifests/$DIGEST

Fix missing tag (v prefix issue)​

If a tag exists with v prefix but not without:

docker pull 192.168.18.20:5000/faiht-fetcher-worker:v2.0.5
docker tag 192.168.18.20:5000/faiht-fetcher-worker:v2.0.5 \
           192.168.18.20:5000/faiht-fetcher-worker:2.0.5
docker push 192.168.18.20:5000/faiht-fetcher-worker:2.0.5

Troubleshooting​

Tags not deleting​

  1. Check registry logs:

    docker logs <registry-container>

  2. Ensure deletes are enabled in registry config.yml:

    storage:
      delete:
        enabled: true

Missing dependencies​

# Install jq
sudo apt-get install jq

# Install curl
sudo apt-get install curl

Monitoring​

Check registry disk usage:

# On lxc-docker-01
du -sh /var/lib/docker/volumes/registry_data