Automatic EC Vacuum — Internals, API & Limits

Technical reference for Automatic EC Vacuum — the detection and compaction phases, safety guarantees, the full configuration table, worker flags, Helm values, health checks, and metrics to plan around.

The worker collects all EC shards for a target volume into its local -workingDir, vacuums the data locally, and then distributes the compacted shards back to volume servers. Volume servers are not overloaded with the CPU and disk work of compaction; they only serve controlled shard transfers and receive the replacement shards.

What Can It Do?

• Detect Deleted Data: Identifies EC volumes where deleted needles are consuming significant space (configurable threshold).
• Compact Volumes: Removes deleted needles from EC shards, reclaiming storage space across the cluster.
• Offload Heavy Work: Performs shard compaction in the worker’s local working directory instead of on volume servers.
• Improve Efficiency: Reduces unnecessary parity shard overhead by eliminating deleted data before the shard is fully utilized.
• Reduce Volume Server Load: Keeps volume servers focused on normal serving work while workers handle vacuum CPU, temporary disk, and coordination.
• Rack-Aware Optimization: Maintains shard distribution across failure domains during compaction.

Why Do You Need It?

In erasure-coded systems, when files are deleted, the space they occupied in the EC shards remains allocated. Over time, this creates “storage waste”:

For example, a 100GB EC 10+4 volume with 40% deleted data:

• Without vacuum: Still consumes 100GB + parity (140GB total with 10+4 ratio)
• After vacuum: Compacted to ~60GB + parity (~84GB total), a 40% space savings

How Does It Work?

The EC Vacuum worker runs as a scheduled background task on your cluster. When a volume crosses the deleted-data threshold, the worker pulls the full EC shard set into local storage, compacts it, verifies the result, and places the compacted shards back onto the cluster.

Detection Phase

The worker periodically scans EC volumes in the cluster:

1. Analyzes each EC volume’s shard composition
2. Calculates the ratio of deleted needles vs. total data from the live generation’s .ecx / .ecj (downloaded into the detector’s own scratch)
3. Compares against the configured threshold (default: 20%)
4. Only processes volumes that exceed the threshold, skipping any volume that already has an active task

Compaction Phase

When a volume triggers vacuum:

1. Analyze: Determines which shards can be safely compacted
2. Collect: Streams all required EC shards from volume servers into the worker’s local -workingDir
3. Vacuum Locally: Removes deleted needles and rebuilds compacted shard files on the worker
4. Verify: Validates that compacted shards match the original data
5. Distribute: Stages the rebuilt shards on every shard-holding server as the next generation .v<N+1>, alongside the live generation
6. Switch: A two-phase-commit protocol flips each server’s .active pointer to the new generation — atomic, with rollback
7. Cleanup: Prior generations are retained briefly for rollback, then pruned; a volume vacuumed to empty is torn down entirely

Technology Advantages

• No in-place volume-server vacuum: Volume servers are not asked to compact EC shards on their own disks. They serve shard data to the worker and receive compacted replacements.
• Resource isolation: CPU, temporary disk I/O, and shard rewrite work are concentrated on worker nodes that operators can size independently.
• Predictable cluster impact: Global and per-worker concurrency limits control how many volumes are vacuumed at once.
• Network-efficient placement: The worker can distribute compacted shards back to appropriate target nodes while preserving rack-aware EC placement.
• Operational simplicity: Adding more worker capacity increases vacuum throughput without changing the volume server role.

Safety Guarantees

• Only proceeds when sufficient healthy shards exist (>= data shard count)
• Validates data integrity before replacing original shards
• Keeps original shards available until compacted replacements are verified and distributed
• Shard distribution follows the same rack-aware rules as EC repair

Configuration

EC Vacuum runs as a plugin worker with configurable thresholds:

You can also filter vacuums by collection (collection_filter) to focus on specific data sets. Because effective parallelism is one job per worker, vacuum throughput scales directly with EC worker count — up to the global concurrency limit.

Deployment

EC Vacuum runs as a plugin worker process, integrated with your EC infrastructure. Start one or more weed worker processes that connect to the admin server.

Starting a Worker

EC Vacuum is automatically included when the erasure_coding handler is enabled:

# Start a worker for EC tasks (includes EC encoding, EC repair, and EC vacuum)
weed worker -admin=admin.example.com:23646 -jobType=erasure_coding \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=2

# Start a worker handling all heavy tasks
weed worker -admin=admin.example.com:23646 -jobType=heavy \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=4

# Start a worker handling all available task types
weed worker -admin=admin.example.com:23646 -jobType=all \
  -workingDir=/var/lib/seaweedfs-plugin

Key Options

Production Recommendations

• Run at least 2 worker instances for high availability
• Allocate sufficient -workingDir disk space for the largest concurrent EC vacuum job, including collected input shards and compacted output
• Set metricsPort to monitor vacuum progress and troubleshoot issues
• Consider grouping EC workers by region/data center to minimize cross-network transfers
• Vacuum works alongside EC repair without coordination

Worker Disk Sizing

A single ec_vacuum job stages all of the following under <workingDir>/task_scratch:

Budget up to ~4 × volume size per job (≈120 GB for 30 GB volumes). Because the erasure_coding worker bundle also serves erasure_coding, ec_repair and ec_bitrot_scrub, one worker can run several volume-staging jobs at once — bound by -maxExecute (and remote_vacuum under -jobType=all/heavy). Budget maxExecute × per-job per worker: roughly 240–360 GB for 30 GB volumes.

Volume servers need transient space too: the staged .v<N+1> generation sits beside the live generation until the switch commits, and prior generations are retained briefly for rollback.

Throughput and Throttling

• Effective parallelism = number of EC-capable workers × 1 job each, capped by global_execution_concurrency (default 8). Raise the cap only while watching volume-server disk and network.
• The ec_vacuum data path — shard download (CopyFile) and staging upload (ReceiveFile) — is not throttled by -volume.maintenanceMBps or -volume.compactionMBps; those flags cover server-to-server maintenance copies and regular volume compaction only. Control ec_vacuum load via job concurrency.

Why a Volume Is Skipped

Detection skips a volume when:

• Its deleted ratio is below ec_deleted_ratio_threshold (default 20%)
• Any shard is missing, or shard holders have not converged on one generation (including servers running binaries without the versioned protocol)
• Another task is already active on that volume (one job per volume cluster-wide)
• The last successful detection ran within ec_min_interval_seconds (default 10 h)
• A collection_filter excludes its collection, or the enterprise license is missing

Each detection run reports skipped_below_threshold, skipped_missing_shards, and failed_volumes counts in the activity log — check there first when expected volumes are not picked up.

Kubernetes

Workers are supported in the SeaweedFS Helm chart:

worker:
  enabled: true
  replicas: 2
  jobType: "heavy"
  maxExecute: 2
  workingDir: "/var/lib/seaweedfs-plugin"
  metricsPort: 9327

Health Checks

Workers expose HTTP endpoints when -metricsPort is set:

• /health — always returns 200
• /ready — returns 200 only when connected to admin
• /metrics — Prometheus metrics

Monitoring and Observability

EC Vacuum integrates with SeaweedFS observability:

How EC Vacuum Complements Other Enterprise Features

• With EC Repair: EC repair restores fault tolerance; EC vacuum improves efficiency
• With Self-Healing: Self-healing detects shard issues; EC vacuum handles scheduled compaction
• With Custom EC Ratios: Higher-density ratios like 20+4 benefit most from vacuum’s space reclamation

Requirements & limits

• Requires a valid SeaweedFS Enterprise license; runs as a plugin worker connected to the admin server.
• Only proceeds when sufficient healthy shards exist (>= data shard count).
• Vacuum is only triggered for volumes past the deleted-ratio threshold (default 0.20).
• Global and per-worker concurrency limits cap how many volumes are vacuumed at once (defaults: 8 global, 1 per worker).