Seaweed KVCache: A Shared LMCache L2 That Outlives Your Inference Fleet

When an inference host restarts, scales down, or gets replaced, LMCache’s fastest L2 options — process memory and local NVMe — go with it. Every KV block that host had cached has to be recomputed by whichever request needs it next. Seaweed KVCache removes that trade-off: a shared, persistent L2 backend for LMCache, built on SeaweedFS, that lives independently of any single inference process. Restart a host, redeploy the fleet, or swap the hardware entirely — the cache is still there, and every other inference host can already see it.

It isn’t slow, either. On qualified RDMA hardware, one connector loaded cached KV blocks at 5,629 MiB/s over RoCE RC — about 5.5 GiB/s, and roughly 8.9x what the same topology moved over plain HTTP — with no RDMA sidecar or gateway process to operate; the connector talks RDMA directly from inside LMCache. And it isn’t capped by one machine: across an 8-host test on Azure, aggregate load throughput scaled 4.2x and save throughput 5.1x over a single host, just by adding storage and connector hosts together.

And a cache Volume doesn’t have to live entirely on your most expensive disks. Because it’s an ordinary SeaweedFS volume, it’s eligible for the same storage tiering that ages other SeaweedFS data from NVMe or SSD down to HDD, erasure coding, or cold remote storage — so you can afford to keep far more history than DRAM or NVMe alone would let you.

The payoff

Four things fall out of that architecture:

• You stop paying to recompute cache on every restart. Filer and Volume restarts keep every live key a hit; only a Volume you’ve deliberately retired shows up as a miss — never a stale read.
• You get RDMA throughput without operating RDMA infrastructure. The connector talks RC or DC directly inside the LMCache process — no sidecar, no gateway hop. In hardware qualification with LMCache’s own bench l2 command, that got one connector to 5,629 MiB/s over RoCE RC and 3,287 MiB/s over DC, against 636 MiB/s for the same workload over plain HTTP.
• You scale capacity and throughput by adding boxes, not by re-architecting. On Azure with plain HTTP transport, growing storage and connector hosts together from one pair to eight took aggregate save throughput from 619 to 3,156 MiB/s (5.1x) and load from 1,081 to 4,486 MiB/s (4.2x).
• You can afford to keep a much bigger, much older cache than DRAM or NVMe alone would let you. A cache Volume is an ordinary SeaweedFS volume with a disk-type tag, so it can use SeaweedFS Enterprise’s storage tiering the same way any other collection does: land fresh KV blocks on NVMe or SSD, then let a tiering profile age quiet volumes down to HDD, erasure-coded HDD (about 1.4x overhead for roughly the durability of 3x replication), or a remote cold store as they stop being reused — instead of deleting them the moment they stop fitting on your fastest disks.

Eviction below that ladder — TTL, delete, disk-pressure retirement — is also just a storage-level operation, not another system to run, and the same namespace stays visible through S3 if you ever want to inspect or manage it with ordinary tools.

What survives a restart

Restarting or losing a storage component costs you time, never correctness. A Filer, Volume, or Master restart never turns a live key into a stale read — you still get a hit, or if the Master genuinely can’t answer, a clean error you can retry or recompute against. The only way a key becomes a miss is if the Volume holding it was deliberately retired.

None of this needed a new cache database or a new on-disk format: LMCache keys are just Filer object paths, and values are just Needle data SeaweedFS already knows how to recover after a restart. Durability here is inherited, not bolted on.

Verified performance

Two topologies. Exact pinned runtime and server revisions for the single-node run are recorded in VALIDATION-PROVENANCE.json if you want to reproduce them.

Single node — one connector, one Filer, one Rust Volume on local NVMe, 100 GbE ConnectX-5 RoCE, LMCache’s bench l2 command: 32 values of 4 MiB per request, two requests in flight.

Direct RDMA isn’t shaving overhead off an already-fast path here — it’s removing the HTTP payload path entirely.

A five-minute sustained run (LMCache’s native multi-process soak) backs that up: RC averaged 6,170.5 MiB/s and DC averaged 3,518.5 MiB/s, reading 3.052 TB combined with zero payload errors, zero fallback reads, and no file-descriptor growth over the run.

Store throughput is also improving. The v0.1.0-preview.1 build you can install today stores about 700 MiB/s in the same 4 MiB same-host workload. A write-path change merged to main since that release — moving payload copy and CRC work outside the Needle append lock, with no change to the on-disk format or to Load throughput — pushed median Store to 1,095.3 MiB/s across six independent runs, roughly 56% higher, with zero active RDMA staging left behind after writing 25.8 GB of Needle data. That build hasn’t shipped in a preview release yet — treat 700 MiB/s as today’s number and 1,095.3 MiB/s as what’s coming.

Multi-node — Azure westus2, one Master/Filer host plus matching sets of Volume hosts and connector hosts (Standard_L8s_v4, local NVMe), HTTP transport, 4 MiB values. This is the scaling story once you’re past a single box:

One pair to eight scaled save 5.1x and load 4.2x — over plain HTTP, with RDMA not even in the picture. If you need more throughput or capacity, the lever is the same either way: add storage and connector hosts. This run used HTTP on different hardware than the single-node RDMA numbers above, so don’t cross-compare the absolute numbers between the two tables — each is evidence for its own claim (RDMA speed on one box; horizontal scaling across many).

Where it sits among L2 options

Pick Seaweed KVCache when the L2 has to be shared, has to survive a host being replaced outright, and you still want a direct data path on whatever RDMA hardware you have.

What you’ll need

• A working LMCache installation — the connector loads inside the LMCache process; it isn’t a separate service you stand up next to it.
• SeaweedFS Master, Filer, and Volume binaries (from a server image or release build). For RC/DC, a Rust Volume built with the RDMA features.
• gh CLI access to seaweedfs/artifactory — the wheel and config bundle ship as release assets on a prerelease, not on public PyPI.
• Python 3.12 for the wheel used below (seaweedkv_native-0.1.0-cp312-cp312-linux_x86_64.whl). The release contains a CPython/architecture-specific wheel, so grab the one matching your interpreter and platform from the release manifest if you’re not on cp312/linux_x86_64.
• For RC or DC: an RDMA-capable NIC (ConnectX-5 is the qualified card) with kernel drivers and firmware already part of your host image.

Step 1 — Install the connector

A release contains the wheel (it bundles the C++ extension and its Rust provider library), configuration examples, a release manifest, and SHA-256 checksums.

RELEASE=kvcache-filer-native-v0.1.0-preview.1
mkdir -p seaweed-kvcache-preview && cd seaweed-kvcache-preview
gh release download "$RELEASE" --repo seaweedfs/artifactory \
  --pattern 'LMCACHE-*' \
  --pattern 'REGISTRY-*' \
  --pattern 'RELEASE-NOTES.md' \
  --pattern 'SHA256SUMS' \
  --pattern 'VALIDATION-*' \
  --pattern 'seaweedkv-native-config-*.zip' \
  --pattern 'seaweedkv_native-*.whl'
sha256sum -c SHA256SUMS
sha256sum -c REGISTRY-SHA256SUMS
python3.12 -m venv .venv
. .venv/bin/activate
pip install ./seaweedkv_native-0.1.0-cp312-cp312-linux_x86_64.whl
python -c 'from seaweedkv_native import SeaweedKVNativeConnector; print("ok")'

Verify both checksum files before installing anything — that’s what confirms the wheel, and later the container images, match what was actually validated. If you deploy via containers, pull by the digest in that release’s REGISTRY-IMAGES.json rather than the mutable preview tag, so a later push to the same tag can’t silently change what’s running under you.

The wheel and config bundle are on the kvcache-filer-native-v0.1.0-preview.1 prerelease; it’s not on public PyPI. The release installation guide covers image-based, bare-metal, and Kubernetes deployment if bare metal below isn’t your target.

Step 2 — Start the storage services

Use persistent directories for Master, Filer, and Volume data — that’s what makes the restart guarantees above hold in practice. Give cache Volumes a large size limit: at 4 MiB values, a 1 GiB Volume fills every 256 keys, and rolling to a new Volume that often costs about 15% of store throughput. Size Volumes for your real block size and expected key count, not the default.

weed master -ip=10.0.0.1 -port=9333 -mdir=/data/master \
  -volumeSizeLimitMB=30000

weed filer -master=10.0.0.1:9333 -ip=10.0.0.1 \
  -port=8888 -port.grpc=18888 -defaultStoreDir=/data/filer

weed-volume --master 10.0.0.1:9333 --ip 10.0.0.3 \
  --port 8080 --port.grpc 18080 --dir /data/volume --max 32 \
  --metricsPort 19327

These are the bare-metal equivalent commands — the same binaries the server image or release build installs. If you have RoCE hardware and want the RC/DC numbers above, build the Rust Volume with the RDMA features, fill in your host’s real device/GID configuration, and add:

weed-volume ... \
  --rdma.enabled --rdma.ip 10.0.0.3 --rdma.port 18980 \
  --rdma.kv-transport both --rdma.transient-volume-source true

Treat the RC/DC listener environment as part of the service definition — it has to be restored after every restart, not just set once by hand.

Step 3 — Point LMCache at it

HTTP needs only the Filer’s gRPC address:

{
  "type": "native_plugin",
  "module_path": "seaweedkv_native",
  "class_name": "SeaweedKVNativeConnector",
  "adapter_params": {
    "filer_grpc": "10.0.0.1:18888",
    "backing": "loader-http",
    "num_workers": 8
  },
  "max_capacity_gb": 8
}

• filer_grpc — your Filer’s gRPC endpoint; the only address HTTP mode needs.
• backing — the transport. Start on loader-http; switch to loader-rdma once your RC or DC client and Volume environment passes preflight.
• num_workers — the connector’s worker pool size.
• max_capacity_gb — the capacity ceiling LMCache enforces for this L2 tier.

Values aren’t fixed at 4 MiB — the connector supports variable sizes with an explicit ceiling for bounded registered memory. Two environment variables are worth tuning once RDMA is on: SW_RDMA_VFS_PIPES controls the transport pipe pool for one-sided RDMA stores (16 or 32 improved store throughput by 10-30% on the test host — a starting point, not a universal default), and SW_RDMA_KVCACHE_PUSH_PIPES controls RC/DC push reads separately.

LMCache’s L1 stays the memory tier, as usual. The connector doesn’t add a second process-local hot cache on top of it — every L2 load is checked against current Filer metadata before bytes come back, so you’re never trading a speed hack for a correctness bug.

Step 4 — Manage capacity once it’s in production

delete(keys) removes the Filer entries immediately, so the next load is a miss. The bytes themselves become reclaimable Needle tombstones, and the normal Volume vacuum compacts the .dat file behind them — no separate garbage collector to run. Filer Entry TTL covers time-based expiry the same way.

If you’d rather age cache data onto cheaper storage than delete it, SeaweedFS Enterprise’s volume tiering applies to a cache collection the same way it applies to any other: tag Volume servers by disk type (NVMe, SSD, HDD, or your own label), then bind your cache collection to a tiering profile whose landing tier is NVMe/SSD and whose ladder demotes volumes to HDD, then erasure-coded HDD, then a remote backend once they’ve been quiet long enough that you don’t expect a hit. A pressure rule can also demote the coldest volumes off a filling fast tier ahead of schedule, so your NVMe/SSD landing tier doesn’t need headroom for your whole retention window. Set your quiet-time thresholds to match your own reuse patterns — see the Volume Tiering guide for exact configuration.

If you need a hard guarantee that a cache collection never fills a disk, an optional Master policy retires the oldest eligible Volume once a disk crosses a high watermark — marks it read-only, drops it from the location map, deletes it. It’s disabled by default, scoped to an explicit collection allowlist, requires a non-zero Volume TTL, and only ever touches non-replicated (replication=000) cache Volumes — it can’t reach your other data. It drops a whole old Volume rather than copying hot survivors out first, so set your allowlist and TTL with that in mind.

One operational note: if a retirement is interrupted after a Volume’s .vif records the intent but before cleanup finishes, the Volume comes back read-only, not writable — the Master won’t return it to the write set on its own. Check its state and retry or remove it by hand.

Seaweed KVCache gets you a shared L2 whose contents are ordinary, persistent, S3-visible SeaweedFS objects — reachable over HTTP anywhere, and over direct RC or DC RDMA wherever the hardware allows.

We’re looking for LMCache users and maintainers to run this against real KV-block size distributions, tell us where it breaks, and help shape the path to an upstream-supported integration.

Sources: LMCache native connectors, LMCache secondary storage, LMCache benchmarking, and SeaweedFS architecture.