Product Quantization Persistence: RocksDB with LZ4 compression

A high-density persistence stack for billion-scale vector search, utilizing PQ for 96% memory reduction and RocksDB with LZ4 for high-throughput disk I/O.

This architecture eliminates the RAM-only constraint for massive embedding deployments. Product Quantization (PQ) decomposes 512-dimension vectors into compact 8-bit codes, while RocksDB manages the resulting key-value pairs using its Log-Structured Merge-tree (LSM) engine. By enabling LZ4 compression, the system maintains a 400 MB/s write ceiling and minimizes disk amplification (often achieving a 2.5:1 ratio). It is the go-to configuration for balancing sub-20ms query latency with the cost-efficiency of NVMe storage: perfect for production environments where data scales faster than hardware budgets.

https://milvus.io/docs/product_quantization.md