Paper Details

Reference:

Siying Dong, Andrew Kryczka, Yanqin Jin, and Michael Stumm,
"Evolution of development priorities in key-value stores serving large-scale applications: The RocksDB experience",
In Proceedings 19th USENIX Conference on File and Storage Technologies (FAST'21), Online, Usenix Association, February, 2021, pp. 33–49.

Download:

PDF ; Talk Slides ; Talk

Abstract:

RocksDB is a key-value store targeting large-scale distributed systems and optimized for Solid State Drives (SSDs). This paper describes how our priorities in developing RocksDB have evolved over the last eight years. The evolution is the result both of hardware trends and of extensive experience running RocksDB at scale in production at a number of organizations. We describe how and why RocksDB's resource optimization target migrated from write amplification, to space amplification, to CPU utilization. Lessons from running large-scale applications taught us that resource allocation needs to be managed across different RocksDB instances, that data format needs to remain backward and forward compatible to allow incremental software rollout, and that appropriate support for database replication and backups are needed. Lessons from failure handling taught us that data corruption errors needed to be detected earlier and at every layer of the system.

Keywords:

Key-value stores, RocksDB, Large-scale distributed systems, SSD, Scalability

Reference Info:

ISBN: 978-1-939133-20-5

BibTeX:

@inproceedings(Dong-Fast-21,
    author = {Siying Dong and Andrew Kryczka and Yanqin Jin and Michael Stumm},
    title = {Evolution of development priorities in key-value stores serving large-scale applications: {T}he {RocksDB} experience},
    booktitle = {Proceedings 19th USENIX Conference on File and Storage Technologies (\textbf{FAST'21})},
    location = {Online},
    publisher = {Usenix Association},
    month = {February},
    year = {2021},
    pages = {33-49},
    isbn = {978-1-939133-20-5},
    keywords = {Key-value stores, RocksDB, Large-scale distributed systems, SSD, Scalability}
)