Paper Details

Reference:

Muthukaruppan Annamalai, Kaushik Ravichandran, Harish Srinivas, Igor Zinkovsky, Luning Pan, Tony Savor, David Nagle, and Michael Stumm,
"Sharding the shards: managing datastore locality at scale with Akkio",
In Proceedings 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI'18), Carlsbad, CA, USA, Usenix Association, October, 2018, pp. 445–460.

Download:

PDF ; Talk Slides ; Talk

Abstract:

Akkio is a locality management service layered between client applications and distributed datastore systems. It determines how and when to migrate data to reduce response times and resource usage. Akkio primarily targets multi-datacenter geo-distributed datastore systems. Its design was motivated by the observation that many of Facebook's frequently accessed datasets have low R/W ratios that are not well served by distributed caches or full replication. Akkio's unit of migration is called a µ-shard . Each µ-shard is designed to contain related data with some degree of access locality. At Facebook, µ-shards have become a first-class abstraction.

Akkio went into production at Facebook in 2014, and it currently manages ∼ 100PB of data. Measurements from our production environment show that Akkio reduces access latencies by up to 50%, cross-datacenter traffic by up to 50%, and storage footprint by up to 40% compared to reasonable alternatives. Akkio is scalable: it can support trillions of µ-shards and process many 10's of millions of data access requests per second. And it is portable: it currently supports five datastore systems

Keywords:

Distributed systems, locality management, data migration, geo-distributed data, multi-datacenter datastores, performance, data access latenencies

Reference Info:

DOI: 10.5555/3291168.3291201
ISBN: 9781931971478
OCLC: 8876980931

BibTeX:

@inproceedings(Annamalai-osdi-18,
    author = {Muthukaruppan Annamalai and Kaushik Ravichandran and Harish Srinivas and Igor Zinkovsky and Luning Pan and Tony Savor and David Nagle and Michael Stumm},
    title = {Sharding the shards: managing datastore locality at scale with Akkio},
    booktitle = {Proceedings 13th USENIX Symposium on Operating Systems Design and Implementation (\textbf{OSDI'18})},
    location = {Carlsbad, CA, USA},
    publisher = {Usenix Association},
    month = {October},
    year = {2018},
    pages = {445-460},
    doi = {10.5555/3291168.3291201},
    isbn = {9781931971478},
    keywords = {Distributed systems, locality management, data migration, geo-distributed data, multi-datacenter datastores, performance, data access latenencies}
)