On May 16, 2024, the research paper titled "On Tuning Raft for IoT Workload in Apache IoTDB" was presented at the prestigious 40th IEEE International Conference on Data Engineering (ICDE 2024). This paper, co-authored by researchers from Tsinghua University and Timecho, showcases innovative advancements in the industrial application of the Raft consensus protocol tailored for IoT workloads within the Apache IoTDB framework. The acceptance of this paper at ICDE, a leading conference in the field, underscores its technical innovation and advanced industrial application, receiving authoritative recognition from the global academic community.
ICDE, organized by the Institute of Electrical and Electronics Engineers (IEEE), is recognized alongside VLDB and SIGMOD as one of the top three international conferences in data management. The ICDE 2024 conference, held in the Netherlands, attracted nearly 1,000 participants from renowned universities and tech companies, including MIT, Stanford University, Oracle, and Google, to discuss cutting-edge issues in AI, databases, and data processing.
The paper, "On Tuning Raft for IoT Workload in Apache IoTDB," provides a systematic performance evaluation of the Raft protocol in IoT scenarios using the open-source time-series database, Apache IoTDB. Raft is widely used as a consensus protocol in various distributed systems due to its simplicity and ease of implementation. However, directly applying Raft may not fully meet the high throughput requirements of IoT scenarios. This research highlights the unique characteristics of IoT applications, such as high concurrency, fluctuating traffic, fixed-size requests, and compressible data, which expose Raft's bottlenecks in log distribution, persistence, and memory management in IoTDB.
Statistics in the Raft process to Identify Bottlenecks
To address these challenges, the paper explores the possibility of tuning the Raft protocol specifically for IoT workloads, including alternative data structures, implementing various compression algorithms, and memory recycling strategies.
Possibility of Tuning the Raft Protocol
Extensive experiments demonstrate that these adjustments improve system parallelism, reduce information redundancy, and enhance resource utilization. The throughput of the database can increase by up to 200% through pre-serialization and reach up to 4 times the original Raft implementation by replacing scheduling data structures.
With the collaboration of worldwide contributors, Apache IoTDB continues to maintain its leading position in the field of time-series database technology through deep integration of production, academia, research, and application. The team at Timecho is dedicated to exploring IoT scenario requirements and achieving breakthroughs in database technologies for data writing, storage, querying, and analysis. With over 30 technical papers published and several papers accepted at top database conferences, Apache IoTDB has gained recognition from international experts in the field.
Looking forward, IoTDB will persist in its path of independent research and continuous innovation, focusing on IoT needs. By integrating academic research with industrial applications, it aims to refine its products with advanced self-developed technologies, providing innovative and competitive IoT database products and services to enterprise users and developers.
