Apache IoTDB is a comprehensive data management engine designed for the collection, storage, and analysis of time series data. The performance and stability of IoTDB are heavily influenced by how memory is configured. This article will introduce the best practices for configuring memory in IoTDB, highlighting the new script parameters and computational logic introduced in recent versions.

Proper memory configuration is crucial to the performance of an IoTDB cluster. It not only boosts data processing efficiency and reduces latency but also helps prevent service interruptions caused by memory overflows. Furthermore, fine-tuning memory usage ensures that the IoTDB process is not terminated by the system's Out of Memory (OOM) Killer due to excessive memory consumption.

Configuration Methods

Method 1: Quick and Easy Configuration

Comfigure only the MEMORY_SIZE parameter. Introduced in version 1.3.0, MEMORY_SIZE is a new memory control parameter that specifies the total memory (including both heap and off-heap) available to IoTDB.

• If MEMORY_SIZE is not set, the script will automatically calculate the memory usage based on the system's total memory. ConfigNode will use 30% of the system memory, while DataNode will use 50% of the system memory. The system will also automatically allocate heap and off-heap memory based on the MEMORY_SIZE, with the specific rules as follows:

  • MEMORY_SIZE < 4G: Allocate 75% of the memory to heap memory and 25% to off-heap memory.

  • 4G <= MEMORY_SIZE < 16G: Allocate 80% of the memory to heap memory and 20% to off-heap memory.

  • 16G <= MEMORY_SIZE < 128G: Allocate 87.5% of the memory to heap memory and 12.5% to off-heap memory.

  • MEMORY_SIZE >= 128G: Allocate MEMORY_SIZE minus 16G to heap memory, with a fixed 16G allocated to off-heap memory.

The specific configuration methods are as follows:

• ConfigNode Memory: Set the MEMORY_SIZE in the conf/confignode-env.sh(bat) file, using either gigabytes (G) or megabytes (M) as the unit.

• DataNode Memory: Set the MEMORY_SIZE in the conf/datanode-env.sh(bat) file, using either gigabytes (G) or megabytes (M) as the unit.

Method 2: Fine-Tuned Configuration

Configure ON_HEAP_MEMORY and OFF_HEAP_MEMORY separately. These parameters, introduced in version 1.3.0, are optimized replacements for the older MAX_HEAP_SIZE, HEAP_NEWSIZE, and MAX_DIRECT_MEMORY_SIZE settings. They offer a more intuitive and simplified configuration process compared to the previous parameters.

When setting up ON_HEAP_MEMORY and OFF_HEAP_MEMORY, the system will allocate heap and off-heap memory based on these user-defined settings, which have higher priority than the automatically allocated memory by the MEMORY_SIZE parameter.

The specific configuration methods are as follows:

• ConfigNode: Set ON_HEAP_MEMORY and OFF_HEAP_MEMORY in the conf/confignode-env.sh(bat) file seperately.

• DataNode: Set ON_HEAP_MEMORY and OFF_HEAP_MEMORY in the conf/datanode-env.sh(bat) file seperately.

Summary

With the introduction of new memory control parameters and automatic calculation logic since the release of version 1.3.0, IoTDB has made memory configuration both more flexible and intelligent. These enhancements empower users to easily fine-tune memory settings according to their specific needs, thereby optimizing the performance of their IoTDB clusters.

For recommended values for system resource planning (such as CPU, memory, and disk), refer to the resource planning section of our documentation.

For any questions or further discussions, please feel free to reach out to us (contact@timecho.com)