1. Qing'an Energy Storage Scenario
Qing'an Energy Storage Technology (Chongqing) Co., Ltd. (hereinafter referred to as Qing'an Energy) is headquartered in the Western (Chongqing) Science City. It is a new energy technology company specialized in energy storage and intelligent management. It is also the leading enterprise in Chongqing focusing on integrated energy storage system security. Qing'an Energy Storage was incubated by an international top R&D team led by Prof. Dr. Ouyang Minggao, member of the Chinese Academy of Science. It has independently developed and manufactured key core components and integrated systems for energy storage systems. It focuses on battery system safety, power electronics, distributed generation, energy management, and other fields, aiming to provide energy storage system solutions and comprehensive energy technology services for the main power grid, new energy power plants, industrial and commercial enterprises, industrial parks, and residential users.
To help users achieve comprehensive monitoring, scheduling, and management of energy storage and energy systems, Qing'an Energy Storage's core R&D team has developed a cloud platform for energy storage. The energy storage cloud platform uses digital twin technology to faithfully reproduce the energy storage scene and provides remote monitoring, alerting, and precise operation and maintenance of energy storage devices and photovoltaic devices from an energy management perspective. It offers a range of customizable cloud platform solutions. The energy storage cloud platform mainly manages the time-series data reported by various energy devices. To achieve high stability and real-time processing performance of the energy storage cloud platform, Qing'an Energy needs to seek a better solution for writing, storing, querying, and analyzing time-series data in the energy field.
After testing and evaluating multiple time-series databases, Qing'an Energy has chosen IoTDB as the core component for time-series data access, storage, and querying in its energy storage cloud platform. IoTDB effectively supports low-latency query responses for multiple core query scenarios, achieves high-speed write performance, and provides ultra-high compression ratio, meeting the growing demand for managing massive time-series data.
2. Requirements and Pain Points of Qing'an Energy
2.1 Tens of Thousands of Tags per Device
Qing'an Energy currently needs to manage more than 50 energy storage devices, recording battery data such as temperature, voltage, and current. And the number of devices is to be doubled in one year. Due to the comprehensive monitoring requirements for the operating status of energy storage devices, the data volume per device is quite large, with the highest number of measurement points reaching several hundred thousand at a real-time reporting frequency of 5 seconds. With the foreseeable growth in data volume, Qing'an Energy has high requirements for real-time data writing and the ability to compress massive data in the time-series database.
2.2 Difficulties in Automatic Mounting
Qing'an Energy's data reporting chain involves integrating main devices and multiple sub-devices into a large field for transmission and reporting. With multiple measurement points per device and variations in device positions and fields, it is challenging to extract the reported data into a fixed model. This presents a challenge for the mounting function of time-series databases, which need to provide a secure and isolated architecture that can uniquely identify the storage devices, measurement points, and other hierarchical levels. It requires efficient marking of the storage device and file system location using methods other than a fixed model.
2.3 Real-time Query of Complex Fields
Qing'an Energy currently reports a large number of fields for its energy devices. Queries often involve multiple fields or multiple levels of paths and require real-time data feedback for various monitoring scenarios in the energy storage cloud platform. Therefore, there is a high requirement for the real-time query performance of the time-series database. Additionally, for complex queries that contain multiple keywords, Qing'an Energy also expects the time-series database to optimize statements for such queries to save time in drafting the query statements.
3. Reasons for Choosing IoTDB
3.1 Tree-like Architecture for Convenient Automatic Mounting
IoTDB's underlying time-series data structure adopts a dedicated tree-like structure for IoT management. It supports convenient data and model management functions such as device template management, sequence labeling, and automatic metadata recognition and generation. Considering the current differences in device locations and fields for Qing'an Energy, the tree-like structure of IoTDB does not require data to be extracted into a fixed model. Instead, data can be categorized by factories, devices, and other devices, facilitating automatic mounting while uniquely identifying the time-series data stored in IoTDB, enabling fine-grained queries.
3.2 10 Million Pts/s Throughput & 10:1 Lossless Compression Ratio
IoTDB can achieve the capability of writing tens of millions of data points per second and processing billions of points for multiple devices. The write rate does not decrease with the increase in data volume, meeting Qing'an Energy's data access requirements for large amounts of data from individual devices and continuous growth in data volume. Additionally, IoTDB's innovative columnar file storage format TsFile, optimized for time series, supports various efficient encoding and compression methods, achieving a 10:1 lossless compression ratio and effectively reducing the cost of storing time-series data.
3.3 Rich Low-Latency Query Features
IoTDB supports typical time-series data query types such as fast data filtering, latest value query, aggregate query, and down-sampling query through pre-aggregation and time-series indexing. This allows filtered data to be queried with a smaller data volume, achieving faster query speeds without affecting the output requirements of the query requester, and achieving millisecond-level response for terabytes of data. Furthermore, IoTDB can use the * symbol as a wildcard, making it more convenient to perform complex time-series data queries with multiple device fields in daily business scenarios.
3.4 Regular Version Updates
During the selection and research process, Qing'an Energy learned that the IoTDB's R&D team consists of several core database technical scientists and industrial experts. The team continuously updates version functionalities in the open-source community, and the related technical documentation is relatively clear. Qing'an Energy believes that the IoTDB team can ensure efficient and reliable operation and management of the database product.
4. Solution Architecture
Qing'an Energy's cloud platform is currently deployed entirely on the cloud. The platform integrates large-field data from energy storage devices, TCP/UDP devices, and third-party platforms. The data is compressed and sent to the cloud using protocols such as MQTT. At the device access layer, the data is decompressed and transformed into a unified data format from different protocols. After the transformation, the data is written into IoTDB in batches, with each batch consisting of 200,000+ measurement points, using a single thread. The data is stored in the IoTDB deployed on the cloud. Additionally, the energy storage cloud platform, based on IoTDB and other components, supports data cleaning, data caching, data subscription, and task scheduling. It also utilizes asynchronous queues to handle time-consuming tasks.
At the business layer, Qing'an Energy has developed a dedicated monitoring platform for the energy storage cloud platform. This platform provides real-time feedback on the operation of different devices in different factory areas, including alerts and exceptions, and performs processing in the cloud. By leveraging IoTDB's rich query capabilities, the monitoring platform achieves millisecond-level response for scenarios such as real-time retrieval of the latest values, time aggregation and statistical queries, and queries on the operating status of key components. It can also flexibly define time series data aggregation based on different intervals, such as minutes, seconds, or hours, and support Java, Python, and other language clients for algorithm analysis. The query performance of IoTDB enables real-time visualization queries on the monitoring platform and further supports users in performing value and benefit calculations based on the data results, as well as issuing control commands.
Currently, Qing'an Energy has been using IoTDB V0.13 standalone version for more than six months. The system runs stably with a single server configuration of 8H64G and a hard disk capacity of over 2TB. IoTDB achieves stable millisecond-level data writing and millisecond-level response for common business queries. The historical data is continuously stored, with an average compression ratio of 90.375. In the future, as the data volume increases, data migration and tiered storage will be considered.
5. Query Scenarios
IoTDB Client Query
Scenario 1: Monitoring status query
Qing'an Energy uses IoTDB client to query the latest data of different components in different time periods, enabling real-time monitoring of data and facilitating operators to monitor the real-time operating status of key components. Query statements and response time are as follows:
Scenario 2: Data trend analysis query
Qing'an Energy uses IoTDB client to perform data statistics and analyze the operational trends of the energy storage system and key component sensor values. For example, monitoring the SOC (State of Charge) and power changes of batteries within 24 hours/7 days to understand the daily range of charging and discharging, as well as power variations. Query statements and response time are as follows:
Internal query within the application server
Qing'an Energy also performs monitoring status queries and data trend analysis using IoTDB within the application server. Query statements and response time are as follows:
Based on the response time of the above query types, it can be seen that IoTDB supports Qing'an Energy's most common query scenarios, both at the client and server sides, and consistently achieves millisecond-level data retrieval.
6. Future Outlook
Currently, IoTDB has helped Qing'an Energy build a stable and efficient time series data management solution for its energy storage cloud platform. To cope with the growing volume of time series data in the future, Qing'an Energy plans to upgrade to the IoTDB cluster version. The IoTDB team will continue working closely with Qing'an Energy to develop a tailored IoTDB data migration solution that seamlessly transitions from the standalone version to the enterprise version, enabling high-speed data import and architecture switching without interrupting production capacity. This will assist Qing'an Energy in efficiently and flexibly managing even larger volumes of time series data.
