Different from traditional internet applications that connect users to users through the network, the components of IoT applications are across domains, such as "end", "edge", and "cloud", forming a more complex application mode.
Different from traditional Internet applications that connect users with users through the network, the participating objects of IoT applications span the "end", "edge" and "cloud", forming a more complex application mode:
(1) End-side: It is usually inside the device, such as a generator set, a logistics vehicle, a fan, etc. The end site usually has less storage and computing resources, and needs to cache or persist the generated data locally when there is no network. The stored data also supports applications such as edge computing.
(2) Edge-side: usually the edge gateway nodes or local control nodes, such as factory-level IoT platforms and centralized control systems. The edge side receives data from multiple devices, and monitors device status in real time, and needs to support efficient insertion, deletion, modification, and query functions.
(3) Cloud side: It is usually a company-level big data platform, and the data of all factories will be aggregated to the cloud platform for data analysis.
Provides a columnar storage file format designed for time series data. It uses devices and measurement model to establish two-dimensional indexes on series and time, and supports writing of tens of millions of points per second, high compression ratio storage and high-performance query.
Provides a complete function of inserting, deleting, querying, and provides advanced functions such as aggregation and time series alignment for time series data.
TsFile or IoTDB can connect to big data analysis engines, such as Spark, and integrate into the big data ecosystem as a data warehouse, providing data capabilities for time series data processing, analysis and machine learning.
Three software form adaptation scenarios:
| End | Edge | Cloud | |
| Lite |
√
|
√
|
√
|
| Standard |
√
|
√
|
√
|
| BigData |
√
|
√
|
In order to cope with the different requirements of "end-edge-cloud" data synchronization delay and traffic consumption in different application scenarios, Apache IoTDB provides two data synchronization solutions.
Solution 1: Low-traffic synchronization based on TsFile
Solution 2: Low-latency synchronization based on operation logs
Comparison of the two solutions:
| Low-traffic synchronization based on TsFile | Operational log-based low-latency synchronization | |
| Latency | Middle | Low |
| Traffic consumption | Low | High |
| Computing resource | Very Low | High |
The low-traffic synchronization solution based on TsFile overcomes the disadvantage of the traditional synchronization method of "one data, two processing", saves network traffic and cloud computing resources, and realizes a new form of data "Processing once, end-cloud sharing".
The data is persisted in the TsFile format on the sender end, and the data is deeply encoded and compressed by using the computing power of the edge side. When a TsFile is formed on the end-side, the TsFile can be transmitted to the receiving end without transmitting the original data, saving 85%-90% of network traffic.
On the receiving end IoTDB, the received TsFile can be directly loaded into the system without decoding and rewriting the data, saving 90%-95% of the computing resources of the receiving end.
It makes full use of edge computing capabilities, relieves cloud-side computing pressure, and supports massive sender instances.
This is the common solution of relational databases, such as MySQL's Binlog synchronization. The synchronization based on operation logs has the characteristics of low latency, but the computing pressure on the edge will also be applied to the cloud.
Write operations on the sender side are sent to the receiver side in real time to ensure low latency, usually synchronizing in seconds. After the receiver receives the write operation, it will re-execute it locally.
When there is a large amount of historical data on the sender side, the corresponding operation log may have been persisted as TsFile. Therefore, IoTDB has optimized this solution. The historical data is synchronized with the TsFile granularity, and the real-time data is synchronized with the operation log.
Contact Us |
