DataX

About 3 min

DataX

1. Overview

DataX is an offline data synchronization framework open-sourced by Alibaba, which enables efficient batch reading and writing of heterogeneous data sources through a plugin mechanism. IoTDB provides two plugins: IoTDBReader and IoTDBWriter, achieving seamless integration with DataX. This supports exporting time-series data from IoTDB to any target endpoint at high speed, as well as writing external data into IoTDB with one click, enabling flexible interconnection between IoTDB and the global data lakehouse.

2. Usage Steps

2.1 Environment Requirements

Software	Version Requirement	Installation Instructions
IoTDB	>=2.0.5.1	Quick Start
DataX	Customized Version	Contact Timecho staff to obtain the installation package

2.2 Reading Data IoTDBReader

IoTDBReader supports reading data from IoTDB database.

2.2.1 Parameter Description

Parameter	Description	Required	Default Value
username	Username	Yes	None
password	Password for the username	Yes	None
host	Host address for connecting to IoTDB database	Yes	None
port	Port	Yes	None
version	IoTDB version	Yes	None
timeColumnPosition	Position of the time column in the Record	No	0
querySqls	Write multiple SQL statements directly, can read in parallel, parameters below will be ignored.	No	None
device	Concept in IoTDB, can be understood as a table in MySQL.	Required when querySqls is empty	None
measurements	Concept in IoTDB, can be understood as fields in MySQL.	Required when querySqls is empty	None
where	Query condition	No	None

2.2.2 Configuration Example

Configure a job to extract data from IoTDB and write to MySQL

Source data in IoTDB is as follows

IoTDB> show timeseries root.mysql.**
+-----------------------------+-----+----------+--------+--------+-----------+----+----------+--------+------------------+--------+
|                   Timeseries|Alias|  Database|DataType|Encoding|Compression|Tags|Attributes|Deadband|DeadbandParameters|ViewType|
+-----------------------------+-----+----------+--------+--------+-----------+----+----------+--------+------------------+--------+
|       root.mysql.device.city| null|root.mysql|    TEXT|   PLAIN|        LZ4|null|      null|    null|              null|    BASE|
|root.mysql.device.temperature| null|root.mysql|  DOUBLE| GORILLA|        LZ4|null|      null|    null|              null|    BASE|
|   root.mysql.device.humidity| null|root.mysql|  DOUBLE| GORILLA|        LZ4|null|      null|    null|              null|    BASE|
|     root.mysql.device.status| null|root.mysql| BOOLEAN|     RLE|        LZ4|null|      null|    null|              null|    BASE|
+-----------------------------+-----+----------+--------+--------+-----------+----+----------+--------+------------------+--------+

IoTDB> select * from root.mysql.** align by device
+-----------------------------+-----------------+----+-----------+--------+------+
|                         Time|           Device|city|temperature|humidity|status|
+-----------------------------+-----------------+----+-----------+--------+------+
|2025-08-19T08:00:00.000+08:00|root.mysql.device|  bj|      31.45|   66.08|  true|
|2025-08-19T09:00:00.000+08:00|root.mysql.device|  bj|      20.86|   60.32| false|
|2025-08-19T11:00:00.000+08:00|root.mysql.device|  bj|      28.73|   63.21| false|
|2025-08-19T12:00:00.000+08:00|root.mysql.device|  bj|      21.82|   48.58|  true|
+-----------------------------+-----------------+----+-----------+--------+------+

iotdb_to_mysql_job.json configuration file:

{
  "job": {
      "setting": {
            "speed": {
                    "channel": 3
          }
   },
  "content": [
      {
       "reader": {
           "name": "iotdbreader",
           "parameter": {
                "username": "root",
                "password": "root",
                "host": "127.0.0.1",
                "port": 6668,
                "fetchSize": 10000,
                "version": "V_1_0",
                "##": "Position where time column is inserted into DataX Record, default is column 0",
                "timeColumnPosition": 0,
                "##":"When querySqls is specified, all parameters below are ignored, suitable for developers, multiple sessions execute multiple SQL statements in parallel",
                "querySqls":[
                 ],
                "device": "root.mysql.device",
                "##":"Time column is not a measurement",
                "measurements": ["city","temperature","humidity","status"],
                "where": "time > 2025-08-19 00:00:00 and time < 2025-08-19 20:00:00"
             }
        },
        "writer": {
            "name": "mysqlwriter",
            "parameter": {
                 "username": "root",
                 "password": "IoTDB@2024",
                 "writeMode": "insert",
                 "## Table needs to be created in advance": "CREATE TABLE device (`time` DATETIME,`city` VARCHAR(255),`temperature` DOUBLE,`humidity` DOUBLE,`status` BOOLEAN);",
                 "column": ["time","city","temperature","humidity","status"],
                 "session": [
                       "set session sql_mode='ANSI'"
                 ],
                 "preSql": [
                       "delete from device"
                 ],
                 "connection": [
                       {
                          "table": [
                              "device"
                           ],                                                                                                                                                                                                                          "##": "Remove brackets from URL below, otherwise will report error, mysqlreader bug, not modified",
                          "jdbcUrl": "jdbc:mysql://localhost:3306/demo?useUnicode=true&allowPublicKeyRetrieval=true&characterEncoding=utf-8"
                       }
                  ]
               }
         }
      }
  ]
  }
}

Create table device in MySQL in advance
Run DataX program:

python bin/datax.py job/iotdb_to_mysql_job.json

Verification: Query data in MySQL

mysql> select * from device;
+---------------------+------+-------------+----------+--------+
| time                | city | temperature | humidity | status |
+---------------------+------+-------------+----------+--------+
| 2025-08-19 08:00:00 | bj   |       31.45 |    66.08 |      1 |
| 2025-08-19 09:00:00 | bj   |       20.86 |    60.32 |      0 |
| 2025-08-19 11:00:00 | bj   |       28.73 |    63.21 |      0 |
| 2025-08-19 12:00:00 | bj   |       21.82 |    48.58 |      1 |
+---------------------+------+-------------+----------+--------+

2.3 Writing Data IoTDBWriter

IoTDBWriter supports writing data to IoTDB database target table (device).

2.3.1 Parameter Description

Parameter	Description	Required	Default Value
username	Username	Yes	None
password	Password for the username	Yes	None
host	Host address for connecting to IoTDB database	Yes	None
port	Port	Yes	None
version	IoTDB version	Yes	None
timeColumnPosition	Position of the time column in the Record	No	0
device	Concept in IoTDB, corresponds to table name in MySQL.	Yes	None
measurements	Concept in IoTDB, corresponds to field collection in MySQL, order should match column order in record.	Yes	None
batchSize	Write every batchSize records as a batch	No	1000
preSql	Whether to execute SQL before insertion	No	None

2.3.2 Configuration Example

Configure a job to write MySQL data to IoTDB:

Data in MySQL is as follows

mysql> select * from device;
+---------------------+------+-------------+----------+--------+
| time                | city | temperature | humidity | status |
+---------------------+------+-------------+----------+--------+
| 2025-08-19 08:00:00 | bj   |       31.45 |    66.08 |      1 |
| 2025-08-19 09:00:00 | bj   |       20.86 |    60.32 |      0 |
| 2025-08-19 11:00:00 | bj   |       28.73 |    63.21 |      0 |
| 2025-08-19 12:00:00 | bj   |       21.82 |    48.58 |      1 |
+---------------------+------+-------------+----------+--------+

iotdb_from_mysql_job.json configuration file

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "root",
            "password": "IoTDB@2024",
            "column": ["time","city","temperature","humidity","status"],
            "splitPk": "",
            "connection": [
              {
                "table": [
                  "device"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://localhost:3306/demo?useUnicode=true&allowPublicKeyRetrieval=true&characterEncoding=utf-8"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "iotdbwriter",
          "parameter": {
            "username": "root",
            "password": "root",
            "host": "127.0.0.1",
            "port": 6668,
            "version": "V_1_0",
            "##": "Position of time column in Reader, default is column 0",
            "timeColumnPosition": 0,
            "batchSize": 1000,
            "device": "root.db.device",
            "measurements": ["city","temperature","humidity","status"],
            "###": "SQL to execute before insertion, default is empty",
            "preSql": []
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 3
      }
    }
  }
}

Run DataX program

python bin/datax.py job/iotdb_from_mysql_job.json

Verification: Query data in IoTDB

IoTDB> select * from root.db.** align by device
+-----------------------------+--------------+----+-----------+--------+------+
|                         Time|        Device|city|temperature|humidity|status|
+-----------------------------+--------------+----+-----------+--------+------+
|2025-08-19T08:00:00.000+08:00|root.db.device|  bj|      31.45|   66.08|  true|
|2025-08-19T09:00:00.000+08:00|root.db.device|  bj|      20.86|   60.32| false|
|2025-08-19T11:00:00.000+08:00|root.db.device|  bj|      28.73|   63.21| false|
|2025-08-19T12:00:00.000+08:00|root.db.device|  bj|      21.82|   48.58|  true|
+-----------------------------+--------------+----+-----------+--------+------+