apache / iotdb-client-go

Apache IoTDB Client for Go

English | 中文

Apache IoTDB

Apache IoTDB (Database for Internet of Things) is an IoT native database with high performance for data management and analysis, deployable on the edge and the cloud. Due to its light-weight architecture, high performance and rich feature set together with its deep integration with Apache Hadoop, Spark and Flink, Apache IoTDB can meet the requirements of massive data storage, high-speed data ingestion and complex data analysis in the IoT industrial fields.

Apache IoTDB Client for Go

Overview

This is the GoLang client of Apache IoTDB.

Apache IoTDB website: https://iotdb.apache.org Apache IoTDB Github: https://github.com/apache/iotdb

Prerequisites

golang >= 1.13

How to Use the Client (Quick Start)

With go mod

export GO111MODULE=on
export GOPROXY=https://goproxy.io

mkdir session_example && cd session_example

curl -o session_example.go -L https://github.com/apache/iotdb-client-go/raw/main/example/session_example.go

go mod init session_example
go run session_example.go

Without go mod

# get thrift 0.15.0
go get github.com/apache/thrift
cd $GOPATH/src/github.com/apache/thrift
git checkout 0.15.0

mkdir -p $GOPATH/src/iotdb-client-go-example/session_example
cd $GOPATH/src/iotdb-client-go-example/session_example

curl -o session_example.go -L https://github.com/apache/iotdb-client-go/raw/main/example/session_example.go
go run session_example.go

How to Use the SessionPool

SessionPool is a wrapper of a Session Set. Using SessionPool, the user do not need to consider how to reuse a session connection. If there is no available connections and the pool reaches its max size, the all methods will hang until there is a available connection. The PutBack method must be called after use

New sessionPool

standalone

config := &client.PoolConfig{
    Host:     host,
    Port:     port,
    UserName: user,
    Password: password,
}
sessionPool = client.NewSessionPool(config, 3, 60000, 60000, false)

cluster or doubleLive

config := &client.PoolConfig{
		UserName: user,
		Password: password,
		NodeUrls: strings.Split("127.0.0.1:6667,127.0.0.1:6668", ","),
	}
sessionPool = client.NewSessionPool(config, 3, 60000, 60000, false)

Get session through sessionPool, putback after use

set storage group

session, err := sessionPool.GetSession()
defer sessionPool.PutBack(session)
if err == nil {
    session.SetStorageGroup(sg)
}

query statement

var timeout int64 = 1000
session, err := sessionPool.GetSession()
defer sessionPool.PutBack(session)
if err != nil {
    log.Print(err)
    return
}
sessionDataSet, err := session.ExecuteQueryStatement(sql, &timeout)
if err == nil {
    defer sessionDataSet.Close()
    printDataSet1(sessionDataSet)
} else {
    log.Println(err)
}

Developer environment requirements for iotdb-client-go

OS

• Linux, Macos or other unix-like OS
• Windows+bash(WSL, cygwin, Git Bash)

Command Line Tools

• golang >= 1.13
• make >= 3.0
• curl >= 7.1.1
• thrift 0.15.0

Troubleshooting

Thrift version compatibility issues

In the branch rel/0.13 and earlier versions, the version of apache/thrift is v0.14.1. In the latest version, apache/thrift has been upgraded to v0.15.0.

The two versions are not compatible on some interfaces. Using mismatched version will cause compilation errors.

The interfaces changed in the two versions are as follows:

1. NewTSocketConf. This function returns two values in the version v0.14.1 and only one value in the version v0.15.0.
2. NewTFramedTransport has been deprecated, use NewTFramedTransportConf instead.

For more details, please take a look at this PR: update thrift to 0.15.0 to fit IoTDB 0.13.0

Parameter name mismatch with actual usage in function 'Open'

The implementation of the function client/session.go/Open() is mismatched with the description. The parameter connectionTimeoutInMs represents connection timeout in milliseconds. However, in the older version, this function did not implement correctly, regarding it as nanosecond instead. The bug is now fixed. Positive value of this parameter means connection timeout in milliseconds. Set 0 for no timeout.

iotdb-client-go's Issues

cannot get data when using some features implemented in iotdb session

I have implemented that change default DB of edgeX project as IoTDB working on docker,
but there are some issues when use GET method when I call it on edgeX.
IoTDB session is connected successfully with DB client of the project.

Description

I'm following printDataSet0() and insertRecord() implemented in session_example.go.

Although Session.InsertRecord() works successfully that returns no error and I can check input data in IoTDB (referenced here for testing it work as typing SQL in IoTDB), cannot retrieve the data in edgeX when use Session.ExecuteQueryStatement(sql) perfectly. More details, ExecuteQueryStatement() returns SessionDataSet without error but when use SessionDataSet.GetText(), it returns "" always but not input data saved in IoTDB.

Use InsertRecord() in AddEvent(): code here
and GetText() in ChangeTypeToEvent(): code here

execute SQL in IoTDB:

empty string returned from IoTDB in edgeX API response:
(deviceId: root.sg27, measurement: deviceName, value: root)

While I use getText(), make isNull() in the function not working because returns false:

// client/rpcdataset.go
func (s *IoTDBRpcDataSet) getText(columnName string) string {
	if s.closed {
		return ""
	}
	if columnName == TimestampColumnName {
		return time.Unix(0, bytesToInt64(s.time)*1000000).Format(time.RFC3339)
	}

	columnIndex := s.getColumnIndex(columnName)
	if columnIndex < 0 || int(columnIndex) >= len(s.values) || { //s.isNull(int(columnIndex), s.rowsIndex-1) {
		s.lastReadWasNull = true
		return ""
	}
	s.lastReadWasNull = false
	return s.getString(int(columnIndex), s.columnTypeDeduplicatedList[columnIndex])
}

• len(s.values) is updated (equal with expected value) but it has no value
• SessionDataSet.GetColumnCount() is updated too and equals with expected value
• But if isNull() is activated, then error occured: runtime error: negative shift amount
  • When the error is ocurred, isNull()'s parameters columnIndex=0, rowsIndex=0
    So in isNull(), parameter rowsIndex has -1 which makes the error because shift operation must be used with non-negative value

Question

• (edited) I think columnIndex and rowsIndex must be updated whenever POST event occurs. But rowsIndex is not.
  Is there any reason why not update it?
• In IoTDB, whenever get data from database if encounters SQL query, it calls ExecuteQueryStatement()?
  (or ExecuteStatement() or ExecuteRawDataQuery() whatever)

If you think reason is different, please share.

Environment

• IoTDB: 1.0.0-standalone (on docker)
• OS: Ubuntu 22.04.1 LTS

go compile error

go version：1.18.1
demo problem views

[BUG]Parameter ‘connectionTimeoutInMs’ mismatch with actual usage in function 'Open'

The implementation of the function client/session.go/Open() is mismatched with the description.
The parameter connectionTimeoutInMs represents connection timeout in milliseconds.
However, in the older version, this function did not implement correctly, regarding it as nanosecond instead.

assignment mismatch: 2 variables but thrift.NewTSocketConf returns 1 value

version:v0.13.0

写入丢包

我需要连续写入数据到数据库中，但是发现中间一直有丢包现象。通过调用insertWbRawData(v.Payload, i)函数，checkError()没有返回错误信息。但是检查数据库确没有发现数据。

func insertWbRawData(val string, cnt int) {
	var (
		deviceId           = "root.call4.care.wbs.raw"
		measurements       = []string{"data", "cnt"}
		values             = []string{}
		timestamp    int64 = time.Now().UTC().UnixNano() / 1000000
	)
	values = append(values, val+" "+strconv.Itoa(cnt))
	values = append(values, strconv.Itoa(cnt))
	checkError(session.InsertStringRecord(deviceId, measurements, values, timestamp))
}

func checkError(status *rpc.TSStatus, err error) {
	if err != nil {
		log.Fatal(err)
	}

	if status != nil {
		if err = client.VerifySuccess(status); err != nil {
			log.Println(err)
		} else {
			fmt.Println("suc", i)
		}
	}
}

使用session pool查询概率性出现StatementId:xxxx doesn't exist in this session xxxx问题

连续执行 2 次查询后报错

如图所示

数据能否自动完成类型转换？

func insertData() {
	var (
		deviceId           = "root.data.usernameId.88c4411b670c.deviceId"
		measurements       = []string{"switch", "temp", "count"}
		values             = []interface{}{false, 32.3, 108}
		dataTypes          = []client.TSDataType{client.BOOLEAN, client.FLOAT, client.INT32}
		timestamp    int64 = 120
	)
	session, err := sessionPool.GetSession()
	defer sessionPool.PutBack(session)
	if err == nil {
		checkError(session.InsertRecord(deviceId, measurements, dataTypes, values, timestamp))
	}

}

报错：values[1] 32.3(float64) must be float32，能否实现自动将数数据转换为实际的类型？