darkwanderer / clickhouse.client

.NET client for ClickHouse

ClickHouse.Client

ADO.NET client for ClickHouse, ultra-fast 'big data' relational database

Key features

• High-throughput
• Fully supports ClickHouse-specific types:
  • Composite types: Array, Tuple, Nullable, Nested, Map, including combinations
  • Specialized types: IPv4, IPv6, UUID, DateTime64, LowCardinality, Enum etc.
  • Large arithmetic types: (U)Int128, (U)Int256, Decimal128, Decimal256
  • Note: JSON type support was officially dropped from ClickHouse itself
• Correctly handles DateTime, including time zones
• Supports bulk insertion
• Uses compressed binary protocol over HTTP(S)
• Available for .NET Core/Framework/Standard

Advantages

Compared to other existing .NET clients, ClickHouse.Client has following advantages

• Does not have to buffer response, reducing memory usage
• Offers wider support for ClickHouse-specific types
• Is more compliant to ADO.NET standards (e.g. does not require calling 'NextResult' on SELECT queries)
• Works with ORM like Dapper, Linq2DB, Entity Framework Core etc.

Documentation

Documentation for the library is available in repository Wiki

clickhouse.client's Issues

Ensure Dapper support

• Support basic queries
• Support array parameters
• Support array return type

Unable to insert rows if table has many columns

I use ClickHouseBulkCopy to insert rows and it throws some http exception.
In my opinion the problem is that query is too long about 22k symbols long.

Aggregate function datatypes are not supported

An exception is thrown when an AggregateFunction datatype is encountered.

Will support for this datatype be available in upcoming releases?

LowCardinality type support

Large Tuple size not supported

Large Tuple column (with more than 7 members) will fail object mapping to the underlying .NET Tuple<> type with an unexpected NRE on this line:

ClickHouse.Client/ClickHouse.Client/Types/TupleType.cs

Line 34 in 5cfb0f6

var genericType = Type.GetType("System.Tuple`" + typeArgs.Length);

Is it possible to fix this? Possibly use ...TRest of the biggest Tuple class or just deserialize as a string, when exception happens?

Add IPv4/IPv6 column type support

Enum type support

ClickHouseBulkCopy nested columns data is not written to the database

When I tries to write data with latest version of library (2.2.2.294) I am getting exception:
ClickHouse.Client.ClickHouseServerException (0x0000003E): Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 1 (line 1, col 1): INSERT INTO DatabaseName.TableName (col1, nested1.col1, nested1.col2) FORMAT RowBinary ?��^��k�?????�k�?????��g�?????Lg�?????���^��k�?????�Lg�?????���^��k�?????��f�??. Unrecognized token (version 20.3.18.10 (official build))
It is working with the previous version of library (2.1.0.269)

Explicitly specify encoding as UTF8, add tests

Query parameters for older clickhouse version <19.11

Sql parameters as http get parameter were introduced in 19.11.3.11 version. For older versions parameters aren`t working

session_id support

Similarly, you can use ClickHouse sessions in the HTTP protocol. To do this, you need to add the session_id GET parameter to the request. You can use any string as the session ID. By default, the session is terminated after 60 seconds of inactivity. To change this timeout, modify the default_session_timeout setting in the server configuration, or add the session_timeout GET parameter to the request. To check the session status, use the session_check=1 parameter. Only one query at a time can be executed within a single session.

Add support for response buffering related parameters of Clickhouse HTTP API

Hi @DarkWanderer,
is it possible to add buffer_size and wait_end_of_query parameters into connection string, or add support for wildcard parameters with prefix to cover other and future HTTP API parameters too (e.g. ch-buffer_size=3000 will pass buffer_size=3000 setting to Clickhouse API?

Main reason to use those for us, is that when some query returns initial data quick enough, and the rest of response takes undeterministic amount of time, there is no way to understand whether this slowness comes from query itself/clickhouse, network, application c# code or driver. Additionally, frequently, there is no need to stream response, and it's more straightforward to work on completed result.

https://clickhouse.tech/docs/en/interfaces/http/#response-buffering

You can enable response buffering on the server-side. The buffer_size and wait_end_of_query URL parameters are provided for this purpose.

buffer_size determines the number of bytes in the result to buffer in the server memory. If a result body is larger than this threshold, the buffer is written to the HTTP channel, and the remaining data is sent directly to the HTTP channel.

To ensure that the entire response is buffered, set wait_end_of_query=1. In this case, the data that is not stored in memory will be buffered in a temporary server file.

KILL Statements are giving exception when there is 0 row returned.

Using 'KILL QUERY' statement and trying to read the results gives error when the result returns 0 row. pls check the example I attached.
clickhouse-session-locked.zip

Add HttpCompletionOptions.ResponseHeadersRead to avoid in-memory buffering

Boolean type support

Selecting Tuple may throw if numeric types don't match exactly

Cast exception for GetInt64 method

Доброго дня!

Обнаружил непредвиденное поведение метода "GetInt64" в последней выпущенной на данный момент версии.

Например, есть запрос:

SELECT
	COUNT(*) AS "Result"
FROM (
	SELECT
		toInt64(1) AS val	
	UNION ALL
	SELECT
		toInt64(2) AS val) nm

При попытке выполнить его и получить значение в таком коде:

            using (var connection = new ClickHouseConnection(_settings.ConnectionString))
            {
                connection.Open();
                using (var cmd = connection.CreateCommand())
                {
                    cmd.CommandText =
                        @"SELECT
	                        COUNT(*) AS ""Result""
                        FROM(
                            SELECT
                                toInt64(1) AS val
                            UNION ALL
                            SELECT
                                toInt64(2) AS val) nm";
                    using (var cmdReader = cmd.ExecuteReader())
                    {
                        if (cmdReader.Read())
                        {
                            var someValue = cmdReader.GetInt64(0);
                        }
                    }
                }
            }

Возникает исключение:

System.InvalidCastException: "Unable to cast object of type 'System.UInt64' to type 'System.Int64'."

Аналогично при вызове любых методов для получения числового значений (GetInt32, GetInt16 и др.)

На сколько я вижу, в последней версии был изменен алгоритм преобразования типов. Вместо класса "Convert" теперь используется обычный CAST'инг.

Было:

public override long GetInt64(int ordinal) => Convert.ToInt64(GetValue(ordinal), CultureInfo.InvariantCulture);

Стало:

public override long GetInt64(int ordinal) => (long)GetValue(ordinal);

Соответственно, явное преобразование из uint в int64 невозможно, что и приводит к ошибке.

Пока использую обходной путь, используя Convert явно:

var someValue= Convert.ToInt64(cmdReader.GetValue(0));

Можем вернуть использование Convert для GetInt64?
Возможно аналогично нужно вернуть его использование для всех методов GetInt*, GetDouble, GetDecimal, GetFloat, т.к. явное преобразование типа возможно только если типы совместимы, а это не всегда так. Сейчас если запрос возвращает тип ULong*, то ни один из методов не может его обработать корректно.

Есть идеи, мысли на этот счет?

"DB::Exception: Session is locked by a concurrent" error on ClickHouseBulkCopy.WriteToServerAsync

Hi,
so I've face the "clickhouse DB::Exception: Session is locked by a concurrent" exception when using the 'WriteToServerAsync' method. at first I thought it was a clickhouse error and well there was an existing issue on clickhouse which isthis.
But after further investigation I've come to realize that issue happens because the usage of the same clickhouse connection instance in parallel tasks here. the problem is that when the number of rows to be copied is like 10 or more times bigger that BatchSize then the same clickhouse connection is used with paralled uncompleted tasks, and that raises "DB::Exception: Session is locked by a concurrent" exception as it should.
instead of using the same connection, the bulk copy operation should have one connection per parallel task.

ALTER TABLE DELETE

Hi,

who can i execute alter query ?
ALETR TABLE TABLE1 DELETE WHERE ....

using stream for bulk copy

Hi,
with current ClickHouseBulkCopy api it's not possible to use streams for bulk inserts. using ClickHouseBulkCopy's WriteToServerAsync method causes high memory consumption when the data to be copied is big(in my case inserting 20 million data with 5-6 columns causes high memory usage). It would be good to be able to use stream for bulk copy. this can be achieve by opening 'PostStreamAsync' of ClickHouseConnection to public api.

Add CI configuration to run tests with older versions

To avoid backwards compatibility issues (e.g.: #47), CI needs to check not only latest ClickHouse version, but older versions as well. Probably best to set up using GitHub actions

Sir,How to bulk insert ? please give us a demo

Sir,How to bulk insert ? please give us a demo

Ensure Entity Framework Core support

Exception: Cannot read all data. Bytes read: 0. Bytes expected: 1

Test project (net fw 4.8) & data: https://i.tk-chel.ru/denisio/TestCH.zip

1. Unpack, create new database and table from table.sql file.
2. Edit Program.cs - connection string & path to 1.bin (msgpack serialized test data, 10K rows)
3. Run. Exception will occured. Clickhouse version 20.1.2.4 (official build)

            using (var bc = new ClickHouseBulkCopy(c) {DestinationTableName = "Tabular"})
                try
                {
                    var values = data.data.Select(p => p.items).ToArray();

                    await bc.WriteToServerAsync(values, data.colNames);
                }
                catch (Exception e)
                {
                    Console.WriteLine((e.InnerException ?? e).Message);
                }

clickhouse-server.err.log :

2020.03.28 11:26:47.802202 [ 46 ] {be18b95d-8cf2-44f2-af75-63cb9007a466} executeQuery: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expect
ed: 1. (version 20.1.2.4 (official build)) (from [::ffff:192.168.0.10]:12205) (in query: INSERT INTO Tabular FORMAT RowBinary ), Stack trace (when copying this message, always include the
lines below):

0. 0xbc31d9c Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) in /usr/bin/clickhouse
1. 0x4f6ccd9 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, int) in /usr/bin/clickhouse
2. 0x4b6efee ? in /usr/bin/clickhouse
3. 0x91ebf5f DB::BinaryRowInputFormat::readRow(std::__1::vector<COWDB::IColumn::mutable_ptrDB::IColumn, std::__1::allocator<COWDB::IColumn::mutable_ptrDB::IColumn > >&, DB::RowRea
   dExtension&) in /usr/bin/clickhouse
4. 0x97e5f69 DB::IRowInputFormat::generate() in /usr/bin/clickhouse
5. 0x91a4c27 DB::ISource::work() in /usr/bin/clickhouse
6. 0x9169435 DB::InputStreamFromInputFormat::readImpl() in /usr/bin/clickhouse
7. 0x8a6d32f DB::IBlockInputStream::read() in /usr/bin/clickhouse
8. 0x8bec745 DB::InputStreamFromASTInsertQuery::readImpl() in /usr/bin/clickhouse
9. 0x8a6d32f DB::IBlockInputStream::read() in /usr/bin/clickhouse
10. 0x8a9e66b DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::atomic*) in /usr/bin/clickhouse
11. 0x8dc3cc4 DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits, std::__1::allocato
    r > const&)>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&)>) in /usr/bin/clickhouse
12. 0x4ffdb8f DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&) in /usr/bin/clickhouse
13. 0x5001a73 DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) in /usr/bin/clickhouse
14. 0x9bb8bec Poco::Net::HTTPServerConnection::run() in /usr/bin/clickhouse
15. 0x9bb5677 Poco::Net::TCPServerConnection::start() in /usr/bin/clickhouse
16. 0x9bb5a6d Poco::Net::TCPServerDispatcher::run() in /usr/bin/clickhouse
17. 0xbca44ff Poco::PooledThread::run() in /usr/bin/clickhouse
18. 0xbca15c8 Poco::ThreadImpl::runnableEntry(void*) in /usr/bin/clickhouse
19. 0xbca2e69 ? in /usr/bin/clickhouse
20. 0x76db start_thread in /lib/x86_64-linux-gnu/libpthread-2.27.so
21. 0x12188f clone in /lib/x86_64-linux-gnu/libc-2.27.so

A string which content is an integer will become an ip address

Try this:
select '123'
The result of reader.GetString(0) is : 0.0.0.123

External data support

Is is possible to add support for external data for the queries? According to clickhouse documentation HTTP api supports it:

$ cat /etc/passwd | sed 's/:/\t/g' > passwd.tsv
$ curl -F '[email protected];' 'http://localhost:8123/?query=SELECT+shell,+count()+AS+c+FROM+passwd+GROUP+BY+shell+ORDER+BY+c+DESC&passwd_structure=login+String,+unused+String,+uid+UInt16,+gid+UInt16,+comment+String,+home+String,+shell+String'

For example command object could have a collection of objects representing "external data" - a class with name, schema and a callback that accepts writer and generates data

Wrong decimal serialization

Привет!

В BinaryStreamWriter.cs
https://github.com/DarkWanderer/ClickHouse.Client/blob/master/ClickHouse.Client/Formats/BinaryStreamWriter.cs#L61

Не верно записывает отрицательные decimal значения (они становятся положительные и не верные)

Исправил по аналогии, как в нативной реализации
https://github.com/killwort/ClickHouse-Net/blob/master/ClickHouse.Ado/Impl/ColumnTypes/DecimalColumnType.cs#L52

Стало нормально.

Expose API to support binary payload for bulkcopy externally

Current 'bulk copy' API expects IEnumerable<object[]> as an input. For performance reasons this API does not suit my needs - I have to insert large amount of "small" objects which implies large number of small array allocations and primitive value boxing which causes excessive memory allocations (and GC). So I need to "format data" into CH binary format outside the BulkCopy class and pass it inside somehow together with the schema. I can not use overload that accepts datareader because internally it still allocates object arrays. Batching might also be necessary, so simplest version would look like:

interface IBulkCopyData
{
     // this property can be passed as another argument also
     ColumnsAndType[] Schema { get; }
     /// return true if there is another batch that should be formatted
     Task<bool> Next(BinaryStreamWriter writter);
}
...
var myBulkCopyData = new MyBulkCopyData(source);
using var bulkCopy = new ClickHouseBulkCopy(clickHouseConnection);
// BatchSize and MaxDegreeOfParallelism are irrelevant because batching is drived
// by IBulkCopyData and maxdegreeofparallelist is limited to 1
await bulkCopy.WriteToServerAsync(dataReader cancellationToken);

Currently it is not possible because BinaryStreamWriter and related "type API" are internal. Also connection.PostStreamAsync(query, stream, true, token) is internal so even if I did rewrote BinaryStreamWriter I still can not use connection to send the data.

Also it would be nice to be able write directly into network stream (like "json formatters" do) instead or writing to memory stream because memory stream growing is another potential bottleneck - currently allocated memory stream is 0.5MB and if I need to send 1Gb of data it would be resized and copied 11 time consuming 2Gb in total and polluting LOH. For this one connection API should be extended.

Https host url not supported

Currently there is no way to define https connection

Add code coverage in CI

Exception (Version string portion was too short or too long) while sending request

Using last version Clickhouse.Client ( 2.0.0.262 ). Exception while processing request (.Query is Dapper extension, working fine with previous versions of Clickhouse.Client):

System.ArgumentException: Version string portion was too short or too long. at System.Version.VersionResult.SetFailure(ParseFailureKind failure, String argument) at System.Version.VersionResult.SetFailure(ParseFailureKind failure) at System.Version.TryParseVersion(String version, VersionResult& result) at System.Version.Parse(String input) at ClickHouse.Client.Utility.FeatureDetectorExtensions.<SupportsHttpParameters>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at ClickHouse.Client.ADO.ClickHouseConnection.d__2

Connection string: Host=192.168.0.113;Port=8123;Database=AGNET;User=default;Password=P@ssw0rd
select version(): 20.5.2.7

Add ability to tweak HTTP timeout

Fix JSON typing issues

JSON driver sometimes returns slightly incorrect types

[Question] But how can I pass parameters?

Could you explain here with little example or add to readme, please?

Invalid (nullable) datetime handling in 1.1.0.197

I have an table with non-nullable datetime column. When I try to read results of the following query when table is empty:

select max(accesstime) as accesstime from access_log

following exception is thrown:

Exception has occurred: CLR/System.InvalidCastException
Exception thrown: 'System.InvalidCastException' in ClickHouse.Client.dll: 'Unable to cast object of type 'System.String' to type 'System.DateTime'.'
at ClickHouse.Client.ADO.Readers.ClickHouseDataReader.GetDateTime(Int32 ordinal)

GetDateTime is called because reader.IsDBNull() returns false. CH response (compact json):

{
	"meta":
	[
		{
			"name": "accesstime",
			"type": "DateTime"
		}
	],

	"data":
	[
		["0000-00-00 00:00:00"]
	],

	"rows": 1,

	"statistics":
	{
		"elapsed": 0.00023376,
		"rows_read": 0,
		"bytes_read": 0
	}
}

Changing query to following:

select maxOrNull(accesstime) as accesstime from access_log

results in different exception:

Exception has occurred: CLR/System.InvalidCastException
Exception thrown: 'System.InvalidCastException' in System.Private.CoreLib.dll: 'Unable to cast object of type 'System.DBNull' to type 'System.String'.'
at ClickHouse.Client.ADO.Readers.ClickHouseJsonCompactReader.TryConvertTo(Object data, ClickHouseType type)
at ClickHouse.Client.ADO.Readers.ClickHouseJsonCompactReader.TryConvertTo(Object data, ClickHouseType type)
at ClickHouse.Client.ADO.Readers.ClickHouseJsonCompactReader.Read()
at System.Data.Common.DbDataReader.ReadAsync(CancellationToken cancellationToken)
CH response (compact json):

{
	"meta":
	[
		{
			"name": "accesstime",
			"type": "Nullable(DateTime)"
		}
	],

	"data":
	[
		[null]
	],

	"rows": 1,

	"statistics":
	{
		"elapsed": 0.000256913,
		"rows_read": 0,
		"bytes_read": 0
	}
}

NullReferenceException when using ClickHouseConnection constructor that accepts HttpClient

Calling ClickHouseConnection constructor that accepts both connection string and HttpClient instance in version 1.1.0.197 throws following exception:

System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
at ClickHouse.Client.ADO.ClickHouseConnection.set_ConnectionString(String value)
at ClickHouse.Client.ADO.ClickHouseConnection..ctor(String connectionString, HttpClient httpClient)

This happens because constructor first sets ConnectionString property and then assigns httpClient field but ConnectionString property's setter sets Timeout on httpClient

Querying merge() table fails

Querying data from merge() table engine does not work.

Error : System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> ClickHouse.Client.ClickHouseServerException: Code: 16, e.displayText() = DB::Exception: There is no physical column _table in table. (version 20.3.8.53 (official build))

   at ClickHouse.Client.ADO.ClickHouseConnection.<HandleError>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ClickHouse.Client.ADO.ClickHouseConnection.<PostSqlQueryAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ClickHouse.Client.ADO.ClickHouseCommand.<ExecuteDbDataReaderAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ClickHouse.Client.ADO.ClickHouseCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()

Retrieve schema information from DB connection

We need to retrieve table columns' information from DB connection. Currently we are getting it from DB connection interfaces (IDBConnection) by using GetSchema interface but for Clickhouse it gives the following error;

System.NotSupportedException: 'Specified method is not supported.'

For Oracle it returns related column information with their data types. I need to use same GetSchema method for Clickhouse connection (ClickHouseConnection) as well. It returns System.Data.DataTable object which we are using for data type differentiation for each coming column name.

As a workaround we wrote our own query to get them like below;

SELECT name ColumnName, CASE WHEN type LIKE ('%Int%') OR type LIKE ('%Float%') OR type LIKE ('%Decimal%') THEN 'NUMBER' WHEN type LIKE ('%Date%') THEN 'DATE' ELSE 'STRING' END DataType FROM system.columns WHERE database = '{schemaName}' and table = '{tableNameWithoutSchema}'

So basically GetSchema could return all of the columns within a System.Data.DataRow object for each row in System.Data.DataTable object.

Insert of Array via named parameter not working

Hi @DarkWanderer,

I'm trying to insert an array of float values float[] via a named parameter, but it gives me a broken pipe exception:

Exception has occurred: CLR/System.Net.Http.HttpRequestException
Exception thrown: 'System.Net.Http.HttpRequestException' in System.Private.CoreLib.dll: 'An error occurred while sending the request.'
 Inner exceptions found, see $exception in variables window for more details.
 Innermost exception 	 System.Net.Sockets.SocketException : Broken pipe

I'm using version 2.0.0.262.

If I use the syntax [1.2342,0.3234,2234.1] in the SQL INSERT query directly, it works. If I use the named parameter syntax {{values:Array(Float32)}}, it doesn't. Am I using the wrong syntax or is there a bug in the Client library?

Thanks!

Invalid complex column name handling in ClickHouseBulkCopy

I've encountered following problem - when column name requires quotation (contains whitespace, comma, semicolon, bracket, etc) ClickHouseBulkCopy,ClickHouseBulkCopy miserably fails with error like:

Code: 62, e.displayText() = DB::Exception: Syntax error: failed at position 40 (line 1, col 40): 2) FORMAT RowBinary
. Expected identifier (version 20.3.20.6 (official build))

Steps to reproduce:

#!/bin/bash

dotnet new console --name CHBulkInsertStrangeColumnName

dotnet add CHBulkInsertStrangeColumnName/CHBulkInsertStrangeColumnName.csproj  package ClickHouse.Client --version 2.3.0.326

cat << EOF > CHBulkInsertStrangeColumnName/Program.cs
using System;
using System.Threading;
using System.Threading.Tasks;
using ClickHouse.Client.ADO;
using ClickHouse.Client.Copy;

namespace CHBulkInsertStrangeColumnName
{
        internal class Program
        {
                internal static async Task Main()
                {
                        var cancellationToken = CancellationToken.None;
                        Console.Write("Please enter connection string: ");
                        var connectionString = Console.ReadLine();
                        using var connection = new ClickHouseConnection(connectionString);
                        using var command = connection.CreateCommand();
                        command.CommandText = $"DROP TABLE IF EXISTS my_table";
                        await command.ExecuteNonQueryAsync(cancellationToken);
                        command.CommandText = $"CREATE TABLE my_table (column_1 Int64, \"column.2\" Int64) ENGINE = Log";
                        await command.ExecuteNonQueryAsync(cancellationToken);
                        using var bulkCopy =
                                new ClickHouseBulkCopy(connection)
                                {
                                        DestinationTableName = "my_table",
                                        BatchSize = 100000,
                                        MaxDegreeOfParallelism = 1
                                };

                        var rows = new[] { new object[] { 1, 2 } };
                        try
                        {
                                await bulkCopy.WriteToServerAsync(rows, cancellationToken);
                                Console.WriteLine("ClickHouseBulkCopy.WriteToServerAsync completed");
                        }
                        catch (Exception e)
                        {
                                Console.WriteLine(e.Message);
                        }
                }
        }
}
EOF

cd CHBulkInsertStrangeColumnName

dotnet run

Script drops and creates table with hardcoded name 'my_table' with a column that contains period in the name. Replacing period with underscore and any letter makes it work but I have to support columns with whitespaces, periods and other characters.

Using overload that accepts column names does not solves it but make problem more clear:

Code: 47, e.displayText() = DB::Exception: Missing columns: 'column' while processing query: 'SELECT column_1, column.2 FROM my_table LIMIT 0', required columns: 'column_1' 'column', source columns: 'column.2' 'column_1' (version 20.3.20.6 (official build))

ServerVersion strange value

Using last version Clickhouse.Client ( 2.0.0.262 ):

Connection string: Host=192.168.0.113;Port=8123;Database=AGNET;User=default;Password=P@ssw0rd

select version(): 20.5.2.7

Add DateTime64 support

Exception in DataAdapter.Fill when Nullable type is present in query results

Exception Nullable Type Generate in sum query.

i cannot simulate on simple query.
i test with DataReader and DataAdaptor.Fill.

same result generate .

ArgumentOutOfRangeException: type Float is not supported

This is a known issue related to introduction "type aliases" (e.g: Float, Varchar) in ClickHouse. This feature has been reverted from database and hence will not be supported in ClickHouse.Client

DB::Exception: Unknown format RowBinaryWithNamesAndTypes

I was getting "DB::Exception: Unknown format RowBinaryWithNamesAndTypes" for the code in the quick start example.
The cause was in that the version of ClickHouse was 18.
As far as I understand RowBinaryWithNamesAndTypes is supported in ClickHouse starting from version 19.

I created this issue just for information for others.

Add example for Dapper to the SQL-Parameters wiki page

I propose to add the following to the end of the https://github.com/DarkWanderer/ClickHouse.Client/wiki/SQL-Parameters page:

---- start of the change

For Dapper

You need to pass parameters as Dictionary<string, object>.

For example, the following:

connection.QueryAsync<string>("select {p1:Int32}", new { p1 = 42 });

will not work. You need do it this way:

connection.QueryAsync<string>("select {p1:Int32}", 
    new Dictionary<string, object> { { "p1", 42 } });

---- end of the change

Can't add parameters to a query when httpParametersSupported

Within the PostSqlQueryAsync method when you have the httpParametersSupported flag as true the resulted query doesn't work properly with 20.3.11.97 version of the ClickHouse DB.

Example query:

var command = connection.CreateCommand();
command.CommandText = "select col from tbl where col2 = '{p1:String}'";
command.AddParameter("p1", "some_string");

"FORMAT RowBinaryWithNamesAndTypes" gets added to the end of the query and it doesn't get the correct results but when you force to replace the parameters within the code by manipulating the execution (i.e. httpParametersSupported = false; ) the same query gets executed just fine and returns the expected data.

Side note:
I didn't see RowBinaryWithNamesAndTypes in the latest documentation of ClickHouse
https://clickhouse-docs.readthedocs.io/en/latest/formats/index.html

HTTP Connection leak under Linux

Здравствуйте,
подскажите пожалуйста, я делаю bulkinsert (для простоты напрямую из MVC api контроллера)

       ClickHouseConnection connection = TestUtilities.GetTestClickHouseConnection();
        using (var bulkCopy = new ClickHouseBulkCopy(connection))
        {
            bulkCopy.DestinationTableName = "Test";
            await bulkCopy.WriteToServerAsync(objectsList.ToArray(), CancellationToken.None);
        };

Здесь TestUtilities это статический класс взятый из вашего примера.

Данные сохраняются в ClickHouse,
но в моем .Net Core 2.2 приложении накапливаются не закрытые коннекты.
Каждый вызов ClickHouseBulkCopy добавляет 1 незакрытый коннект.
После достижения лимита 30000 (в моем случае это пара дней) приложение аварийно закрывается.

Проблема существует если запускать из Ubuntu (контейнера).
Если запускать из Windows10, то незакрытых коннектов нет.

Количество открытых коннектов проверяю через:

long currentConnections = IPGlobalProperties.GetIPGlobalProperties()
                .GetTcpIPv4Statistics()
                .CurrentConnections;

Подскажите, что я делаю не так, можно ли этого избежать?

ServerVersion could contains error text if select version() fails

If there is an error in reading version, than ServerVersion will contains error text and will fail at following Version.Parse calls.
maybe try parse version in openAsync call

Output format support

Quite often all appication is doing is just feeding query result to a "processing pipeline" in some format ("returning json from rest api to some client, maybe some SPA" is not an exception here). As clickhouse already supports great variety of output formats that could be fed to the "pipeline" (json, protobuf, avro, arrow, parquet, csv) it looks reasonable to allow query results to be returned in desired format to cut ADO and formatting libraries all together. There can be an extension method on a reader object named, for example, "export", accepting stream where data should be copied:

var command = connection.CreateCommand();
command.Text = ".... Format Protobuf";
using var reader = await command.ExecuteReaderAsync();
await reader.Export(output);

Another option is having extension method for command:

var command = connection.CreateCommand();
command.Text = ".... Format Protobuf";
using var exporter = await command.ExecuteExportAsync();
await exporter.Export(output);

It is important to separate query execution from "copying results" to be able to handle errors