executeNonQuery hangs... about clickhouse-net

[VitaliyMF]

@vboilay I was able to reproduce the issues you reported.

First, it seems explicit list of columns in insert command is required (@killwort please correct me if I wrong); I guess some error message should be shown instead of just handing forever.

Regarding "Checksum doesn't match: corrupted data", it seems lz4 compression doesn't work correctly for insert commands. I've tried to use command parameters:

ClickHouseConnection conn; /// correct opened CH connection
var cmd = conn.CreateCommand();
cmd.CommandText = "insert into vince_test (fakedate,csa,server) values (@p0, @p1, @p2)";
cmd.AddParameter("p0", DbType.Date, new DateTime(2017, 05, 18) );
cmd.AddParameter("p1", DbType.String, "CSA_CPTY1233");
cmd.AddParameter("p2", DbType.Int32, 0);
cmd.ExecuteNonQuery();

and also got 'Checksum verification failed' exception. But when I disabled the compression ("Compress=False") I was able to execute insert successfully. Please note that DbType should be defined for parameters explicitly, without them I also get strange errors like "Cannot convert parameter with type AnsiString to Date."

Hope this helps.

@killwort what do you think about the following changes/enhancements:

1. When "ExecuteNonQuery" is called for non-opened connection NullReferenceException is thrown (not very informative, actually) - possibly it is better to check the connection state and return more relevent exception message
2. If insert syntax without columns list is not supported, connector should behave more adequate instead of just hanging up forever.
3. Should compression work for inserts? If not, I guess more relevant error message should be shown too.
4. If DbType is not specified, I guess it is good idea to infer it from the value - say, cmd.AddParameter("p0", new DateTime(2017, 05, 18) ); should also work fine.

[vboilay]

hello, first I am glad you took some time to look at my case. I appreciate. I will come back to you with detailed answers in a forthcoming mail. regards. Le 5 nov. 2017 16:02, "Vitaliy Fedorchenko" <[email protected]> a écrit :

…

@vboilay <https://github.com/vboilay> first of all I was able to reproduce the issues you reported. First, it seems explicit list of columns in insert command is required ( @killwort <https://github.com/killwort> please correct me if I wrong); I guess some error message should be shown instead of just handing forever. Regarding "Checksum doesn't match: corrupted data", it seems lz4 compression doesn't work correctly for insert commands. I've tried to use command parameters: ClickHouseConnection conn; /// correct opened CH connection var cmd = conn.CreateCommand(); cmd.CommandText = "insert into vince_test (fakedate,csa,server) values ***@***.***, @p1, @p2)"; cmd.AddParameter("p0", DbType.Date, new DateTime(2017, 05, 18) ); cmd.AddParameter("p1", DbType.String, "CSA_CPTY1233"); cmd.AddParameter("p2", DbType.Int32, 0); cmd.ExecuteNonQuery(); and also got 'Checksum verification failed' exception. But when I disabled the compression ("Compress=False") I was able to execute insert successfully. Please note that DbType should be defined for parameters explicitly, without them I also get strange errors like "Cannot convert parameter with type AnsiString to Date." Hope this helps. @killwort <https://github.com/killwort> what do you think about the following changes/enhancements: 1. When "ExecuteNonQuery" is called for non-opened connection NullReferenceException is thrown (not very informative, actually) - possibly it is better to check the connection state and return more relevent exception message 2. If insert syntax without columns list is not supported, connector should behave more adequate instead of just hanging up forever. 3. Should compression work for inserts? If not, I guess more relevant error message should be shown too. 4. If DbType is not specified, I guess it is good idea to infer it from the value - say, cmd.AddParameter("p0", new DateTime(2017, 05, 18) ); should also work fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJYoijlQqQtY4KVkins-QEBKSfI3nFvzks5szc32gaJpZM4QRSBH> .

[killwort]

I've added checksum verification only in latest version, and it is still somewhat experimental. Please downgrade to 1.1.2 from nuget and give it a try. Meanwhile, I'll try to reproduce your error and maybe i'll be able to fix it.

PS: Don't use INSERT syntax with parametrized values, even with list of them, it is VERY inefficient! See readme.md to learn how to insert in clickhouse-friendly way. In most cases it leads to some in-memory data collection and then flushing it all in one jumbo insert.

[killwort]

As for INSERT syntax, its structure is checked in the client library, see /ClickHouse.Ado/Impl/ATG/Insert/Insert.atg. My grammar does not include that case (I think its a generally bad idea not to specify field list), however I'll rewrite this grammar to support fieldless inserts as nothing keeps from doing that. Maybe there would be some problems on the server side as clickhouse is not SQL-92 (or any other version) compliant, time will show :)

[killwort]

@VitaliyMF :
3: It should, however checksum errors indicate that something is wrong with decompression code itself
4: I could try implement reasonable DbType detection from value although ClickHouse has type system completely incompartible with DbType enum and it will limit its applicability.

[VitaliyMF]

@killwort

I've added checksum verification only in latest version, and it is still somewhat experimental.

what about adding special settings option (smth like "ChecksumCheck") and keep it disabled by default? It will allow to use latest version with compression for all types of commands, and enable checksum verification when needed.

It would be nice to have some unit tests (even quote primitive) for things like checksum check, SQL parser and data types handling etc. I understand that this might need some time that you possibly don't have.

[killwort]

I thought about adding checksum check as a connection string parameter. Can't say if it would be disabled by default as it validates message integrity in case of compression, it would be better to leave this enabled (provided that I'll repair it).
As for tests - there're some tests in ClickHouse.Test folder/project yet it is incomplete and covers only core functionality. If you'd like, you could write some test cases and pull request 'em.

[killwort]

I've fixed both errors, fieldless inserts now work (still, it is bad practice) and checksums are now correctly counted (there were small mistake with checksums for strings of length 16 - 127).
Checksum checking on client side are now controlled by CheckCompressedHash connection string parameter which is on by default.

Update to nuget v.1.1.4 to get these fixes. All older releases are prone to these checksum errors!