tango-controls-hdbpp / libhdbpp-mysql Goto Github PK

When using the hdb-++es-srv to archive images (251x251)(for example double_image of TangoTest DS) on insertion, during the preparation of the query the error "Prepared statement contains too many placeholders" is encountred by the Mysql server

Expected Behavior

When archiving a image, the method HdbPPMySQL::store_array (LibHdb++MySQL.cpp, line:1624) should insert the array without error whatever is the dimension of the image.

Current Behavior

The method HdbPPMySQL::store_array return "Prepared statement contains too many placeholders" error when the number of placeholders is superior to 65 536. In the current code, there are 12 palceholders per point. Which limit the archiving of images with up to 5461 pixels which is very limiting.

Possible Solution

The simplest solution would be to chunk the INSERT request in as many insertions containing 65 536 placeholders (or 5461 points) as needed in HdbPPMySQL::store_array method

The other solution would be to print the complete, allready prepared request into a tmp file and use LOAD DATA INFILE to insert it.

Please refer to this forum post for more details:
https://www.tango-controls.org/community/forum/c/development/java/eventsubscriber-ds-crashes

If libmariadb is used instead of libmysqlclient the following error occurs when storing VARCHAR columns:
mysql_stmt_bind_param() failed, err=Buffer type is not supported

After connection to MySQL server is dropped, reconnection occurs but existing prepared statements are no more valid and fail with the "Lost connection to MySQL server during query" error message.

Simone who posted this topic on the tango-controls forum (http://www.tango-controls.org/community/forum/c/general/other/hdb-usage-and-subscriber-errors/?page=1#post-2941), encountered errors related to timestamp default when creating the HDB++ tables in MySQL.

The only modification he needed to apply was to add at the beginning of the sql script:
SET SQL_MODE='ALLOW_INVALID_DATES';

The mysql table partitions are hardcoded here and outdated:
libhdbpp-mysql/etc/create_hdb++_mysql_innodb.sql

for example:

CREATE TABLE IF NOT EXISTS att_scalar_devboolean_rw
(
...
) ENGINE=InnoDB COMMENT='Scalar Boolean ReadWrite Values Table'
PARTITION BY RANGE COLUMNS(data_time) (
	PARTITION p000 VALUES LESS THAN ('2015-01-01'),
        PARTITION p2015 VALUES LESS THAN ('2016-01-01'),
        PARTITION p2016 VALUES LESS THAN ('2017-01-01'),
        PARTITION p2017 VALUES LESS THAN ('2018-01-01'),
        PARTITION p2018 VALUES LESS THAN ('2019-01-01'),
        PARTITION future       VALUES LESS THAN MAXVALUE
);

Instead a rotating partitioning could be used, which:

• does not need any modifications or updates over time
• still partitions the data in a yearly partition

This can be achieved by:

CREATE TABLE IF NOT EXISTS att_scalar_devboolean_rw
(
...
) ENGINE=InnoDB COMMENT='Scalar Boolean ReadWrite Values Table'
PARTITION BY HASH(YEAR(data_time)) PARTITIONS 100;

This will create 100 partitions,
att_scalar_devboolean_rw#p#p0.ibd till att_scalar_devboolean_rw#p#p99.ibd
where the hashing currently assigns the att_scalar_devboolean_rw#p#p21.ibd as the current parttion.
Confirmed working on mariadb and followed on mysql documentation.

Note that the parttions rotate by the hash function:
HASH = MOD(expr, num)
where num is 100 as set above and expr is YEAR(data_time).
eg: 2021 and 2121 will both write to att_scalar_devboolean_rw#p#p21.ibd

The value of 100 is chosen by personal preference.

see also more info here: https://dev.mysql.com/doc/refman/5.7/en/partitioning-hash.html

Please let me know, what you think of this. IMHO this simplifies usage and documentation a good deal.

I can also help with the code / documentation changes.

Cheers! Sebastian

Dear libhdbpp-mysql contributor (via commits, comments on issue or pull requests, issue creator),
If you have contributed in any way (commit, issue or Pull Request creation or comment) to libhdbpp-mysql, please log into Gitlab.com using your Github account ASAP.
See https://gitlab.com/tango-controls/TangoTickets/-/issues/47 for details.
Please click on the following link and add your github account name to this framapad when you've already done the login to Gitlab.com using your Github account:
https://mensuel.framapad.org/p/migration_ready_2548763689
Please note that login on gitlab.com using your Github account is safe and doesn't give your Github password to Gitlab.
For convenience, here is the list of contributors we could already find (and who did not signal they had done it yet for another repo):

• @chedburgh
• @prokofyevDmitry

Thank you for your cooperation.

Would be nice if the debian files have the instructions to generate the develop package

Please add release 1.2.0 to GitHub releases