pingcap / mysql-tester Goto Github PK

View Code? Open in Web Editor NEW

60.0 10.0 61.0 182 KB

A Golang implementation of MySQL Test Framework

License: Apache License 2.0

Makefile 0.12% Go 99.88%

mysql-tester's Introduction

MySQL Tester

This is a golang implementation of MySQL Test Framework.

Requirements

All the tests should be put in t, take t/example.test as an example.
All the expected test results should be put in r. Result file has the same file name with the corresponding test file, but with a .result file suffix, take r/example.result as an examle.

How to use

Build the mysql-tester binary:

make

Basic usage:

Usage of ./mysql-tester:
  -all
        run all tests
  -host string
        The host of the TiDB/MySQL server. (default "127.0.0.1")
  -log-level string
        The log level of mysql-tester: info, warn, error, debug. (default "error")
  -params string
        Additional params pass as DSN(e.g. session variable)
  -passwd string
        The password for the user.
  -port string
        The listen port of TiDB/MySQL server. (default "4000")
  -record
        Whether to record the test output to the result file.
  -reserve-schema
        Reserve schema after each test
  -retry-connection-count int
        The max number to retry to connect to the database. (default 120)
  -user string
        The user for connecting to the database. (default "root")
  -xunitfile string
        The xml file path to record testing results.

By default, it connects to the TiDB/MySQL server at 127.0.0.1:4000 with root and no passward:

./mysql-tester # run all the tests
./mysql-tester example # run a specified test
./mysql-tester example1 example2   example3 # seperate different tests with one or more spaces

For more details about how to run and write test cases, see the Wiki page.

Contributing

Contributions are welcomed and greatly appreciated. You can help by:

writing user document about how to use this framework
triaging issues
submitting new test cases
fixing bugs of this test framework
adding features that mysql test has but this implementation does not
...

In case you have any problem, discuss with us in the tidbcommunity slack workspace.

See CONTRIBUTING.md for details.

License

MySQL Tester is under the Apache 2.0 license. See the LICENSE file for details.

mysql-tester's People

Contributors

Stargazers

Watchers

mysql-tester's Issues

tidb parser dependency is at old location

The parser used by the mysql tester is from github.com/pingcap/parser not github.com/pingcap/tidb/parser.

This is a problem because some of the tests require newer features:

FATA[0000] run test [ctype_gbk] err: sql:SET NAMES gbk;: [parser:1064]You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use [parser:1115]Unknown character set: 'gbk'

mysql test is not writing row_count() to result file

Row_count() is displayed as 0
Sample code :
--disable_warnings
drop table if exists t1;
--enable_warnings
create table t1 (id int primary key, data int);
insert into t1 values (1, 1), (2, 2), (3, 3);
select * from t1;
select row_count();

Result file:
drop table if exists t1;
create table t1 (id int primary key, data int);
insert into t1 values (1, 1), (2, 2), (3, 3);
select * from t1;
id data
1 1
2 2
3 3
select row_count();
row_count()
0

Q_IF not implemented (if statements in the test files)

There are references to if statements (Q_IF) in the code, but it is not implemented.

In MySQL's mysql-test-run / mysqltest binary some control statements are supported like if and while:
https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_MYSQL_TEST_FLOW_CONTROL.html

It would be nice if they were supported by mysql-tester as well.

Feature: add reserve-schema option to avoid dropping schema

I want to migrate these mysql tests to ticdc DDL/DML validation(end to end validation), e.g. TiDB -> TiCDC -> TiDB(MySQL).
but I find that schema will be drop after each test case.

Expect Feature:
Add an option to control the drop schema behavior.

Support additional tester macros

In addition to #24

The following macros in tests would be really helpful:

--replace_column and --disable_result_log (these are used to patch out results from the output file. Used when testing RAND() and temporal values)
--let and --eval - might be harder to implement but used by a lot of tests.
--include

Mysql test runner Crashing while using set session transaction

REPEATABLE READ is supported by tidb but test runner is no allowing.
same SET @@autocommit=0; this is also not working .

connect (con1,localhost,root,,test);
connection default;
SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;
CREATE TABLE t1 (s1 INT) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1),(2);
COMMIT;

Output is

skip building tidb-server, using existing binary: /Users/sachinagrawal/.tiup/components/tidb/v5.2.1/tidb-server
building mysql-test binary: ./mysql_test
build mysql_test successfully
start tidb-server, log file: ./mysql-test.out
tidb-server(PID: 69322) started
record result for case: "crashing"
INFO[0000] recording tests: [crashing]                  
WARN[0000] Create new db&{0 {root:@tcp(127.0.0.1:4001)/test?strict=true&time_zone=%27Asia%2FShanghai%27 0x1c56d40} 0 {0 0} [0xc0000aa870] map[] 0 1 0xc00008c120 false map[0xc0000aa870:map[0xc0000aa870:true]] map[] 0 0 0 0 <nil> 0 0 0 0 0x109b340} 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x10e27a6]

goroutine 7 [running]:
database/sql.(*Tx).Commit(0x0, 0x14eac00, 0x1)
	/usr/local/go/src/database/sql/sql.go:2151 +0x26
main.(*tester).commit(...)
	/Users/sachinagrawal/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:673
main.(*tester).stmtExecute(0xc0000aa750, 0xc00002aa94, 0x6, 0xc00002aa94, 0x7, 0x3, 0x2, 0x14fa6c8, 0xc0001320a0, 0x1, ...)
	/Users/sachinagrawal/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:566 +0xb6
main.(*tester).execute(0xc0000aa750, 0xc00002aa94, 0x6, 0xc00002aa94, 0x7, 0x3, 0x2, 0x0, 0x0)
	/Users/sachinagrawal/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:621 +0x13d
main.(*tester).Run(0xc0000aa750, 0x0, 0x0)
	/Users/sachinagrawal/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:307 +0x666
main.testBatch.Run(0xc00000c1e0, 0x1, 0x1e)
	/Users/sachinagrawal/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:928 +0xb3
created by main.executeTests
	/Users/sachinagrawal/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:958 +0xc5

Allow this framework to run GBK related tests

Allows this framework to parse GBK-related SQL

mysql tester crashes when transactions is not started and we commit

connect (con1,localhost,root,,test);

connection con1;
COMMIT;

so error message is

WARN[0000] Create new db&{0 {root:@tcp(127.0.0.1:4001)/test?strict=true&time_zone=%27Asia%2FShanghai%27 0x1059168} 0 {0 0} [0xc0001ee630] map[] 0 1 0xc000026240 false map[0xc0001ee630:map[0xc0001ee630:true]] map[] 0 0 0 0 <nil> 0 0 0 0 0x49cfa0} 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x4e6a06]

goroutine 7 [running]:
database/sql.(*Tx).Commit(0x0, 0x8edf28, 0x1)
	/usr/local/go/src/database/sql/sql.go:2151 +0x26
main.(*tester).commit(...)
	/home/morgo/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:673
main.(*tester).stmtExecute(0xc0001ee510, 0xc000024b57, 0x6, 0xc000024b57, 0x7, 0x2b, 0x2, 0x8fd648, 0xc00012e100, 0x1, ...)
	/home/morgo/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:566 +0xb6
main.(*tester).execute(0xc0001ee510, 0xc000024b57, 0x6, 0xc000024b57, 0x7, 0x2b, 0x2, 0x0, 0x0)
	/home/morgo/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:621 +0x13d
main.(*tester).Run(0xc0001ee510, 0x0, 0x0)
	/home/morgo/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:307 +0x666
main.testBatch.Run(0xc00000c1e0, 0x1, 0x1e)
	/home/morgo/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:928 +0xb3
created by main.executeTests
	/home/morgo/go/pkg/mod/github.com/pingcap/mysql-tester@v0.0.0-20211026071309-3ee4da8b351d/src/main.go:958 +0xc5
+ echo 'tidb-server(PID: 295933) stopped'
tidb-server(PID: 295933) stopped
+ kill -9 295933

Unneeded warning for 'Create new db', please move to Debug level.

For each test case a new database is created, and a new warning is given on stdout, which is just confusing.

WARN[0000] Create new db&{0 0xc000010330 0 {0 0} [0xc000096990] map[] 0 1 0xc0000bad20 false map[0xc000096990:map[0xc000096990:true]] map[] 0 0 0 0 <nil> 0 0 0 0 0x48a560}

Please remove it or change it to log.Debug.

mysql-tester should run all tests and then report error

Now, mysql tester is always Fatal when first error occurs, but in integration tests that run too much cases, it is very slow to figure out how many case it broke and fix in concurrence

dependency on github.com/go-sql-driver/mysql v1.3.0 is old

The dependency on the Go mysql driver is very old.

The mysql-tester depends on strict mode to fetch warnings, which has since been removed: https://github.com/go-sql-driver/mysql/wiki/strict-mode

(Call For Participation)*: Different expected error behavior when recording the test output.

For example, given an test file, say t/some_test.test:

sh> cat t/some_test.test -n
     1	drop database if exists tdb;
     2	create database tdb;
     3	use tdb;
     4	
     5	create table t(a int primary key);
     6	insert into t values (1);
     7	# an incorrect expected error
     8	--Error 1062
     9	insert into t values (2);

Notice that at line 8, an incorrect expected error("--Error 1062") is given, and when we try to record the result, MySQL test framework reports an error like the followings:

sh>mysqltest -u root < t/mysql_replace.test
drop database if exists tdb;
create database tdb;
use tdb;
create table t(a int primary key);
insert into t values (1);
mysqltest: At line 9: Query 'insert into t values (2)' succeeded, should have failed with error '1062'
not ok

But in this framework, recording is done successfully:

sh>run-tests.sh -s tidb-server -r some_test -b n
skip building tidb-server, using existing binary: ./tidb-server
skip building mysqltest, using existing binary: ./mysql_test
start tidb-server, log file: ./mysql-test.out
tidb-server(PID: 10971) started
record result for case: "some_test"
./t/some_test.test: ok! 2 test cases passed, take time 0.023170004 s

Great, All tests passed
mysqltest end
tidb-server(PID: 10971) stopped

mysql-tester fails with exit code 0.

mysql tester crashes when selecting truncated incorrect DECIMAL value

When selecting a truncated incorrect value, mysql tester crashes.

select 9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999 as x;
select 9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999 + 1 as x;

create table t1 (s varchar(100));
insert into t1 values (0.00000000010000000000000000364321973154977415791655470655996396089904010295867919921875);
drop table t1;

CREATE TABLE t2 (i DECIMAL (30,27));
INSERT INTO t2 VALUES (6.8926822182388193469056146);
DROP TABLE t2;

SET sql_mode="";
CREATE TABLE t (a int);
INSERT INTO t() VALUES(),(),(),();
SELECT
lag(1,96,
-66812354762147309124165421419678600705366236356475480.892682218238819346905614650696)
over()
FROM t;

DROP TABLE t;
SET sql_mode = default;

create table t1 (c1 decimal(64));
insert into t1 values(
89000000000000000000000000000000000000000000000000000000000000000000000000000000000000000);
--error ER_DATA_OUT_OF_RANGE
insert into t1 values(
99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999 *
99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999);
insert into t1 values(1e100);
select * from t1;
drop table t1;

Output is

panic: This branch is not implemented. This is because you are trying to test something specific to TiDB's MyDecimal implementation. It is recommended to do this in TiDB repository.

goroutine 7 [running]:
github.com/pingcap/tidb/parser/test_driver.fixWordCntError(...)
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/test_driver/test_driver_mydecimal.go:36
github.com/pingcap/tidb/parser/test_driver.(*MyDecimal).FromString(0x30, {0xc00042e000, 0x82, 0xc000056d80})
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/test_driver/test_driver_mydecimal.go:226 +0x3db
github.com/pingcap/tidb/parser/test_driver.init.0.func1({0xc0000ee127, 0x82})
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/test_driver/test_driver.go:34 +0x58
github.com/pingcap/tidb/parser.toDecimal({0x144ed58, 0xc000096048}, 0xc0000961a0, {0xc0000ee127, 0x1})
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/yy_parser.go:274 +0x3f
github.com/pingcap/tidb/parser.toInt({0x144ed58, 0xc000096048}, 0xc0000961a0, {0xc0000ee127, 0x82})
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/yy_parser.go:258 +0x150
github.com/pingcap/tidb/parser.(*Scanner).Lex(0xc000096048, 0xc0000961a0)
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/lexer.go:218 +0x3f3
github.com/pingcap/tidb/parser.yylex1({0x144ed58, 0xc000096048}, 0xc0000961a0)
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/parser.go:11023 +0x4a
github.com/pingcap/tidb/parser.yyParse({0x144ed58, 0xc000096048}, 0xc000096000)
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/parser.go:11077 +0x79ef7
github.com/pingcap/tidb/parser.(*Parser).ParseSQL(0xc000096000, {0xc0000ee120, 0x8f}, {0xc0001b1bd0, 0x2, 0x377})
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/yy_parser.go:154 +0x2b3
github.com/pingcap/tidb/parser.(*Parser).Parse(...)
        /Users/youracho/go/pkg/mod/github.com/pingcap/tidb/[email protected]/yy_parser.go:174
main.(*tester).execute(0xc000090000, {{0xc0000ee120, 0x6}, {0xc0000ee120, 0x8f}, 0x37c, 0x2})
        /Users/youracho/go/pkg/mod/github.com/pingcap/[email protected]/src/main.go:638 +0x11b
main.(*tester).Run(0xc000090000)
        /Users/youracho/go/pkg/mod/github.com/pingcap/[email protected]/src/main.go:318 +0x645
main.testBatch.Run({0xc00000c1e0, 0x1, 0x0})
        /Users/youracho/go/pkg/mod/github.com/pingcap/[email protected]/src/main.go:948 +0xbd
created by main.executeTests
        /Users/youracho/go/pkg/mod/github.com/pingcap/[email protected]/src/main.go:978 +0x1b3
tidb-server(PID: 35799) stopped

Mysql-test runner gives error for Isolation level read committed but works in TIDB

SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
START TRANSACTION;

In Mysql test runner error

sachinagrawal@Sachins-MacBook-Pro mysql_test % ./run-tests.sh -s ~/pingcap/Github/tidb/bin/tidb-server -r crashing
skip building tidb-server, using existing binary: /Users/sachinagrawal/pingcap/Github/tidb/bin/tidb-server
building mysql-test binary: ./mysql_test
build mysql_test successfully
start tidb-server, log file: ./mysql-test.out
tidb-server(PID: 89440) started
record result for case: "crashing"
INFO[0000] recording tests: [crashing]                  
WARN[0000] Create new db&{0 0xc000010010 0 {0 0} [0xc0000aa000] map[] 0 1 0xc00009c000 false map[0xc0000aa000:map[0xc0000aa000:true]] map[] 0 0 0 0 <nil> 0 0 0 0 0x109bda0} 
FATA[0000] run test [crashing] err: sql:SET TRANSACTION ISOLATION LEVEL READ COMMITTED;: run "SET TRANSACTION ISOLATION LEVEL READ COMMITTED;" at line 1 err Error 1568: Transaction characteristics can't be changed while a transaction is in progress 
tidb-server(PID: 89440) stopped

In TIDB

mysql> SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
Query OK, 0 rows affected (0.00 sec)

mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)

Tester should fail test if expected error message is not returned.

Tester is not failing when error message is expected and following command is not resulting error.

Hope to print the tested case name when starting a case

Now we only print the info after a test case is finish, such as ./t/index_merge_intersection_trs.test: ok! 4356 test cases passed, take time 12.45473737 s. However, some tests may hang and we can not print the messages as the test could not end forever. It will make the programmer hard to know which case is hung.

Empty tests for failed suites in junit xml report file.

https://prow.tidb.net/?repo=PingCAP-QE%2Ftidb-test&job=*mysql*&state=failure

can not know which case failed:

WITH xxx not support

SQL like WITH xxx will not record result.

Because mysql-tester doesn't recognize WITH xxx as a normal query.

The tester should create a database with the test name

This is a convention used by the mysql test script, and it will help in the event that a previous execution of another test did not clean up correctly.

MySQL Tester Improvement Plan

Doc

Update related doc and readme.

Bug Fix

Avoid use txn directly #109
Avoid import tidb/parser #109

Enhancement

Warning are not tracked in result file

Warning should be tracked in result file.

it should be possible to skip certain warnings via configuration

When running the mysql-tester against a MySQL 8.0 server a lot of the tests will complain about:

integer display width is deprecated.
utf8[mb3] is deprecated.

It should be possible to silence these generically so test output can be compared to MySQL using the same tester.

line number in error msg is not correct

Use the following test file:

select 1;
select 1;

# following drop SQL should give error, because there is no t1 existing.

drop table t1;

Error msg said error happens in line 2. But actually it's in line 6. If we have very a very long test. It is very annoying to manually find the line number of the specific error every time an error occurs.

INFO[0000] recording tests: [example]
WARN[0000] Create new db&{0 {root:root@tcp(127.0.0.1:3307)/test?strict=true&time_zone=%27Asia%2FShanghai%27&allowNativePasswords=true&allowAllFiles=true 0x1068278} 0 {0 0} [0xc0001a07e0] map[] 0 1 0xc0000781e0 false map[0xc0001a07e0:map[0xc0001a07e0:true]] map[] 0 0 0 0 <nil> 0 0 0 0 0x49b7a0}
FATA[0000] run test [example] err: sql:drop table t1;: run "drop table t1;" at line 2 err Error 1051: Unknown table 'example.t1'

Mysql test runner doesn't parse CTE statement correlty.

create table t1(a int);
insert into t1 values(1),(2);
with qn(a) as (select 1 from t1 limit 2)
select * from qn where qn.a=(select * from qn qn1 limit 1) union select 2;
drop table t1;

ERRO[0000] run test [with_non_recursive_bugs] err: sql:with qn(a) as (select 1 from t1 limit 2): [parser:1064]You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 40 near ""