datafuselabs / databend-perf

Databend Performance Data and Visualization

Databend Performance Monitoring & Benchmarking

This repository contains these primary dirs:

• collector: stores performance data (benchmark & reload) for nightly release.
• benchmarks: contains the benchmark suite defined by the yaml format.
• reload: contains the reload suite defined by the yaml format.

databend-perf's Issues

fails create hits schema with CLUSTER BY

      CREATE TABLE hits (
        WatchID BIGINT NOT NULL,
        JavaEnable SMALLINT NOT NULL,
        Title VARCHAR NOT NULL,
        GoodEvent SMALLINT NOT NULL,
        EventTime TIMESTAMP NOT NULL,
        EventDate Date NOT NULL,
        CounterID INTEGER NOT NULL,
        ClientIP INTEGER NOT NULL,
        RegionID INTEGER NOT NULL,
        UserID BIGINT NOT NULL,
        CounterClass SMALLINT NOT NULL,
        OS SMALLINT NOT NULL,
        UserAgent SMALLINT NOT NULL,
        URL VARCHAR NOT NULL,
        Referer VARCHAR NOT NULL,
        IsRefresh SMALLINT NOT NULL,
        RefererCategoryID SMALLINT NOT NULL,
        RefererRegionID INTEGER NOT NULL,
        URLCategoryID SMALLINT NOT NULL,
        URLRegionID INTEGER NOT NULL,
        ResolutionWidth SMALLINT NOT NULL,
        ResolutionHeight SMALLINT NOT NULL,
        ResolutionDepth SMALLINT NOT NULL,
        FlashMajor SMALLINT NOT NULL,
        FlashMinor SMALLINT NOT NULL,
        FlashMinor2 VARCHAR NOT NULL,
        NetMajor SMALLINT NOT NULL,
        NetMinor SMALLINT NOT NULL,
        UserAgentMajor SMALLINT NOT NULL,
        UserAgentMinor VARCHAR(255) NOT NULL,
        CookieEnable SMALLINT NOT NULL,
        JavascriptEnable SMALLINT NOT NULL,
        IsMobile SMALLINT NOT NULL,
        MobilePhone SMALLINT NOT NULL,
        MobilePhoneModel VARCHAR NOT NULL,
        Params VARCHAR NOT NULL,
        IPNetworkID INTEGER NOT NULL,
        TraficSourceID SMALLINT NOT NULL,
        SearchEngineID SMALLINT NOT NULL,
        SearchPhrase VARCHAR NOT NULL,
        AdvEngineID SMALLINT NOT NULL,
        IsArtifical SMALLINT NOT NULL,
        WindowClientWidth SMALLINT NOT NULL,
        WindowClientHeight SMALLINT NOT NULL,
        ClientTimeZone SMALLINT NOT NULL,
        ClientEventTime TIMESTAMP NOT NULL,
        SilverlightVersion1 SMALLINT NOT NULL,
        SilverlightVersion2 SMALLINT NOT NULL,
        SilverlightVersion3 INTEGER NOT NULL,
        SilverlightVersion4 SMALLINT NOT NULL,
        PageCharset VARCHAR NOT NULL,
        CodeVersion INTEGER NOT NULL,
        IsLink SMALLINT NOT NULL,
        IsDownload SMALLINT NOT NULL,
        IsNotBounce SMALLINT NOT NULL,
        FUniqID BIGINT NOT NULL,
        OriginalURL VARCHAR NOT NULL,
        HID INTEGER NOT NULL,
        IsOldCounter SMALLINT NOT NULL,
        IsEvent SMALLINT NOT NULL,
        IsParameter SMALLINT NOT NULL,
        DontCountHits SMALLINT NOT NULL,
        WithHash SMALLINT NOT NULL,
        HitColor VARCHAR NOT NULL,
        LocalEventTime TIMESTAMP NOT NULL,
        Age SMALLINT NOT NULL,
        Sex SMALLINT NOT NULL,
        Income SMALLINT NOT NULL,
        Interests SMALLINT NOT NULL,
        Robotness SMALLINT NOT NULL,
        RemoteIP INTEGER NOT NULL,
        WindowName INTEGER NOT NULL,
        OpenerName INTEGER NOT NULL,
        HistoryLength SMALLINT NOT NULL,
        BrowserLanguage VARCHAR NOT NULL,
        BrowserCountry VARCHAR NOT NULL,
        SocialNetwork VARCHAR NOT NULL,
        SocialAction VARCHAR NOT NULL,
        HTTPError SMALLINT NOT NULL,
        SendTiming INTEGER NOT NULL,
        DNSTiming INTEGER NOT NULL,
        ConnectTiming INTEGER NOT NULL,
        ResponseStartTiming INTEGER NOT NULL,
        ResponseEndTiming INTEGER NOT NULL,
        FetchTiming INTEGER NOT NULL,
        SocialSourceNetworkID SMALLINT NOT NULL,
        SocialSourcePage VARCHAR NOT NULL,
        ParamPrice BIGINT NOT NULL,
        ParamOrderID VARCHAR NOT NULL,
        ParamCurrency VARCHAR NOT NULL,
        ParamCurrencyID SMALLINT NOT NULL,
        OpenstatServiceName VARCHAR NOT NULL,
        OpenstatCampaignID VARCHAR NOT NULL,
        OpenstatAdID VARCHAR NOT NULL,
        OpenstatSourceID VARCHAR NOT NULL,
        UTMSource VARCHAR NOT NULL,
        UTMMedium VARCHAR NOT NULL,
        UTMCampaign VARCHAR NOT NULL,
        UTMContent VARCHAR NOT NULL,
        UTMTerm VARCHAR NOT NULL,
        FromTag VARCHAR NOT NULL,
        HasGCLID SMALLINT NOT NULL,
        RefererHash BIGINT NOT NULL,
        URLHash BIGINT NOT NULL,
        CLID INTEGER NOT NULL
      ) CLUSTER BY (CounterID, EventDate, UserID, EventTime, WatchID);

error:

#3: query failed: query query-perf in org orgchasen-slug-test has error: &{1065 error: 
    --> SQL:107:15
    |
104 |   RefererHash BIGINT NOT NULL,
105 |   URLHash BIGINT NOT NULL,
106 |   CLID INTEGER NOT NULL
107 | ) CLUSTER BY (CounterID, EventDate, UserID, EventTime, WatchID);
    |               ^^^^^^^^^ column doesn't exist
 }, stats: <nil>

https://github.com/datafuselabs/databend-perf/runs/7449833921?check_suite_focus=true

hangs on executing Q10 SELECT avg(cnt) FROM (SELECT Year,Month,count(*) AS cnt FROM default.ontime WHERE DepDel15=1 GROUP BY Year,Month) a;

cluster: 2 instances.

hangs on start query , and without query ID returned at all.

graphs base demo

base demo and pages frame

Upload data on rerun ci success

Currently if the CI is rerun, even if it is successful, we will not be able to upload the data correctly.

 INPUT_PUSH_OPTIONS: 
To https://github.com/datafuselabs/databend-perf
 ! [rejected]        main -> main (non-fast-forward)
error: failed to push some refs to 'https://github.com/datafuselabs/databend-perf'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

use a standalone org in databend cloud to execute perf test

currently we are using a shared databend cloud org in the perf test and the daily tests.

the perf-test warehosue maybe interleaved with our manual operations, and get the following logs.

2022/07/08 05:48:10 webapi.doRequest POST url=https://test.datafusecloud.com/api/v1/orgs/orgchasen-slug-test/tenant/warehouses req={"imageTag":"v0.7.121-nightly","name":"query-perf","size":"***"} resp={"success":true}
[20](https://github.com/datafuselabs/databend-perf/runs/7246282653?check_suite_focus=true#step:6:21)22/07/08 05:48:10 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
2022/07/08 05:48:11 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
20[22](https://github.com/datafuselabs/databend-perf/runs/7246282653?check_suite_focus=true#step:6:23)/07/08 05:48:13 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
2022/07/08 05:48:17 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
2022/07/08 05:48:25 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
2022/07/08 05:48:42 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
2022/07/08 05:49:14 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Started", TotalInstances:1}
2022/07/08 05:50:18 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Suspended", TotalInstances:1}
2022/07/08 05:52:[26](https://github.com/datafuselabs/databend-perf/runs/7246282653?check_suite_focus=true#step:6:27) org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Suspended", TotalInstances:1}
2022/07/08 05:56:43 org: orgchasen-slug-test warehouse status: &framework.WarehouseStatusDTO{Name:"query-perf", ReadyInstances:0, Size:"***", State:"Suspended", TotalInstances:1}
2022/07/08 05:56:43 Error creating warehouse: All attempts fail:

it'd better to make a standalone tenant to seperate the perf-test warehouse.

[Feat]: Create share links for each card chart

Goal: Add a share link for each card chart, Easy to share and spread.

ontime Q3 hangs 2 hours

2022/07/16 02:38:02 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT Origin, count(*) AS c FROM default.ontime WHERE DepDelay\u003e10 AND Year \u003e= 2000 AND Year \u003c= 2008 GROUP BY Origin ORDER BY c DESC LIMIT 10;"} resp={"id":"dcdf9631-585a-4796-b678-4833cb7a8cd6","session_id":"7fd7ab6d-264a-47a2-881a-027c986054c5","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":60000000,"bytes":1020000480},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1025.8725849999998},"stats_uri":"/v1/query/dcdf9631-585a-4796-b678-4833cb7a8cd6","final_uri":"/v1/query/dcdf9631-585a-4796-b678-4833cb7a8cd6/final","next_uri":"/v1/query/dcdf9631-585a-4796-b678-4833cb7a8cd6/page/0","kill_uri":"/v1/query/dcdf9631-585a-4796-b678-4833cb7a8cd6/kill"}
2022/07/16 04:38:07 finish warm up 3 for query Q3 : SELECT Origin, count(*) AS c FROM default.ontime WHERE DepDelay>10 AND Year >= 2000 AND Year <= 2008 GROUP BY Origin ORDER BY c DESC LIMIT 10;

https://github.com/datafuselabs/databend-perf/runs/7367600623?check_suite_focus=true

Mobile view broken

No route to host (os error 113) while executing perf test

https://github.com/datafuselabs/databend-perf/runs/7313539246?check_suite_focus=true

2022/07/13 02:47:07 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"e9e5911a-183a-4a90-98b6-672ccd2d6602","session_id":"ef563545-1050-4e34-87ea-3a65c312e377","schema":{"fields":[{"name":"Carrier","default_expr":null,"data_type":{"type":"String"}},{"name":"count()","default_expr":null,"data_type":{"type":"UInt64"}}],"metadata":{}},"data":[["WN",296451],["AA",179769],["MQ",152292],["OO",147019],["US",140199],["UA",135061],["XE",108571],["EV",104055],["NW",102205],["DL",98427],["CO",81039],["YV",79553],["FL",64583],["OH",60532],["AS",54326],["B6",53716],["9E",48578],["F9",24100],["AQ",6764],["HA",4059]],"state":"Succeeded","error":null,"stats":{"scan_progress":{"rows":14000000,"bytes":224000112},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":20,"bytes":368},"running_time_ms":351.568896},"stats_uri":"/v1/query/e9e5911a-183a-4a90-98b6-672ccd2d6602","final_uri":"/v1/query/e9e5911a-183a-4a90-98b6-672ccd2d6602/final","next_uri":null,"kill_uri":"/v1/query/e9e5911a-183a-4a90-98b6-672ccd2d6602/kill"}
2022/07/13 02:47:08 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"68b1627f-32c5-4d5f-8587-db1552ffc8ef","session_id":"241713b9-b325-4e97-b72d-d0c96b321c87","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":14000000,"bytes":224000112},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1025.676934},"stats_uri":"/v1/query/68b1627f-32c5-4d5f-8587-db1552ffc8ef","final_uri":"/v1/query/68b1627f-32c5-4d5f-8587-db1552ffc8ef/final","next_uri":"/v1/query/68b1627f-32c5-4d5f-8587-db1552ffc8ef/page/0","kill_uri":"/v1/query/68b1627f-32c5-4d5f-8587-db1552ffc8ef/kill"}
2022/07/13 02:51:56 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"647b9ae6-e0d6-4728-b146-3a1597b3c15c","session_id":"b8d757b2-3629-4351-a7a5-690a069be57e","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1340.341515},"stats_uri":"/v1/query/647b9ae6-e0d6-4728-b146-3a1597b3c15c","final_uri":"/v1/query/647b9ae6-e0d6-4728-b146-3a1597b3c15c/final","next_uri":"/v1/query/647b9ae6-e0d6-4728-b146-3a1597b3c15c/page/0","kill_uri":"/v1/query/647b9ae6-e0d6-4728-b146-3a1597b3c15c/kill"}
2022/07/13 02:51:58 finish warm up 3 for query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:51:58 the 0 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:51:59 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"274d3950-3c0b-48a4-a408-9898d9f9d466","session_id":"cd346487-996b-4443-b55e-9c0[334](https://github.com/datafuselabs/databend-perf/runs/7313539246?check_suite_focus=true#step:6:335)5ef64e","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1035.5629470000001},"stats_uri":"/v1/query/274d3950-3c0b-48a4-a408-9898d9f9d466","final_uri":"/v1/query/274d3950-3c0b-48a4-a408-9898d9f9d466/final","next_uri":"/v1/query/274d3950-3c0b-48a4-a408-9898d9f9d466/page/0","kill_uri":"/v1/query/274d3950-3c0b-48a4-a408-9898d9f9d466/kill"}
2022/07/13 02:52:01 the 0 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.88.192:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:01 the 1 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:02 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"c054c0b2-dc2b-4a0c-ac63-232c658dfa92","session_id":"8c9c5ca2-e847-43ae-a020-cc625c412276","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1033.3019530000001},"stats_uri":"/v1/query/c054c0b2-dc2b-4a0c-ac63-232c658dfa92","final_uri":"/v1/query/c054c0b2-dc2b-4a0c-ac63-232c658dfa92/final","next_uri":"/v1/query/c054c0b2-dc2b-4a0c-ac63-232c658dfa92/page/0","kill_uri":"/v1/query/c054c0b2-dc2b-4a0c-ac63-232c658dfa92/kill"}
2022/07/13 02:52:04 the 1 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.105.210:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:04 the 2 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:05 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"e02f5010-cbf2-4cdf-92c5-fe59ac240c10","session_id":"20b9c7e4-8caa-4e7b-af8f-4cf0406e8444","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1035.120751},"stats_uri":"/v1/query/e02f5010-cbf2-4cdf-92c5-fe59ac240c10","final_uri":"/v1/query/e02f5010-cbf2-4cdf-92c5-fe59ac240c10/final","next_uri":"/v1/query/e02f5010-cbf2-4cdf-92c5-fe59ac240c10/page/0","kill_uri":"/v1/query/e02f5010-cbf2-4cdf-92c5-fe59ac240c10/kill"}
2022/07/13 02:52:07 the 2 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.113.118:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:07 the 3 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:09 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"87280ff5-b4f1-4baa-89fb-2f6ef408d9b6","session_id":"64364c3c-f7c6-46bc-a69e-a10e088a6023","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1033.769738},"stats_uri":"/v1/query/87280ff5-b4f1-4baa-89fb-2f6ef408d9b6","final_uri":"/v1/query/87280ff5-b4f1-4baa-89fb-2f6ef408d9b6/final","next_uri":"/v1/query/87280ff5-b4f1-4baa-89fb-2f6ef408d9b6/page/0","kill_uri":"/v1/query/87280ff5-b4f1-4baa-89fb-2f6ef408d9b6/kill"}
2022/07/13 02:52:11 the 3 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.88.192:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:11 the 4 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:12 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"9d1654c1-3d27-4e59-acc6-64157795f7d5","session_id":"bc59cc97-647c-424a-aa2c-3278deb93499","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1035.648265},"stats_uri":"/v1/query/9d1654c1-3d27-4e59-acc6-64157795f7d5","final_uri":"/v1/query/9d1654c1-3d27-4e59-acc6-64157795f7d5/final","next_uri":"/v1/query/9d1654c1-3d27-4e59-acc6-64157795f7d5/page/0","kill_uri":"/v1/query/9d1654c1-3d27-4e59-acc6-64157795f7d5/kill"}
2022/07/13 02:52:14 the 4 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.113.118:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:14 the 5 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:15 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"7302107a-50c0-4235-882c-f70b90741ff3","session_id":"7d6d077c-8c07-457d-a4b1-2403a44bf1d1","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1037.361202},"stats_uri":"/v1/query/7302107a-50c0-4235-882c-f70b90741ff3","final_uri":"/v1/query/7302107a-50c0-4235-882c-f70b90741ff3/final","next_uri":"/v1/query/7302107a-50c0-4235-882c-f70b90741ff3/page/0","kill_uri":"/v1/query/7302107a-50c0-4235-882c-f70b90741ff3/kill"}
2022/07/13 02:52:17 the 5 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.124.12:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:17 the 6 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:18 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"3e0640b2-4785-4429-819c-a9c3797e1890","session_id":"aba1fb06-30af-4c68-a998-d04365c833a8","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1036.750172},"stats_uri":"/v1/query/3e0640b2-4785-4429-819c-a9c3797e1890","final_uri":"/v1/query/3e0640b2-4785-4429-819c-a9c3797e1890/final","next_uri":"/v1/query/3e0640b2-4785-4429-819c-a9c3797e1890/page/0","kill_uri":"/v1/query/3e0640b2-4785-4429-819c-a9c3797e1890/kill"}
2022/07/13 02:52:28 the 6 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.107.63:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:28 the 7 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:29 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"9ebc4628-cf63-43ff-915e-2cb21b077125","session_id":"848b7555-5cc5-4418-972d-ae39a32ed93c","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1031.7258869999998},"stats_uri":"/v1/query/9ebc4628-cf63-43ff-915e-2cb21b077125","final_uri":"/v1/query/9ebc4628-cf63-43ff-915e-2cb21b077125/final","next_uri":"/v1/query/9ebc4628-cf63-43ff-915e-2cb21b077125/page/0","kill_uri":"/v1/query/9ebc4628-cf63-43ff-915e-2cb21b077125/kill"}
2022/07/13 02:52:31 the 7 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.67.71:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:31 the 8 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:32 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"07d31206-0bb6-4493-823a-17a9843bc5f3","session_id":"b37ee994-1423-48a4-bd1c-60fa6fbed852","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1035.87[352](https://github.com/datafuselabs/databend-perf/runs/7313539246?check_suite_focus=true#step:6:353)99999998},"stats_uri":"/v1/query/07d31206-0bb6-4493-823a-17a9843bc5f3","final_uri":"/v1/query/07d31206-0bb6-4493-823a-17a9843bc5f3/final","next_uri":"/v1/query/07d31206-0bb6-4493-823a-17a9843bc5f3/page/0","kill_uri":"/v1/query/07d31206-0bb6-4493-823a-17a9843bc5f3/kill"}
2022/07/13 02:52:41 the 8 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.90.67:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
2022/07/13 02:52:41 the 9 time to run query Q4 : SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay>10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;
2022/07/13 02:52:42 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM default.ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;"} resp={"id":"cef96421-d6ba-4f7e-ace7-92b8298abc0e","session_id":"c8b18255-3b93-4ee3-a56d-5038ace2b3ae","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1034.959242},"stats_uri":"/v1/query/cef96421-d6ba-4f7e-ace7-92b8298abc0e","final_uri":"/v1/query/cef96421-d6ba-4f7e-ace7-92b8298abc0e/final","next_uri":"/v1/query/cef96421-d6ba-4f7e-ace7-92b8298abc0e/page/0","kill_uri":"/v1/query/cef96421-d6ba-4f7e-ace7-92b8298abc0e/kill"}
2022/07/13 02:52:44 the 9 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.124.203:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
panic: test case failed 10 times

failed to get dsn on executing loading task

Run bendsql cloud login --email=*** --*** --org=orgchasen-slug-test
config file /home/runner/.config/bendsql/config.toml not found, creating a new one
time="2022-12-15T08:53:02Z" level=info msg="logged in orgchasen-slug-test of Databend Cloud http://webapi-service.cloud-platform.svc:8080/ successfully."
time="2022-12-15T08:53:02Z" level=info msg="you can use `bendsql cloud configure` to switch to another org and warehouse"
Run bendsql cloud warehouse create reload --size=Small
warehouse reload is creating, please wait...
warehouse reload created, size is SmallNow using warehouse reload
Run bash -ex ./reload/tpch/lineitem.sh
+ set -e
+ cat
+ bendsql query
failed to get dsn: please use `bendsql cloud configure` to select organization and warehouse first

extreme slow on execute Q12 ontime

SELECT OriginCityName, DestCityName, count(*) AS c FROM default.ontime GROUP BY OriginCityName, DestCityName ORDER BY c DESC LIMIT 10;

it's not slow in a 2 instances cluster but extreme slow on an 8 intances cluster.

unexpectedly long query run time

Typically, most queries on the ontime dataset can be completed in less than 1 second.

However, in several recent performance tests, some statements may experience severe timeouts, even taking tens of minutes, until jwt expired.

query	expected	actual	history
SELECT DayOfWeek, count(*) AS c FROM default.ontime WHERE DepDelay>10 AND Year >= 2000 AND Year <= 2008 GROUP BY DayOfWeek ORDER BY c DESC;	< 0.5s	42min
SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(DepDelay) * 1000 AS c3 FROM default.ontime WHERE Year >= 2000 AND Year <= 2008 GROUP BY Carrier;	< 1s	> 60min	https://github.com/datafuselabs/databend-perf/runs/7172566428?check_suite_focus=true
SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay\u003e10 as Int8))*1000 AS c3 FROM default.ontime WHERE Year\u003e=2000 AND Year \u003c=2008 GROUP BY Carrier ORDER BY c3 DESC;	<0.5s		https://github.com/datafuselabs/databend-perf/runs/7159465292?check_suite_focus=true

instance crashed during the perf test

2022/07/07 00:54:54 the 1 time to run query Q9 : SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 10;
2022/07/07 00:55:31 the 1 time result has error All attempts fail:
#1: query failed: request got bad status: 500 req={"sql":"SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 10;"} resp={"error":{"kind":"HealthCheckFailed","message":"health check failed: backend instance maybe crashed: Get \"[http://10.1.103.65:9000/health\](http://10.1.103.65:9000/health/)": dial tcp 10.1.103.65:9000: connect: connection refused"}}

perf.databend.rs is down?

Getting this on Chrome:

This site can’t be reached
Check if there is a typo in perf.databend.rs.
DNS_PROBE_FINISHED_NXDOMAIN

slow on COPY INTO tasks

datafuselabs/databend#6751

add a workflow to reload ontime dataset, and output the profiling of data ingest

currently we relies on the ontime dataset to run the perf profiling. but the perf dataset is mainly ingested by human being.

it'd better to ingest the data by an automatic workflow. this workflow can be human triggered or daily triggered.

also, it'd better to get the performance history about ontime dataset ingesting performance.

Feature: support tpch queries

cc @flaneur2020

[Tracking] Add hits dataset & queries

links:

• https://github.com/ClickHouse/ClickHouse/blob/484055984661a2ace7f6a62a1d9f9204ea7f04af/benchmark/compatible/databend/benchmark.sh
• https://datasets.clickhouse.com/hits_compatible/hits.csv.gz
• https://github.com/datafuselabs/databend/pull/6463/files

tasks:

• #81
• add hits data into repo.databend.rs/hits/hits.tsv
• add partitoned hits data into repo.databend.rs/hits_p/*.tsv
• datafuselabs/databend#6553
• datafuselabs/databend#6554
• #88
• add queries about hits

feat: Time range logic adjustment

Automatic raises performance degration issues to the databend repo

we've got databend-perf to discovery the performance degrations in databend. but the performance degration are mainly discovered by human-eyes, we can try raising github issues on the databend repo by a bot on getting performance degration.

Caveats

• only raise issues when the degration is obvious (like +10%) to reduce noise.
• only raise issues on scheduled perf tests, not raises on Pull Request workflows

Fix frontend display for long long query

Support ontime JOIN

Q5

SELECT Carrier, c, c2, c*100/c2 as c3
FROM
(
    SELECT
        IATA_CODE_Reporting_Airline AS Carrier,
        count(*) AS c
    FROM ontime
    WHERE DepDelay>10
        AND Year=2007
    GROUP BY Carrier
) q
JOIN
(
    SELECT
        IATA_CODE_Reporting_Airline AS Carrier,
        count(*) AS c2
    FROM ontime
    WHERE Year=2007
    GROUP BY Carrier
) qq USING Carrier
ORDER BY c3 DESC;

Q6

SELECT Carrier, c, c2, c*100/c2 as c3
FROM
(
    SELECT
        IATA_CODE_Reporting_Airline AS Carrier,
        count(*) AS c
    FROM ontime
    WHERE DepDelay>10
        AND Year>=2000 AND Year<=2008
    GROUP BY Carrier
) q
JOIN
(
    SELECT
        IATA_CODE_Reporting_Airline AS Carrier,
        count(*) AS c2
    FROM ontime
    WHERE Year>=2000 AND Year<=2008
    GROUP BY Carrier
) qq USING Carrier
ORDER BY c3 DESC;

Q7

SELECT Year, c1/c2
FROM
(
    select
        Year,
        count(*)*100 as c1
    from ontime
    WHERE DepDelay>10
    GROUP BY Year
) q
JOIN
(
    select
        Year,
        count(*) as c2
    from ontime
    GROUP BY Year
) qq USING (Year)
ORDER BY Year;

Invalid physical plan with PhysicalPlan::Exchange

2022/08/05 02:49:50 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT avg(number) FROM numbers_mt(10000000000);"} resp={"id":"","session_id":null,"session":null,"schema":null,"data":[],"state":"Failed","error":{"code":1015,"message":"Invalid physical plan with PhysicalPlan::Exchange"},"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":0.0},"affect":null,"stats_uri":null,"final_uri":null,"next_uri":null,"kill_uri":null}

[Tracking] perf-test stabilities

• #92
• #82
• #79
• #109
• #106

add etharchive data

https://github.com/deepeth/etharchive.org

add email warning

compare page

• deal data
• Display data in tabular form
  #20

[front]feat: Adaptive mobile terminal

• Adaptation to mobile
  #39

replace c5n.9xlarge to DatabendCloud Large Warehouse (64 cores)

take bendctl to replace perf-tool

now we have got the cli to access our cloud platform bendctl, we can consider replacing the cli perf-tool to bendctl.

the problems with perf-tool:

• the test logic is hard coded inside the cli tool, hard to customize the test logic by python script.
• the api client is manual written and have some adhoc logic inside, it does not handle some corner cases like 520 well.
• hard to reproduce the failed case on desktop.

the pros about bendctl:

• allowing us seperate the api client logic with the test workflow logic (like run each query 10 times for ontime dataset)
• have a more stable fundation about the api access logic like session management, warehouse management, retry, etc.

4000 error by aws s3 on loading data

time="2022-09-22T13:18:32Z" level=error msg="error on query: query has error: &{Code:4000 Message:other error (backend error: (context: {"bucket": "repo.databend.rs"}, source: sending request: https://s3.amazonaws.com/repo.databend.rs: Custom { kind: Interrupted, error: Transport { kind: Io, message: None, url: Some(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("s3.amazonaws.com")), port: None, path: "/repo.databend.rs", query: None, fragment: None }), source: Some(Custom { kind: ConnectionAborted, error: "Unexpected EOF" }) } })) Kind:}"

520 while COPY INTO hits

https://github.com/datafuselabs/databend-perf/runs/7383513221?check_suite_focus=true

2022/07/18 05:31:48 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"COPY INTO hits FROM 's3://repo.databend.rs/hits/hits.tsv' credentials=(aws_key_id='***' aws_secret_key='***') file_format=(type='CSV' field_delimiter='\\\\t' record_delimiter='\\\\n' skip_header=1);\n"} resp={"id":"453daa6f-8ac2-4392-84c1-14e48142dd85","session_id":"d2b1cf86-441b-4d44-b142-4c6dcde2dbeb","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":30000,"bytes":27196162},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1035.515376},"stats_uri":"/v1/query/453daa6f-8ac2-4392-84c1-14e48142dd85","final_uri":"/v1/query/453daa6f-8ac2-4392-84c1-14e48142dd85/final","next_uri":"/v1/query/453daa6f-8ac2-4392-84c1-14e48142dd85/page/0","kill_uri":"/v1/query/453daa6f-8ac2-4392-84c1-14e48142dd85/kill"}
2022/07/18 06:25:27 the 0 time result has error request got bad status: 520 req= resp=<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>test.datafusecloud.com | 520: Web server is returning an unknown error</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" />

Refactor the workflow

Current logic:

• Pull benchmark data from s3 in github actions
• Push data to the main branch
• Push transformed data to the gh-pages branch
• Frontend will be built to gh-pages either

With the power of vercel, we can improve the workflow to

• Pull benchmark data from s3 in github actions
• Push data to the main branch
• Build front and transfer data in vercel

Notes:

• We can do the json transform inside vite, so that we don't need to introduce golang here.
• Every time new data pushed or frontend changes, vercel will deploy a new version.

failed to reload data on executing select count(*) from hits

2022/07/07 08:31:28 the 0 time to run query Select-Count : SELECT count(*) FROM hits;
2022/07/07 08:31:30 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT count(*) FROM hits;"} resp={"id":"92d[58](https://github.com/datafuselabs/databend-perf/runs/7229673058?check_suite_focus=true#step:6:59)a24-a2d9-4706-9c46-d7c247b126cd","session_id":"4e8e1bc5-db35-4e8b-8e03-b4483662e9f5","schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":0,"bytes":0},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1016.9196229999999},"stats_uri":"/v1/query/92d58a24-a2d9-4706-9c46-d7c247b126cd","final_uri":"/v1/query/92d58a24-a2d9-4706-9c46-d7c247b126cd/final","next_uri":"/v1/query/92d58a24-a2d9-4706-9c46-d7c247b126cd/page/0","kill_uri":"/v1/query/92d58a24-a2d9-4706-9c46-d7c247b126cd/kill"}
2022/07/07 08:31:39 the 0 time result has error query has error: &{Code:1038 Message:can not connect to http://10.1.65.197:9191/, error: tonic::transport::error::Error: transport error source: error trying to connect: tcp connect error: No route to host (os error 113) source: tcp connect error: No route to host (os error 113) source: No route to host (os error 113) Kind:}, stats: <nil>
panic: test case failed 1 times

hangs at SELECT Year, avg(DepDelay) FROM default.ontime GROUP BY Year;

2022/08/23 03:42:03 webapi.doRequest POST url=https://test.datafusecloud.com/v1/query req={"sql":"SELECT Year, avg(DepDelay) FROM default.ontime GROUP BY Year;"} resp={"id":"2cbeaf72-b0b4-414c-baba-f456b0b17305","session_id":"072f6478-ea50-49fb-a377-de874a944eb0","session":null,"schema":{"fields":[],"metadata":{}},"data":[],"state":"Running","error":null,"stats":{"scan_progress":{"rows":56558570,"bytes":339351420},"write_progress":{"rows":0,"bytes":0},"result_progress":{"rows":0,"bytes":0},"running_time_ms":1510.46[491](https://github.com/datafuselabs/databend-perf/runs/7965073878?check_suite_focus=true#step:6:492)60000001},"affect":null,"stats_uri":"/v1/query/2cbeaf72-b0b4-414c-baba-f456b0b17305","final_uri":"/v1/query/2cbeaf72-b0b4-414c-baba-f456b0b17305/final","next_uri":"/v1/query/2cbeaf72-b0b4-414c-baba-f456b0b17305/page/0","kill_uri":"/v1/query/2cbeaf72-b0b4-414c-baba-f456b0b17305/kill"}

https://github.com/datafuselabs/databend-perf/runs/7965073878?check_suite_focus=true

Pretty the *.json file

https://github.com/datafuselabs/databend-perf/blob/main/collector/ontime/2022-03-28-ontime.json

{
   "metadata":{
      "table":"ontime",
      "tag":"v0.7.0-nightly",
      "size":"XXXLarge"
   },
   "schema":[
      {
         "name":"Q1",
         "sql":"SELECT DayOfWeek, count(*) AS c FROM ontime WHERE Year \u003e= 2000 AND Year \u003c= 2008 GROUP BY DayOfWeek ORDER BY c DESC;",
         "min":0.135,
         "max":0.281,
         "median":0.151,
         "std_dev":0.041065313830531,
         "read_row":61000000,
         "read_byte":183000000,
         "time":[
            0.138,
            0.135,
            0.18,
            0.281,
            0.156,
            0.148,
            0.154,
            0.147,
            0.178,
            0.141
         ],
         "error":[
            
         ],
         "mean":0.1618845925136391
      },
      {
         "name":"Q2",
         "sql":"SELECT DayOfWeek, count(*) AS c FROM ontime WHERE DepDelay\u003e10 AND Year \u003e= 2000 AND Year \u003c= 2008 GROUP BY DayOfWeek ORDER BY c DESC;",
         "min":0.21,
         "max":0.38,
         "median":0.228,
         "std_dev":0.053633944475490525,
         "read_row":61000000,
         "read_byte":427000000,
         "time":[
            0.213,
            0.225,
            0.38,
            0.242,
            0.231,
            0.314,
            0.223,
            0.296,
            0.21,
            0.216
         ],
         "error":[
            
         ],
         "mean":0.25009052210415667
      },
      {
         "name":"Q3",
         "sql":"SELECT Origin, count(*) AS c FROM ontime WHERE DepDelay\u003e10 AND Year \u003e= 2000 AND Year \u003c= 2008 GROUP BY Origin ORDER BY c DESC LIMIT 10;",
         "min":0.277,
         "max":0.471,
         "median":0.2955,
         "std_dev":0.057627163733780955,
         "read_row":61000000,
         "read_byte":1037000488,
         "time":[
            0.374,
            0.306,
            0.303,
            0.321,
            0.286,
            0.277,
            0.471,
            0.278,
            0.285,
            0.288
         ],
         "error":[
            
         ],
         "mean":0.31453498226201787
      },
      {
         "name":"Q4",
         "sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, count() FROM ontime WHERE DepDelay\u003e10 AND Year = 2007 GROUP BY Carrier ORDER BY count() DESC;",
         "min":0.1,
         "max":0.141,
         "median":0.127,
         "std_dev":0.01398713694792469,
         "read_row":8000000,
         "read_byte":128000064,
         "time":[
            0.114,
            0.126,
            0.116,
            0.139,
            0.135,
            0.1,
            0.102,
            0.141,
            0.128,
            0.133
         ],
         "error":[
            
         ],
         "mean":0.12257341467963002
      },
      {
         "name":"Q5",
         "sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay\u003e10 as Int8))*1000 AS c3 FROM ontime WHERE Year=2007 GROUP BY Carrier ORDER BY c3 DESC;",
         "min":0.116,
         "max":0.156,
         "median":0.1405,
         "std_dev":0.011920151005754916,
         "read_row":8000000,
         "read_byte":128000064,
         "time":[
            0.141,
            0.141,
            0.148,
            0.116,
            0.14,
            0.133,
            0.155,
            0.134,
            0.125,
            0.156
         ],
         "error":[
            
         ],
         "mean":0.1383760315561474
      },
      {
         "name":"Q6",
         "sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(cast(DepDelay\u003e10 as Int8))*1000 AS c3 FROM ontime WHERE Year\u003e=2000 AND Year \u003c=2008 GROUP BY Carrier ORDER BY c3 DESC;",
         "min":0.319,
         "max":0.394,
         "median":0.337,
         "std_dev":0.025049750497759454,
         "read_row":61000000,
         "read_byte":976000488,
         "time":[
            0.329,
            0.319,
            0.338,
            0.329,
            0.337,
            0.394,
            0.337,
            0.342,
            0.393,
            0.331
         ],
         "error":[
            
         ],
         "mean":0.3440409873025485
      },
      {
         "name":"Q7",
         "sql":"SELECT IATA_CODE_Reporting_Airline AS Carrier, avg(DepDelay) * 1000 AS c3 FROM ontime WHERE Year \u003e= 2000 AND Year \u003c= 2008 GROUP BY Carrier;",
         "min":0.312,
         "max":0.332,
         "median":0.3205,
         "std_dev":0.006785278181474955,
         "read_row":61000000,
         "read_byte":976000488,
         "time":[
            0.322,
            0.319,
            0.331,
            0.325,
            0.316,
            0.314,
            0.322,
            0.313,
            0.312,
            0.332
         ],
         "error":[
            
         ],
         "mean":0.3205285749620005
      },
      {
         "name":"Q8",
         "sql":"SELECT Year, avg(DepDelay) FROM ontime GROUP BY Year;",
         "min":0.442,
         "max":0.483,
         "median":0.459,
         "std_dev":0.01346699669562593,
         "read_row":202687655,
         "read_byte":1216125930,
         "time":[
            0.474,
            0.452,
            0.448,
            0.468,
            0.481,
            0.456,
            0.462,
            0.483,
            0.452,
            0.442
         ],
         "error":[
            
         ],
         "mean":0.4616045035847688
      },
      {
         "name":"Q9",
         "sql":"SELECT Year, count(*) as c1 FROM ontime GROUP BY Year;",
         "min":0.275,
         "max":0.436,
         "median":0.2845,
         "std_dev":0.04607266000569102,
         "read_row":202687655,
         "read_byte":405375310,
         "time":[
            0.285,
            0.286,
            0.282,
            0.306,
            0.436,
            0.276,
            0.287,
            0.282,
            0.284,
            0.275
         ],
         "error":[
            
         ],
         "mean":0.2970543413447912
      },
      {
         "name":"Q10",
         "sql":"SELECT avg(cnt) FROM (SELECT Year,Month,count(*) AS cnt FROM ontime WHERE DepDel15=1 GROUP BY Year,Month) a;",
         "min":0.379,
         "max":0.62,
         "median":0.4065,
         "std_dev":0.07188351688669663,
         "read_row":202687655,
         "read_byte":1418813585,
         "time":[
            0.509,
            0.402,
            0.446,
            0.411,
            0.386,
            0.379,
            0.429,
            0.389,
            0.62,
            0.385
         ],
         "error":[
            
         ],
         "mean":0.43048758810748167
      },
      {
         "name":"Q11",
         "sql":"SELECT avg(c1) FROM (SELECT Year,Month,count(*) AS c1 FROM ontime GROUP BY Year,Month) a;",
         "min":0.363,
         "max":0.421,
         "median":0.3805,
         "std_dev":0.015829087149927503,
         "read_row":202687655,
         "read_byte":608062965,
         "time":[
            0.363,
            0.378,
            0.421,
            0.373,
            0.383,
            0.394,
            0.384,
            0.367,
            0.385,
            0.37
         ],
         "error":[
            
         ],
         "mean":0.38148231464266147
      },
      {
         "name":"Q12",
         "sql":"SELECT OriginCityName, DestCityName, count(*) AS c FROM ontime GROUP BY OriginCityName, DestCityName ORDER BY c DESC LIMIT 10;",
         "min":2.714,
         "max":2.773,
         "median":2.751,
         "std_dev":0.018573098825990245,
         "read_row":202687655,
         "read_byte":8577734458,
         "time":[
            2.727,
            2.72,
            2.75,
            2.714,
            2.737,
            2.765,
            2.758,
            2.773,
            2.752,
            2.752
         ],
         "error":[
            
         ],
         "mean":2.744737085189844
      },
      {
         "name":"Q13",
         "sql":"SELECT OriginCityName, count(*) AS c FROM ontime GROUP BY OriginCityName ORDER BY c DESC LIMIT 10;",
         "min":1.024,
         "max":1.119,
         "median":1.0495,
         "std_dev":0.023473602194805973,
         "read_row":202687655,
         "read_byte":4288897398,
         "time":[
            1.045,
            1.061,
            1.051,
            1.119,
            1.059,
            1.057,
            1.048,
            1.042,
            1.024,
            1.047
         ],
         "error":[
            
         ],
         "mean":1.055045298266354
      },
      {
         "name":"Q14",
         "sql":"SELECT count(*) FROM ontime;",
         "min":0.013,
         "max":0.036,
         "median":0.015,
         "std_dev":0.006564297372910522,
         "read_row":1,
         "read_byte":1,
         "time":[
            0.013,
            0.019,
            0.014,
            0.013,
            0.015,
            0.036,
            0.016,
            0.017,
            0.013,
            0.015
         ],
         "error":[
            
         ],
         "mean":0.016254610291963664
      }
   ]
}

pre-run the Queries before loop 10 times

current order:

for 10 times {
  Q1
}
for 10 times {
  Q2
}
...
for 10 times{
  Q15
}

when got stable error on one query, it wasted lots of time on rerun the ci jobs.

we can consider add a seperate job before the 10 times perf executions:

  Q1
  Q2
  Q3
  ..
  Q15

thus we can fail retry the single job quickly.

add tpch test

• replace reload script with bendctl
• load data into repo.databend.rs
• add schema about tpch
• add reload script
• add tpch queries