marketsquare / dbbot

DbBot is a tool to serialize Robot Framework test run results into a SQLite database.

This project is currently not maintained.

DbBot

DbBot is a Python script to serialize Robot Framework output files into a SQLite database. This way the future Robot Framework related tools and plugins will have a unified storage for the test run results.

Requirements

• Python 2.6 or newer installed
• Robot Framework 2.7 or newer installed

Robot Framework version 2.7.4 or later is recommended as versions prior to 2.7.4 do not support storing total elapsed time for test runs or tags.

How it works

The script takes one or more output.xml files as input, initializes the database schema, and stores the respective results into a database (robot_results.db by default, can be changed with options -b or --database). If database file is already existing, it will insert the new results into that database.

Installation

This tool is installed with pip with command:

$ pip install dbbot

Alternatively you can download the source distribution, extract it and install using:

$ python setup.py install

What is stored

Both the test data (names, content) and test statistics (how many did pass or fail, possible errors occurred, how long it took to run, etc.) related to suites and test cases are stored by default. However, keywords and related data are not stored as it might take order of magnitude longer for massive test runs. You can choose to store keywords and related data by using -k or --also-keywords flag.

Usage examples

Typical usage with a single output.xml file:

python -m dbbot.run atest/testdata/one_suite/output.xml

If the database does not already exist, it's created. Otherwise the test results are just inserted into the existing database. Only new results are inserted.

The default database is a file named robot_results.db.

Additional options are:

Short format	Long format	Description
-k	--also-keywords	Parse also suites' and tests' keywords
-v	--verbose	Print output to the console.
-b DB_FILE_PATH	--database=DB_FILE_PATH	SQLite database for test run results
-d	--dry-run	Do everything except store the results.

Specifying custom database name:

$ python -m dbbot.run  -b my_own_database.db atest/testdata/one_suite/output.xml

Parsing test run results with keywords and related data included:

python -m dbbot.run -k atest/testdata/one_suite/output.xml

Giving multiple test run result files at the same time:

python -m dbbot.run atest/testdata/one_suite/output.xml atest/testdata/one_suite/output_latter.xml

Database

You can inspect the created database using the sqlite3 command-line tool:

$ sqlite3 robot_results.db

sqlite> .tables
arguments        suite_status     test_run_errors  tests
keyword_status   suites           test_run_status
keywords         tag_status       test_runs
messages         tags             test_status

sqlite> SELECT count(), tests.id, tests.name
        FROM tests, test_status
        WHERE tests.id == test_status.test_id AND
        test_status.status == "FAIL"
        GROUP BY tests.name;

Please note that when database is initialized, no indices are created by DbBot. This is to avoid slowing down the inserts. You might want to add indices to the database by hand to speed up certain queries in your own scripts.

For information about the database schema, see doc/robot_database.md.

Migrating from Robot Framework 2.7 to 2.8

In Robot Framework 2.8, output.xml has changed slightly. Due this, the databases created with 2.7 need to migrated to be 2.8 compatible.

To migrate the existing database, issue the following script:

python tools/migrate27to28 -b <path_to_robot_results_db>

Use case example: Most failing tests

One of the common use cases for DbBot is to get a report of the most commonly failing suites, tests and keywords. There's an example for this purpose in examples/FailBot/bin/failbot.

Failbot is a Python script used to produce a summary web page of the failing suites, tests and keywords, using the information stored in the DbBot database. Please adjust (the barebone) HTML templates in examples/FailBot/templates to your needs.

Writing your own scripts

Please take a look at the modules in examples/FailBot/failbot as an example on how to build on top of the classes provided by DbBot to satisfy your own scripting needs.

License

DbBot is released under the Apache License, Version 2.0.

See LICENSE.TXT for details.

dbbot's Issues

Suite_id is empty

Hi,

I use DBBOT and when I record the file output.xml in the database, the field “suite_id” is empty in all the tables safe in the table “tests”.

thank you for your assistance

test_run_errors table is always empty...

When I execute the following commands:
$ python -m dbbot.run output_2pass_2fail.xml
$ sqlite3 robot_results.db
sqlite > .tables
sqlite > select * from test_run_errors;

test_run_errors appears to be empty???

I expected it to contain the 2 errors as shown in the log.html and screen shot of the output.xml file.

Dbbot does not save keyword statuses correctly

Dbbot saves one instance of a keyword to the join table keyword_status. This is wrong, as keyword can be referred multiple times in test execution (which most of the time is the case) and Dbbot should save all of these instances

Update database documentation

Update doc/robot_database.md and doc/robot_database.png

It would be nice if these could be generated automatically

unable to Parse keywords and error messages.

Traceback (most recent call last):
File "C:\Users\Arjun.Choyikkandy\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Arjun.Choyikkandy\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\run.py", line 53, in
DbBot().run()
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\run.py", line 44, in run
self._parser.xml_to_db(xml_file, self._options.build_number, self._options.branch_repo)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 56, in xml_to_db
self._parse_suite(test_run.suite, test_run_id, suite_metadata)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 127, in _parse_suite
self._parse_suites(suite, test_run_id, suite_metadata, suite_id)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 147, in _parse_suites
[self._parse_suite(subsuite, test_run_id, suite_metadata, parent_suite_id) for subsuite in suite.suites]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 147, in
[self._parse_suite(subsuite, test_run_id, suite_metadata, parent_suite_id) for subsuite in suite.suites]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 132, in _parse_suite
self._parse_keywords(suite.keywords, test_run_id, suite_id, None)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in _parse_keywords
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 219, in _parse_keyword
self._parse_keywords(keyword.keywords, test_run_id, None, None, keyword_id)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in _parse_keywords
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 219, in _parse_keyword
self._parse_keywords(keyword.keywords, test_run_id, None, None, keyword_id)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in _parse_keywords
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 219, in _parse_keyword
self._parse_keywords(keyword.keywords, test_run_id, None, None, keyword_id)
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in _parse_keywords
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 190, in
[self._parse_keyword(keyword, test_run_id, suite_id, test_id, keyword_id) for keyword in keywords]
File "D:\sunday\d\DbBot-SQLAlchemy\dbbot\reader\robot_results_parser.py", line 215, in _parse_keyword
self._parse_messages(keyword.messages, keyword_id)
AttributeError: 'For' object has no attribute 'messages'

dbbot startup script for windows

It would be good if DbBot works on Windows too. This means that the project needs dbbot.bat or similar that calls dbbot correctly in Windows environment

Test case start and end time can be added to test_status table

In test_runs table, the column started_at and finished_at are added for the entire suite to get the time elapsed.
There is a requirement to get the start and end time for every test case.
In test_status table, two new columns can be started_at and finished_at can be added to get the time for every test case.

tests only verify test_run_errors == 0; never verify when test_run_errors != 0

See issue #16

Add copyright notice to source files

DbBot should list copyright in the source files

Critical test fails in CI

On CI, one critical test fails with wrong error message

See: http://robot.radiaatto.ri.fi/job/DbBot

Error from Jenkins console output:

With --files and an invalid XML file | FAIL |
'dbbot: error: Invalid XML: Reading XML source '/var/lib/jenkins/jobs/DbBot/workspace/atests/dbbot/../testdata/invalid_output.xml' failed: SyntaxError: mismatched tag: line 452, column 6
' does not contain 'dbbot: error: Invalid XML: Reading XML source '/var/lib/jenkins/jobs/DbBot/workspace/atests/dbbot/../testdata/invalid_output.xml' failed: ParseError:'

Add support for using different databases

DbBot should enable users to choose which database is used.

dbbot/reader/database_writer.py and dbbot/reader/robot_results_parser.py are also poorly named, as they are sqlite3-specific. they should be renamed.

It should be possible to give database-specific options from command line too

Tool can not be used in Robot Framework 3.1 + Python 3.5

I am new to this tool but found it quite useful.
But when I try to parse the report which generated by Robot Framework 3.1 + Python 3.5, It failed to run.

agilis@ubuntu:~/Code/Agilis/Tests/SLC_MIT/TestReport$ python -m dbbot.run output.xml
Traceback (most recent call last):
File "/home/agilis/miniconda3/envs/py35/lib/python3.5/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/agilis/miniconda3/envs/py35/lib/python3.5/runpy.py", line 153, in _get_module_details
code = loader.get_code(mod_name)
File "", line 799, in get_code
File "", line 759, in source_to_code
File "", line 222, in _call_with_frames_removed
File "/home/agilis/miniconda3/envs/py35/lib/python3.5/site-packages/dbbot/run.py", line 43
except DataError, message:
^
SyntaxError: invalid syntax

AttributeError: 'TotalStatistics' object has no attribute 'all'

Issue found not able to move ahead.

(Robot) C:\Users\Rahul.Mangal>python -m dbbot.run E:/routput.xml
Traceback (most recent call last):
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\site-packages\dbbot\run.py", line 51, in <module>
    DbBot().run()
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\site-packages\dbbot\run.py", line 41, in run
    self._parser.xml_to_db(xml_file)
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\site-packages\dbbot\reader\robot_results_parser.py", line 51, in xml_to_db
    self._parse_suite(test_run.suite, test_run_id)
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\site-packages\dbbot\reader\robot_results_parser.py", line 116, in _parse_suite
    self._parse_suite_status(test_run_id, suite_id, suite)
  File "C:\Users\Rahul.Mangal\.conda\envs\Robot\lib\site-packages\dbbot\reader\robot_results_parser.py", line 125, in _parse_suite_status
    'passed': suite.statistics.all.passed,
AttributeError: 'TotalStatistics' object has no attribute 'all'

keyword_status -table has invalid FK

keyword_status has a foreign key to table keyword when it should be keywords

remove --files option

It is weird that file paths to output.xmls are given via command line option, as they are anyway mandatory. It should be removed.

<TypeError: 'NoneType' object has no attribute '__getitem__'> error occurred when i using dbbot convert xml to db

test files and output.xml: https://gist.github.com/QuentinJi/8488592

First I run pybot hello.txt world.txt. and then run dbbot command, and:

Traceback (most recent call last):
  File "bin/dbbot", line 39, in <module>
    DbBot().run()
  File "bin/dbbot", line 29, in run
    self._parser.xml_to_db(xml_file)
  File "/home/dbbot/reader/robot_results_parser.py", line 33, in xml_to_db
    self._parse_suite(test_run.suite, test_run_id)
  File "/home/dbbot/reader/robot_results_parser.py", line 86, in _parse_suite
    'source': suite.source
  File "/home/dbbot/reader/database_writer.py", line 171, in fetch_id
    return self._connection.execute(sql_statement, criteria.values()).fetchone()[0]
TypeError: 'NoneType' object has no attribute '__getitem__'

I think this issue may be caused by ExecutionResult(). When I execute ExecutionResult("output.xml"), the suite.source always be None. (RF issue?)

hiiiiiiiiiii

Test status text from output.xml is not parsed

It seems that error message in test status is not parsed from output.xml and saved to db. Fixing this would only require adding two lines of code (one for parsing and one for column creation):

1. robot_results_parser.py --> _parse_test_status --> 'message': test.message
2. database_writer.py --> _create_table_test_status --> 'message': 'TEXT'

E.g. "Keyword 'Wait Until Element Is Visible' failed ..." should be parsed (simplified example from output):
<test><status status="FAIL" endtime="20190201 13:21:01.497" critical="yes" starttime="20190201 13:20:09.986">Keyword 'Wait Until Element Is Visible' failed ...</status></test>

Tool installation

setup.py would be great and if the tool could be in pip it would be even greater!

Error when a top-level suite is created using rebot

I've rebot_output.xml generated using rebot with top level suite which is generated using rebot -o output.xml --name ConsolidatedOutput rebot_output.xml.

So here the top-level suite is ConsolidatedOutput and which does not contain the Start and End time. If I run the DbBot on the output.xml, then I get the following error:

python Traceback (most recent call last): File "C:\Python27\lib\runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "C:\Python27\lib\runpy.py", line 72, in _run_code exec code in run_globals File "C:\Python27\lib\site-packages\dbbot\run.py", line 51, in <module> DbBot().run() File "C:\Python27\lib\site-packages\dbbot\run.py", line 41, in run self._parser.xml_to_db(xml_file) File "C:\Python27\lib\site-packages\dbbot\reader\robot_results_parser.py", lin e 40, in xml_to_db 'started_at': self._format_robot_timestamp(test_run.suite.starttime), File "C:\Python27\lib\site-packages\dbbot\reader\robot_results_parser.py", lin e 215, in _format_robot_timestamp return datetime.strptime(timestamp, '%Y%m%d %H:%M:%S.%f') TypeError: must be string, not None

Allow continuous build information and metadata to be serialized along with test runs

It would be useful, if the dbbot could save a table for continuous integration server information for each test run. Analysing the test results in context of CI server information would be a useful feature.

Some of the columns that could be saved

CI_REPO - repository name of the current build
CI_BRANCH - branch name for the current build
CI_COMMIT - git sha for the current build
CI_BUILD_NUMBER - build number for the current build
CI_PULL_REQUEST - pull request number of the current build
CI_BUILD_URL - url for the current build
CI_TAG - tag name for the current build 

This is essentially more metadata for a test run and can be provided as a separate file or arguments to the command line.

dbbot tests don't work in Windows

In acceptance tests, the path to dbbot is resolved wrong. Also, this should point to correct startup script depending on the environment (dbbot.bat in Windows, dbbot everywhere else)

hiiiiiiiiiii

dbbot doesn't work in solaris

I'm using Solaris on x86 machine and copy DbBot-master to ~/pkgs.
The version of RF is 2.8.4 and the version of Python is 2.7.2

I go to the folder ~/pkgs/DbBot-master/bin and execute the command ./dbbot ~/yqleng_20140309_225128_output.xml (This file is generated by pybot. Please get it in the attachment.)

Then there is some error returned as below:
Traceback (most recent call last):
File "./dbbot", line 52, in
DbBot().run()
File "./dbbot", line 32, in init
self._db = DatabaseWriter(database_path, verbose_stream)
File "/home/users/yqleng/pkgs/DbBot-master/dbbot/reader/database_writer.py", line 21, in init
self._init_schema()
File "/home/users/yqleng/pkgs/DbBot-master/dbbot/reader/database_writer.py", line 25, in _init_schema
self._create_table_test_runs()
File "/home/users/yqleng/pkgs/DbBot-master/dbbot/reader/database_writer.py", line 46, in _create_table_test_runs
}, ('hash',))
File "/home/users/yqleng/pkgs/DbBot-master/dbbot/reader/database_writer.py", line 161, in _create_table
self._connection.execute(sql_statement)
sqlite3.OperationalError: disk I/O error.

Please note:
I tried the same SQL in python as below and it works:

import sqlite3
con=sqlite3.connect("./file.db")
cur=con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS test_runs (id INTEGER PRIMARY KEY, started_at DATETIME, finished_at DATETIME, source_file TEXT, hash TEXT NOT NULL, imported_at DATETIME NOT NULL, CONSTRAINT unique_test_runs UNIQUE (hash))")
<sqlite3.Cursor object at 0x28c160>

I also tried in Window7 and dbbot worked well and generated the file robot_results.db.

Improvement request: Avoiding orphans in the case of deletion of parent table’s record

Assuming that your Sqlite db is growing too big and you like to delete irrelevant records. We are currently having a db that has size of almost 600G.
Of course, the simplest way to solve this is to delete whole db and let DbBot create new one for you. But this way the fetching of history data is difficult, since after delete even yesterday is gone.
Then, if you like to be bit clever, you may like to delete all records from the robot_results.db that are older than certain date and time. Or make a clever script that leaves only those records of test runs that are related to some delivered SW and deletes everything else.
As you know, deleting record(s) is trivial: DELETE FROM test_runs WHERE test_runs.finished_at < '2014-08-08 08:00:00.00000'. So that’s it, or is it after all? No, because this isn’t the whole story, since doing this (with current robot_results.db) leaves bunch of orphan records that are not deleted /1/. And sadly for us, this is correct behavior with current DbBot setup.
Solution to this would be that, if Foreign Key Constraints could have clause ‘ON DELETE CASCADE’. Then possible delete action of a parent table’s record is propagated, and no orphan will be left.
So I suggest that DbBot would have this ‘ON DELETE CASCADE’ clause in every Foreign Key. I can’t imagine any user case in where you want to delete one record from the parent table, and want to leave the orphans. So this should be a safe move.

/1/ Of course, this wouldn’t be allowed, if PRAGMA foreign_keys = ON; hasn’t been entered, but this isn't a solution either.