When a test crashes, trainer outputs something about how the test crashed into the failure node (i.e. whatever is picked up from the TestSummaries.plist). Sometimes this has what the main thread was doing when the test crashed; sometimes it just says the test crashed. When a test crashes, the .crash files are available in the xcresult bundle as an "attachment." Inside of the TestSummaries.plist, attachments that belong to a given test are listed for each test. It would be great if trainer would take advantage of this to make richer results when a test crashes, by dumping the contents of the .crash file into the failure node (maybe after a nice horizontal separator to separate crash info from test output).

With this improvement, you could get the full crash details without having to download the xcresult bundle from your CI environment and dig through the .crash files yourself (as well as pair the .crash files to their tests yourself).

When running tests against multiple destinations (e.g. iOS 8 + 9 simulators), xcodebuild produces a separate TestSummaries.plist for each destination. It would be super cool to aggregate these into a single JUnit report.

I propose flattening all the test runs into a single array of testsuites in the resulting XML. If we do that, we'll need to have some metadata at the testsuite level to distinguish one run of a target from another on a different destination.

Using a properties element allows us to add arbitrary key-value pairs to the testsuite.

Lately, I'm only able to have trainer run successfully when scan succeeds and passes all tests. If even one test fails, I get the following error:

    Unit tests failed

Followed by the sad explosion icon next to trainer and the fastlane finished with errors message.

I wanted to look into this so I saw there was a commit 7 days ago, the only instance of the message I could find the following, from the trainer_action.rb file, lines 11-13:

resulting_paths.each do |path, test_successful|
    UI.test_failure!("Unit tests failed") unless test_successful
end

I can provide any working examples you might need to reproduce this. I'm very stumped after an hour or so of searching for the solution. If my tests pass, trainer runs. If any tests fail, trainer kills the lane.

Some CI environments are capable of surfacing test attachments alongside the test report. XCTest supports attachments, so supporting viewing those attachments in the test reports would be fabulous. Supporting the known environments would seem to require dumping the attachments from the test results file format into images in a structured directory tree and adding a format option to massage the test results into NUnit 3 format in addition to the default of the JUnit report format.

Known CI Environments Supporting Test Attachments

Jenkins

Jenkins support for test attachments is available using the JUnit Attachments plugin. It can correlate files in a directory tree named to match the test tree and display them together.

Azure Pipelines

Azure Pipelines relies on NUnit output format support to surface test attachments. The attachments can then be accessed from a tab when the test failure info is pulled up.

I'm using:

• trainer 0.2.0
• fastlane 1.99.0
• scan 0.10.1

When attempting to run trainer within fastlane, I get the following error:

fastlane-plugin-trainer-0.2.0/lib/fastlane/plugin/trainer/actions/trainer_action.rb:7:in `run': [!] uninitialized constant Fastlane::Actions::SharedValues::SCAN_GENERATED_PLIST_FILE (NameError)

Does this code depend on a version of something newer that I don't have? What could be the issue here?

Xcode 11 uses a new format for xctest (version 3), which doesn't have a TestSummary.plist file, and uses Zstandard-compressed "database-like" files.

Xcode 11 Release Notes state:

The format of result bundles changed in Xcode 11. A result bundle is an asset produced by Xcode 11 with the xcresult file extension that contains information about the build, tests, code coverage, and more. Any xcresult files produced with Xcode 10 or earlier cannot be read by Xcode 11. A result bundle can be produced by passing -resultBundlePath ./Example.xcresult to an xcodebuild invocation and the Example.xcresult can then be opened in Xcode. Xcode also creates result bundles in Derived Data. The current result bundle version number is 3, which can be specified by passing the xcodebuild flag -resultBundleVersion 3. Version 3 is the default in Xcode 11, but it is still recommended for automation to explicitly pass the flag, so that any potential future versions that become the default do not cause issues to existing tools. Result bundles can be inspected using xcresulttool. A JSON representation of the root object of the result bundle can be exported using xcrun xcresulttool get --format json --path ./Example.xcresult and any nested object, identified by its reference found in the JSON output, can be exported by adding the flag --id REF. xcresulttool also provides the description of its format using xcrun xcresulttool formatDescription. (41633595)

Basically, to obtain results from an Xcode 11 xcresult, one would have to do the following:

// The output of this command will be useful to automatically parse the output of the next commands
$ xcrun xcresulttool formatDescription get --format json
[...]

$ xcrun xcresulttool get --format json --path Project.xcresult
[...]
          "testsRef" : {
            "_type" : {
              "_name" : "Reference"
            },
            "id" : {
              "_type" : {
                "_name" : "String"
              },
              "_value" : "0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=="
            },
[...]

// We take the value of the `testsRef` key (if any) for each `ActionRecord` entry
$ xcrun xcresulttool get --format json --path Project.xcresult --id '0~YVEC1htpkY3jLmeu4DB8ah-sJueP7_vUdqA2rL1_R3xT_aftqqEgGUSCHZ0mmpojBSyy4mEL4LX2xyFlfr4pEQ=='
{
  "_type" : {
    "_name" : "ActionTestPlanRunSummaries"
  },
  "summaries" : {
    "_type" : {
      "_name" : "Array"
    },
    "_values" : [
[...]

Attempting to install trainer fails, as the required dependencies cannot be found.

Gemfile:

source "https://rubygems.org"
gem 'trainer'

Running bundle install fails with the following error:

Bundler could not find compatible versions for gem "credentials_manager":
  In Gemfile:
    trainer was resolved to 0.1.1, which depends on
      fastlane_core (< 1.0.0, >= 0.48.1) was resolved to 0.48.1, which depends on
        credentials_manager (< 1.0.0, >= 0.16.0)

Could not find gem 'credentials_manager (< 1.0.0, >= 0.16.0)', which is required by gem 'fastlane_core (< 1.0.0, >= 0.48.1)', in any of the sources.

Similar errors occurred when I specified a version number in the Gemfile.

I've installed trainer using bundler (I have this in my Gemfile: gem 'trainer', '0.7.0')

When I try to run it: bundle exec trainer --help

I receive the following error:

LoadError: cannot load such file -- fastlane
  /Users/cosminstirbu/api-sdk-ios/ruby/2.2.0/gems/trainer-0.7.0/lib/trainer.rb:1:in `require'
  /Users/cosminstirbu/api-sdk-ios/ruby/2.2.0/gems/trainer-0.7.0/lib/trainer.rb:1:in `<top (required)>'
  /Users/cosminstirbu/api-sdk-ios/ruby/2.2.0/gems/trainer-0.7.0/bin/trainer:4:in `require'
  /Users/cosminstirbu/api-sdk-ios/ruby/2.2.0/gems/trainer-0.7.0/bin/trainer:4:in `<top (required)>'
  /Users/cosminstirbu/api-sdk-ios/ruby/2.2.0/bin/trainer:23:in `load'
  /Users/cosminstirbu/api-sdk-ios/ruby/2.2.0/bin/trainer:23:in `<top (required)>'

LOGS:

fastlane add_plugin trainer [✔] 🚀 [14:48:58]: fastlane detected a Gemfile in the current directory [14:48:58]: however it seems like you don't use bundle exec`
[14:48:58]: to launch fastlane faster, please use
[14:48:58]:
[14:48:58]: $ bundle exec fastlane add_plugin trainer
[14:48:58]:
[14:48:58]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
+-------------------------+---------+---------+
| Used plugins |
+-------------------------+---------+---------+
| Plugin | Version | Action |
+-------------------------+---------+---------+
| fastlane-plugin-trainer | 0.4.1 | trainer |
+-------------------------+---------+---------+

[14:49:00]: Make sure to commit your Gemfile, Gemfile.lock and Pluginfile to version control
Installing plugin dependencies...

Your user account isn't allowed to install to the system RubyGems.
You can cancel this installation and run:

  bundle install --path vendor/bundle

to install the gems into ./vendor/bundle/, or you can enter your password
and install the bundled gems to RubyGems using sudo.

Password:

Your user account isn't allowed to install to the system RubyGems.
You can cancel this installation and run:

  bundle install --path vendor/bundle

to install the gems into ./vendor/bundle/, or you can enter your password
and install the bundled gems to RubyGems using sudo.

Password:
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory:
/private/var/folders/4q/5xhgmkwj5ysgwjl7dw09nn740000gn/T/bundler20191011-86603-1qbmauajson-2.2.0/gems/json-2.2.0/ext/json/ext/generator
/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby -r ./siteconf20191011-86603-1elvwhw.rb extconf.rb
mkmf.rb can't find header files for ruby at /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/folders/4q/5xhgmkwj5ysgwjl7dw09nn740000gn/T/bundler20191011-86603-1qbmauajson-2.2.0/gems/json-2.2.0 for inspection.
Results logged to
/var/folders/4q/5xhgmkwj5ysgwjl7dw09nn740000gn/T/bundler20191011-86603-1qbmauajson-2.2.0/extensions/universal-darwin-18/2.3.0/json-2.2.0/gem_make.out

An error occurred while installing json (2.2.0), and Bundler cannot continue.
Make sure that gem install json -v '2.2.0' --source 'https://rubygems.org/' succeeds before bundling.

In Gemfile:
fastlane-plugin-trainer was resolved to 0.4.1, which depends on
trainer was resolved to 0.9.1, which depends on
fastlane was resolved to 2.133.0, which depends on
json`

If I run tests in fastlane like this

run_tests scheme: 'MyTests', output_types: 'junit'

fastlane automatically puts the test results in $SCAN_OUTPUT_DIRECTORY/MyTests.test_result and generates a junit report in $SCAN_OUTPUT_DIRECTORY.

I expected that the fastlane plugin for trainer would work like a drop-in replacement and so would mirror this behavior, but instead I have to run it like this

trainer path: "#{ENV['SCAN_OUTPUT_DIRECTORY']}/#{scheme_name}.test_result", extension: '.junit', output_directory: ENV['SCAN_OUTPUT_DIRECTORY']

Either the plugin should have default paths using those environment variables or it should have an option like replace_scan: true that causes it to use them. Ideally it could also check every *.test_result in its path and generate a matching report in the output directory, so a single run of trainer could generate all of your reports.

Not sure what this is about, but thought I would report it.

[10:42:51]: ---------------------
[10:42:51]: --- Step: trainer ---
[10:42:51]: ---------------------

+------------------+-------------------------+
|         Summary for trainer 0.7.0          |
+------------------+-------------------------+
| output_directory | ./build/BonMot-iOS/scan |
| path             | .                       |
| extension        | .xml                    |
+------------------+-------------------------+

[10:42:51]: `Helper.gem_path` is deprecated. Use the `ROOT` constant from the appropriate tool module instead.
Successfully generated './build/BonMot-iOS/scan/EA99A443-D009-4687-A938-CFCDEB5F97AB_TestSummaries.xml'

I would like to be able to extract the total number of tests executed and total failures from the xcresult file. Is that an output available with trainer or can it be added to the lane_context

Even though I have the fail_build setting set to false, occasionally if xcodebuild returns an exit status of 65, trainer will fail.

** TEST FAILED **
[09:13:18]: Exit status: 65
+--------------------+-----+
|       Test Results       |
+--------------------+-----+
| Number of tests    | 202 |
| Number of failures | 1   |
+--------------------+-----+

[09:13:18]: ---------------------
[09:13:18]: --- Step: trainer ---
[09:13:18]: ---------------------

+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                        Summary for trainer 0.8.0                                                                                        |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| output_directory | ./fastlane/test_output                                                                                                                                                               |
| extension        | .junit                                                                                                                                                                               |
| fail_build       | false                                                                                                                                                                                |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[09:13:18]: `Helper.gem_path` is deprecated. Use the `ROOT` constant from the appropriate tool module instead.
Successfully generated './fastlane/test_output/TestSummaries.junit'
+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                     Lane Context                                                                                                      |
+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| DEFAULT_PLATFORM           | ios                                                                                                                                                                                      |
| PLATFORM_NAME              | ios                                                                                                                                                                                      |
| LANE_NAME                  | ios pr                                                                                                                                                                                   |
| BUILD_NUMBER               | 81.289                                                                                                                                                                                   |
| SCAN_DERIVED_DATA_PATH     | /Users/jenkins/Library/Developer/Xcode/DerivedData/REDACTED-euytkosnmedfedgjjjodqbkwmzhe                                                                                          |
| SCAN_GENERATED_PLIST_FILES | ["/Users/jenkins/Library/Developer/Xcode/DerivedData/REDACTED-euytkosnmedfedgjjjodqbkwmzhe/Logs/Test/Run-REDACTED-2018.09.07_09-09-41--0400.xcresult/TestSummaries.plist"] |
| SCAN_GENERATED_PLIST_FILE  | /Users/jenkins/Library/Developer/Xcode/DerivedData/REDACTED-euytkosnmedfedgjjjodqbkwmzhe/Logs/Test/Run-REDACTED-2018.09.07_09-09-41--0400.xcresult/TestSummaries.plist     |
+----------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[09:13:18]: Unit tests failed

+------+-----------------------------------------+-------------+
|                       fastlane summary                       |
+------+-----------------------------------------+-------------+
| Step | Action                                  | Time (in s) |
+------+-----------------------------------------+-------------+
| 1    | default_platform                        | 0           |
| 2    | Verifying fastlane version              | 0           |
| 3    | default_platform                        | 0           |
| 4    | import_from_git                         | 0           |
| 5    | xcversion                               | 1           |
| 6    | cocoapods                               | 15          |
| 7    | Switch to ios incrementBuildNumber lane | 0           |
| 8    | increment_build_number                  | 2           |
| 9    | Switch to ios runTestsWithCoverage lane | 0           |
| 10   | reset_simulator_contents                | 20          |
| 11   | swiftlint                               | 1           |
| 12   | scan                                    | 237         |
| 💥   | trainer                                 | 0           |
+------+-----------------------------------------+-------------+

[09:13:18]: fastlane finished with errors

[!] Unit tests failed

Looking at the code, I’m not sure how the “Unit tests failed” message appears when fail_build is set to false. Here’s what the relevant portion of my Fastfile looks like:

    scan(
      scheme: ENV["SCAN_SCHEME"],
      workspace: ENV["SCAN_WORKSPACE"],
      configuration: 'Debug',
      output_types: '',
      device: ENV["DEVICE"] || 'iPhone 8 (12.0)',
      code_coverage: true,
      fail_build: false
    )

    trainer(
      output_directory: './fastlane/test_output',
      extension: '.junit',
      fail_build: false
    )

trying to migrate from xcpretty to trainer in order to parallelize unit tests on CI.

have very few other gems installed.

when i attempt to install trainer, it makes it through some portion of the gem installation, but then encounters the following.

Fetching unf_ext 0.0.7.5
Installing unf_ext 0.0.7.5 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /x/beta/.gems/gems/unf_ext-0.0.7.5/ext/unf_ext /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby -r ./siteconf20190307-81514-nw7co1.rb extconf.rb
checking for main() in -lstdc++... *** extconf.rb failed ***

not clear from the Readme.md or any other documentation here what pre-requisites are required prior to performing this gem install.

Hi,
i'm running "fastlane test" in a simple example project and using the "trainer" plugin to generate JUnit XML report.
However, whenever I run fastlane it is not able of finding the results.

This is my "fastlane/Fastfile"

default_platform(:ios)

platform :ios do
  desc "Run tests"
  lane :test do
    scan(scheme: "UnitTest-Calculator",
       output_types: "",
       fail_build: false)

    trainer(output_directory: "build/reports/")
  end

end

This is the output obtained:

+----------------------------+-------------------------------------------------------------------------------------------+
|                                                      Lane Context                                                      |
+----------------------------+-------------------------------------------------------------------------------------------+
| DEFAULT_PLATFORM           | ios                                                                                       |
| PLATFORM_NAME              | ios                                                                                       |
| LANE_NAME                  | ios test                                                                                  |
| SCAN_DERIVED_DATA_PATH     | /Users/smsf/Library/Developer/Xcode/DerivedData/UnitTest-Calculator-fbcsrzqmxnbtunckthqv  |
|                            | azzqktol                                                                                  |
# This file contains the fastlane.tools configuration
| SCAN_GENERATED_PLIST_FILES | []                                                                                        |
| SCAN_GENERATED_PLIST_FILE  |                                                                                           |
+----------------------------+-------------------------------------------------------------------------------------------+
[12:10:06]: No test result files found in directory '.', make sure the file name ends with 'TestSummaries.plist' or '.xcresult'

+------+------------------+-------------+
|           fastlane summary            |
+------+------------------+-------------+
| Step | Action           | Time (in s) |
+------+------------------+-------------+
| 1    | default_platform | 0           |
| 2    | scan             | 23          |
| 💥   | trainer          | 0           |
+------+------------------+-------------+

[12:10:06]: fastlane finished with errors

[!] No test result files found in directory '.', make sure the file name ends with 'TestSummaries.plist' or '.xcresult'

This is the content of the build directory, under the "DerivedData" default one:

$ ls /Users/smsf/Library/Developer/Xcode/DerivedData/UnitTest-Calculator-fbcsrzqmxnbtunckthqvazzqktol/Logs/Test/
LogStoreManifest.plist                                     Run-UnitTest-Calculator-2019.10.24_12-05-37-+0100.xcresult

The "Run-UnitTest-Calculator-2019.10.24_12-05-37-+0100.xcresult" is not a file but a directory instead.
It does not contain any TestSummaries.plist neither a .xcresult file.

Am I missing something basic in the configuration of fastlane or in the build itself to generate .xresult file? As it seems, XCode 11 is generating a .xcresult directory with many files.
From the trainer code, from my understanding it does not go into .xcresult directories under the DerivedData folder.. but anyway, I don't see a specific .xcresult there.

I just want a command line tool that will convert the Xcode plist files to JUnit XML reports. This seems like it would be a relatively straightforward and lightweight task.

The summary for trainer looked promising:

Convert xcodebuild plist files to JUnit reports

Great, that's what I want to do. Is this lightweight?

trainer was built to have the minimum code to generate JUnit reports for your CI system.

Great. "Minimum code" sounds nice.

So I decide to install this. I run:

sudo gem install trainer

Then I am horrified by the number of gems that get installed on my laptop. It takes 4 minutes and 30 seconds to install everything. When it is over, it has installed 58 gems, including google-cloud-storage, slack-notifier, uber, etc. Why do I need the Slack Notifier to parse a plist file?? I don't understand. I feel totally violated.

Details of the requirements:

trainer: Installed using gem and not using fastlane.
CI: Jenkins

To convert xcodebuild output to xml I am using trainer tool.

I have a test suite of 20 test cases written in swift using XCTest UITesting and setup a jenkins job to run the suite. The job runs on scheduled time and send an Email to the recipients. But, when I received the Email, the email is empty.
I am using Email-plugin for jenkins that uses xml file generated by job and uses jelly script to send the Emails in html/text format.
When I am checking console output of test job, it is nicely formatted.

xcodebuild -workspace ../../NYTimes.xcworkspace -scheme "$scheme" -sdk "$1" -destination "$2" -derivedDataPath $3 test
run trainer --path ../NYTimesUITests/test-output/Logs/Test -e .xml --output ../NYTimesUITests/Scripts/build/reports

In first line above, using test-output for 3 and it is generating the test output in the workspace path.
In second line I am running trainer to specify the path where to find the derived data path and it should convert the output to xml at desired location.

Please help me in resolving this issue. I already had a Email discussion with Felix and as suggested i am creating this github issue.

I know I am missing something but not able to figure it out.
I am attaching screenshots here.

I'm using trainer 0.5.0 standalone (not as a fastlane plugin). I'm using RVM ruby and installing the gem with bundle.

I get this output:

$ bundle exec trainer
/Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/lib/trainer.rb:1:in `require': cannot load such file -- fastlane_core (LoadError)
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/lib/trainer.rb:1:in `<top (required)>'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/bin/trainer:4:in `require'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/gems/trainer-0.5.0/bin/trainer:4:in `<top (required)>'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/trainer:22:in `load'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/trainer:22:in `<main>'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `eval'
	from /Users/max.friedrich/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `<main>'

fastlane_core is indeed required in lib/trainer.rb but was recently removed as a dependency in trainer.gemspec.

My scheme includes two test targets, one for logic tests and one for application tests. Xcode generates one test summaries plist for both, although when one of them fails to launch (thanks Xcode), the resulting plist contains empty list of tests for that target.

Here's an excerpt from plist:

<dict><!-- the target that has failed to launch -->
        <key>ProjectPath</key>
        <string>Whatever.xcodeproj</string>
        <key>TargetName</key>
        <string>LogicTests</string>
        <key>TestName</key>
        <string>LogicTests</string>
        <key>TestObjectClass</key>
        <string>IDESchemeActionTestableSummary</string>
        <key>Tests</key>
        <array/>
</dict>

Trainer considers this as a success.

I'm not sure how to solve this, though.

In fastlane/fastlane#19624, the code from this repo was copied directly into the Fastlane repo. Since then, there have been further updates to the trainer code within Fastlane, for example a new feature in fastlane/fastlane#19957 a couple of weeks ago.

On the other hand, this repo has not been updated for several months. Is the intention that all further development of trainer will take place within the Fastlane fork of the code? If so, should this repo be updated with a message to inform people and avoid potential confusion? Perhaps it should even be archived.

There is support now for skipping tests in XCTest ( https://developer.apple.com/documentation/xctest/methods_for_skipping_tests?language=objc )

Would be great to have support for processing this output in trainer.

This may just be user error, but I'm not seeing a difference in the junit reports generated when xcpretty_naming is true vs when it is false. Looking at the code, it looks like this is the expected behavior when the testData is in a .xcresult file. Anyway, not sure if the bug is in the code, the documentation, or just good old-fashioned user error.

expected output:

<testsuites name='Target.xctest' tests='8' failures='0'>
  <testsuite name = 'Target.TestClass1' tests='4' failures='0'>
      <testcase classname='Target.TestClass1' name ='test1' time='.001'/>
      <testcase classname='Target.TestClass1' name ='test2' time='.001'/>
      <testcase classname='Target.TestClass1' name ='test3' time='.001'/>
      <testcase classname='Target.TestClass1' name ='test4' time='.001'/>
  </testsuite>
  <testsuite name = 'Target.TestClass2' tests='4' failures='0'>
      <testcase classname='Target.TestClass2' name ='test1' time='.001'/>
      <testcase classname='Target.TestClass2' name ='test2' time='.001'/>
      <testcase classname='Target.TestClass2' name ='test3' time='.001'/>
      <testcase classname='Target.TestClass2' name ='test4' time='.001'/>
  </testsuite>
</testsuites>

actual output:

<testsuites tests='8' failures='0'>
  <testsuite name = 'Target' tests='8' failures='0' time='.008>
      <testcase classname='TestClass1' name ='test1()' time='.001'>
      </testcase>
      <testcase classname='TestClass1' name ='test2()' time='.001'>
      </testcase>
      <testcase classname='TestClass1' name ='test3()' time='.001'>
      </testcase>
      <testcase classname='TestClass1' name ='test4()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test1()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test2()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test3()' time='.001'>
      </testcase>
      <testcase classname='TestClass2' name ='test4()' time='.001'>
      </testcase>
 </testsuite>
</testsuites>

Hello,

The JUnit XML file format is not standardised across the heterogeneous testing tools.
In particular, the failure node is parsed in different ways by different tools although it's a critical component of the test report.

To solve this problem I propose to introduce formatting options.
Format:

• f : file_name
• l : line_number
• m : message

Usage:
trainer -assert_message_format "m (f:l)"
produces
<failure message="the assert message (file_name:12)"/>

trainer -assert_message_format "m" -failure_string_format "f:l"
produces
<failure message="the assert message">file_name:12</failure>

To cover even more situations, we could also add the failure type (But I don't think it's managed by the current version of the code).

What do you think ?