a7ex / xcresultparser Goto Github PK

If run tests simultaneously on 2 devices for example we run tests on iPhone + iPad simultaneously, and as I understood the xcresult structure will be different from the run on 1 device, so txt results will be corrupted (it contains wrong failed tests count + doesn't contain any data about failed tests and locales)

and txt result after run the command: xcresultparser -o txt -s failed -f result.xcresult > results.txt

Would it be possible for you to add binaries to the releases here on GitHub? It would be helpful for a workflow we're using at my job.

We have a testplan with ~17 configurations (the difference between configurations are system region + system language) we use it to perform visual testing on different locales, also we've added 2 native retries in testplan configuration. I made some changes to fail 1 test on 2 locales.
And after generation txt result with this command:
xcresultparser -o txt -s failed -f result.xcresult > results.txt

I see in Summary 4 failed tests, but I assume that number should be 2, because other 2 it's just a retries, so the correct number should be 2 failed.

Hey! We have found a bit of a bug/issue when using xcresultparser to convert a .xcresult file using the cobertura method. We use the number of hits in the outputted report to verify how many times something has been tested, however the created cobertura report has the incorrect amount of hits by a large amount.

After spending a while analysing this we figured out that the number of hits reported in the cobertura file matches the first digit of the number of hits found in Xcode. As an example I've attached a couple images below:

(Left Xcode with the number of hits in the right hand bar, Right Cobertura report for the associated file from Xcode)

In the above image, line 70 in Xcode (case "body":), the tests report that the line gets 12k test hits but in cobertura it reports the hits as 1.

Line 71 in Xcode gets hit 5346 times but in cobertura it reports the hits as 5.
This supports our theory that the converter is only getting the first digit from the .xcresultbundle file as the number of hits to include in the report.

This is consistent with our other files, I've added another below:

xcresultparser version - 1.5.2
xcode version - 15.3

I've tried running the parser from brew and the binary from the latest release (1.4.2) and they both error out with the same error message/code combo:

XcresultparserLib/resource_bundle_accessor.swift:40: Fatal error: unable to find bundle named Xcresultparser_XcresultparserLib

The command I'm running follows the following pattern:

xcresultparser -p my-dir -o cobertura results/result.xcresult > output/coverage.xml

The weird thing to me is that I was running the parser successfully yesterday, using the exact same command, but today it just doesn't work. This makes me think that I'm the one making the error, but I can't figure out any difference between yesterday and today. One thing that I do notice is that the Package.swift declares a XcresultparserLib library and target, but there is no such directory with that name in Sources, only xcresultparser. I'm left wondering if that might be the issue.

First of all thanks for the great project!
It would be nice if it's possible to add json format to output type since we are using xcresult summaries in slack report messages, so It would be much easier if the whole summary data would be in json format, cuz it's easier to work with this format later :)

If the cobertura.sourceforge.net server is down, the tool will fatalError
Why not back up a local dtd file to avoid this problem ? @fadookie

Hey, first of all, thanks for the awesome tool. We're using it internally for test result processing, and it's been great so far.

We'd like to extend the JUnit XML output to include additional attributes included in the .xcresult bundle, like the name of the source file of the test, the column and row number where the failure occurred, etc.

We're more than happy to contribute upstream, we'd love to hear if you're interested so we can get aligned on this. :)

When I run xcresultparser -o xml Test-logging.xcresult
output is like below.

<?xml version="1.0" encoding="UTF-8"?>
<testExecutions version="1">
    <file path="LoggingTests">
        <testCase name="testWhenSuccess()" duration="5"/>
        <testCase name="testWhenFailure()" duration="17"/>
    </file>
</testExecutions>

but when I passed this to sonar-scanner, it wasn't processed.
because sonar-scanner expects,
<file path="Full/Path/LoggingTests.swift">
https://docs.sonarqube.org/latest/analyzing-source-code/test-coverage/generic-test-data/#generic-test-execution

Is it possible to put the fullpath info here?

Our project repo has capital letters in it, and we're using $(pwd) to get the current path (and using the --project-root option to correct the file paths so SonarQube can ingest the xml files). Test execution output seems to work fine with this (the paths are properly turned into relative paths), but coverage output seems to require the exact case-sensitive path as the option for --project-root.

Is it possible to ignore specific directories/files? I'm currently using the parser for a Swift package and the parser is including in the results the files in ~/Library/Developer/Xcode/DerivedData which are coming from other package dependencies for which I'm not interested in their code coverage.

Thank you for this awesome project, it's been really useful.

We have a testplan with ~17 configurations (the difference between configurations are system region + system language) we use it to perform visual testing on different locales. Some of the tests are disabled:
example:

func testOne() throws {
throw XCTSkip("Disabled")
}

In txt report I see that skipped tests shows as failed on different locales, but I assume they should be excluded from the report and shouldn't be recognized as failed tests

the command I used to generate txt report is:
xcresultparser -o txt -s failed -f result.xcresult > results.txt

Hey!

Around 2-4pm BST Thur 13th June, I started to consistently get an issue on my build servers where xcresultparser is called. The issue simply says that the bundle named Xcresultparser_XcresultparserLib cannot be found. We've made no updates to pipelines or CI runners, just seems to have happened out of the blue.

CI output of a GitLab job that calls xcresultparser

$ xcresultparser -o cobertura --quiet --project-root "$CI_PROJECT_PATH/" "build_report.xcresult" > "cobertura.xml"
XcresultparserLib/resource_bundle_accessor.swift:44: Fatal error: unable to find bundle named Xcresultparser_XcresultparserLib
bash: line 284: 87547 Trace/BPT trap: 5       xcresultparser -o cobertura --quiet --project-root "$CI_PROJECT_PATH/" "build_report.xcresult" > "cobertura.xml"

Unfortunately I don't have any ideas what could have caused this, any help/ideas would be appreciated!

Note: Thanks for rolling out 1.6.0 we've been using it for 2 weeks without any issues

Hello we have an issue where using the markdown format wont render span tag to Github actions step summary. Is there an existing way to fix this? If not I suggest having a new output format for markdown compatible with Github. Github has some alternatives for highlighting text listed here https://stackoverflow.com/questions/11509830/how-to-add-color-to-githubs-readme-md-file and in documentation https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax

Example command and result below.

https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

      - if: failure()
        name: Generate Github Markdown
        run: |
          xcresultparser \
              --failed-tests-only \
              --output-format md \
              --quiet \
              'reports/TestResultsBundle.xcresult' >> $GITHUB_STEP_SUMMARY

I see that release/1.6.0 was merged into main - should a GitHub release + tag be made for it now?

On two separate macOS machines, one with fresh brew install xcresultparser and one with brew upgrade xcresultparser, running into build issues with 1.6.2 that just came out to fix the DTD problem.

==> Upgrading a7ex/formulae/xcresultparser
1.6.0 -> 1.6.2
==> make install prefix=/opt/homebrew/Cellar/xcresultparser/1.6.2
Last 15 lines from /Users/mark.thormann/Library/Logs/Homebrew/xcresultparser/01.make:
Building targets in dependency order
Target dependency graph (10 targets)
0%: Gather provisioning inputs
0%: Create build description
0%: Discovering version info for swiftc
0%: Discovering version info for clang
0%: Discovering version info for ld
Build description signature: 8bf363b728c1427a1a4c5b54073e4b96
Build description path: /private/tmp/xcresultparser-20240614-29219-4nmvyk/xcresultparser-1.6.2/.build/apple/Intermediates.noindex/XCBuildData/8bf363b728c1427a1a4c5b54073e4b96.xcbuilddata
Missing package product 'SwiftLintBuildToolPlugin@11'
Missing package product 'SwiftLintBuildToolPlugin@11'
Build cancelled
make: *** [build] Error 1
If reporting this issue please do so at (not Homebrew/brew or Homebrew/homebrew-core):
https://github.com/a7ex/homebrew-formulae/issues

I'm evaluating a few options for producing html from xcresult bundles and xcresultparser is one of them.
I just want to ask if you have considered publishing the library on Homebrew? It would make a CI setup a lot easier.
Thanks