a7ex / xcresultparser Goto Github PK
View Code? Open in Web Editor NEWParse the binary xcresult bundle from Xcode builds and testruns
License: MIT License
Parse the binary xcresult bundle from Xcode builds and testruns
License: MIT License
Our project repo has capital letters in it, and we're using $(pwd)
to get the current path (and using the --project-root
option to correct the file paths so SonarQube can ingest the xml files). Test execution output seems to work fine with this (the paths are properly turned into relative paths), but coverage output seems to require the exact case-sensitive path as the option for --project-root
.
If run tests simultaneously on 2 devices for example we run tests on iPhone + iPad simultaneously, and as I understood the xcresult structure will be different from the run on 1 device, so txt results will be corrupted (it contains wrong failed tests count + doesn't contain any data about failed tests and locales)
and txt result after run the command: xcresultparser -o txt -s failed -f result.xcresult > results.txt
I see that release/1.6.0
was merged into main
- should a GitHub release + tag be made for it now?
We have a testplan with ~17 configurations (the difference between configurations are system region + system language) we use it to perform visual testing on different locales. Some of the tests are disabled:
example:
func testOne() throws {
throw XCTSkip("Disabled")
}
In txt report I see that skipped tests shows as failed on different locales, but I assume they should be excluded from the report and shouldn't be recognized as failed tests
the command I used to generate txt report is:
xcresultparser -o txt -s failed -f result.xcresult > results.txt
If the cobertura.sourceforge.net server is down, the tool will fatalError
Why not back up a local dtd file to avoid this problem ? @fadookie
We have a testplan with ~17 configurations (the difference between configurations are system region + system language) we use it to perform visual testing on different locales, also we've added 2 native retries in testplan configuration. I made some changes to fail 1 test on 2 locales.
And after generation txt result with this command:
xcresultparser -o txt -s failed -f result.xcresult > results.txt
I see in Summary 4 failed tests, but I assume that number should be 2, because other 2 it's just a retries, so the correct number should be 2 failed.
Trying to use the output jnit xml file in Jenkins generates the following:
WARNING: The file '/UnitTests.xml' is an invalid file.
WARNING: At line 3 of file:/UnitTests.xml:cvc-pattern-valid: Value '0.9729' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type 'SUREFIRE_TIME'.
WARNING: At line 3 of file:/UnitTests.xml:cvc-complex-type.4: Attribute 'errors' must appear on element 'testsuite'.
WARNING: The result file '/UnitTests.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
It looks like the Jenkins plugin (https://plugins.jenkins.io/xunit/) expects:
.
in the time
attributeerrors
attribute, even if errors=0
on the testsuite
I've tried running the parser from brew and the binary from the latest release (1.4.2
) and they both error out with the same error message/code combo:
XcresultparserLib/resource_bundle_accessor.swift:40: Fatal error: unable to find bundle named Xcresultparser_XcresultparserLib
The command I'm running follows the following pattern:
xcresultparser -p my-dir -o cobertura results/result.xcresult > output/coverage.xml
The weird thing to me is that I was running the parser successfully yesterday, using the exact same command, but today it just doesn't work. This makes me think that I'm the one making the error, but I can't figure out any difference between yesterday and today. One thing that I do notice is that the Package.swift
declares a XcresultparserLib
library and target, but there is no such directory with that name in Sources
, only xcresultparser
. I'm left wondering if that might be the issue.
I'm evaluating a few options for producing html from xcresult bundles and xcresultparser is one of them.
I just want to ask if you have considered publishing the library on Homebrew? It would make a CI setup a lot easier.
Thanks
Would it be possible for you to add binaries to the releases here on GitHub? It would be helpful for a workflow we're using at my job.
Hello we have an issue where using the markdown format wont render span
tag to Github actions step summary. Is there an existing way to fix this? If not I suggest having a new output format for markdown compatible with Github. Github has some alternatives for highlighting text listed here https://stackoverflow.com/questions/11509830/how-to-add-color-to-githubs-readme-md-file and in documentation https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax
Example command and result below.
https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/
- if: failure()
name: Generate Github Markdown
run: |
xcresultparser \
--failed-tests-only \
--output-format md \
--quiet \
'reports/TestResultsBundle.xcresult' >> $GITHUB_STEP_SUMMARY
Is it possible to ignore specific directories/files? I'm currently using the parser for a Swift package and the parser is including in the results the files in ~/Library/Developer/Xcode/DerivedData which are coming from other package dependencies for which I'm not interested in their code coverage.
Thank you for this awesome project, it's been really useful.
Hey, first of all, thanks for the awesome tool. We're using it internally for test result processing, and it's been great so far.
We'd like to extend the JUnit XML output to include additional attributes included in the .xcresult
bundle, like the name of the source file of the test, the column and row number where the failure occurred, etc.
We're more than happy to contribute upstream, we'd love to hear if you're interested so we can get aligned on this. :)
On two separate macOS machines, one with fresh brew install xcresultparser
and one with brew upgrade xcresultparser
, running into build issues with 1.6.2 that just came out to fix the DTD problem.
==> Upgrading a7ex/formulae/xcresultparser
1.6.0 -> 1.6.2
==> make install prefix=/opt/homebrew/Cellar/xcresultparser/1.6.2
Last 15 lines from /Users/mark.thormann/Library/Logs/Homebrew/xcresultparser/01.make:
Building targets in dependency order
Target dependency graph (10 targets)
0%: Gather provisioning inputs
0%: Create build description
0%: Discovering version info for swiftc
0%: Discovering version info for clang
0%: Discovering version info for ld
Build description signature: 8bf363b728c1427a1a4c5b54073e4b96
Build description path: /private/tmp/xcresultparser-20240614-29219-4nmvyk/xcresultparser-1.6.2/.build/apple/Intermediates.noindex/XCBuildData/8bf363b728c1427a1a4c5b54073e4b96.xcbuilddata
Missing package product 'SwiftLintBuildToolPlugin@11'
Missing package product 'SwiftLintBuildToolPlugin@11'
Build cancelled
make: *** [build] Error 1
If reporting this issue please do so at (not Homebrew/brew or Homebrew/homebrew-core):
https://github.com/a7ex/homebrew-formulae/issues
First of all thanks for the great project!
It would be nice if it's possible to add json format to output type since we are using xcresult summaries in slack report messages, so It would be much easier if the whole summary data would be in json format, cuz it's easier to work with this format later :)
Hey!
Around 2-4pm BST Thur 13th June, I started to consistently get an issue on my build servers where xcresultparser is called. The issue simply says that the bundle named Xcresultparser_XcresultparserLib
cannot be found. We've made no updates to pipelines or CI runners, just seems to have happened out of the blue.
CI output of a GitLab job that calls xcresultparser
$ xcresultparser -o cobertura --quiet --project-root "$CI_PROJECT_PATH/" "build_report.xcresult" > "cobertura.xml"
XcresultparserLib/resource_bundle_accessor.swift:44: Fatal error: unable to find bundle named Xcresultparser_XcresultparserLib
bash: line 284: 87547 Trace/BPT trap: 5 xcresultparser -o cobertura --quiet --project-root "$CI_PROJECT_PATH/" "build_report.xcresult" > "cobertura.xml"
Unfortunately I don't have any ideas what could have caused this, any help/ideas would be appreciated!
Note: Thanks for rolling out 1.6.0 we've been using it for 2 weeks without any issues
Unable to download the DTD
http://cobertura.sourceforge.net/xml/coverage-04.dtd
Similar ticket here which was closed in the past.
#19
Current Error:
An error has been encountered in accessing this page.
Reporting this problem: The problem you have encountered is with a project web site hosted by SourceForge.net. This issue should be reported to the SourceForge.net-hosted project (not to SourceForge.net).
If this is a severe or recurring/persistent problem, please do one of the following, and provide the error text (numbered 1 through 7, above):
Contact the project via their designated support resources.
Contact the project administrators of this project via email (see the upper right-hand corner of the Project Summary page for their usernames) at [email protected]
If you are a maintainer of this web content, please refer to the Site Documentation regarding web services for further assistance.
NOTE: As of 2008-10-23 directory index display has been disabled by default. This option may be re-enabled by the project by placing a file with the name ".htaccess" with this line:
Options +Indexes
When I run xcresultparser -o xml Test-logging.xcresult
output is like below.
<?xml version="1.0" encoding="UTF-8"?>
<testExecutions version="1">
<file path="LoggingTests">
<testCase name="testWhenSuccess()" duration="5"/>
<testCase name="testWhenFailure()" duration="17"/>
</file>
</testExecutions>
but when I passed this to sonar-scanner, it wasn't processed.
because sonar-scanner expects,
<file path="Full/Path/LoggingTests.swift">
https://docs.sonarqube.org/latest/analyzing-source-code/test-coverage/generic-test-data/#generic-test-execution
Is it possible to put the fullpath info here?
Hey! We have found a bit of a bug/issue when using xcresultparser to convert a .xcresult file using the cobertura method. We use the number of hits in the outputted report to verify how many times something has been tested, however the created cobertura report has the incorrect amount of hits by a large amount.
After spending a while analysing this we figured out that the number of hits reported in the cobertura file matches the first digit of the number of hits found in Xcode. As an example I've attached a couple images below:
(Left Xcode with the number of hits in the right hand bar, Right Cobertura report for the associated file from Xcode)
In the above image, line 70 in Xcode (case "body":
), the tests report that the line gets 12k test hits but in cobertura it reports the hits as 1.
Line 71 in Xcode gets hit 5346 times but in cobertura it reports the hits as 5.
This supports our theory that the converter is only getting the first digit from the .xcresultbundle file as the number of hits to include in the report.
This is consistent with our other files, I've added another below:
xcresultparser version - 1.5.2
xcode version - 15.3
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.