launchableinc / cli Goto Github PK

View Code? Open in Web Editor NEW

23.0 23.0 11.0 165.72 MB

Launchable CLI

Home Page: https://pypi.org/project/launchable/

License: Apache License 2.0

Python 88.95% Dockerfile 0.06% Java 9.63% Ruby 0.01% C# 0.26% Gherkin 0.40% Starlark 0.69%

cli's People

Contributors

Stargazers

Watchers

Forkers

alexeagle yoshiori tchia04 jshinn704 takanabe mamh-mixed tammysue321 basil vsanna lemeurherve awilkes

cli's Issues

Launchable is killing test runs

Due to errors that launchable runs into, it ends up exiting with a non-zero status and killing a test run, despite assurances that adding Launchable to one's testing processes will not make one's test process more fragile. Having Launchable kill a test run that would have otherwise run fine is adding fragility to our test runs. I have complained about this multiple times in the past but this just seems to go on and on. I cannot convince my team to continue to use Launchable if it's going to make our CI processes more fragile.

Here's the latest:

[2022-08-11T05:29:46.918Z] + launchable record build --name jenkins-daos-stack-daos-ci-daos-do-packer-ofed-images-PR-2-release-2.0-20 --source src=.
[2022-08-11T05:29:47.854Z] Exception in thread "main" java.lang.IllegalArgumentException: One of setGitDir or setWorkTree must be called.
[2022-08-11T05:29:47.854Z] 	at org.eclipse.jgit.lib.BaseRepositoryBuilder.requireGitDirOrWorkTree(BaseRepositoryBuilder.java:636)
[2022-08-11T05:29:47.854Z] 	at org.eclipse.jgit.lib.BaseRepositoryBuilder.setup(BaseRepositoryBuilder.java:600)
[2022-08-11T05:29:47.854Z] 	at org.eclipse.jgit.lib.BaseRepositoryBuilder.build(BaseRepositoryBuilder.java:625)
[2022-08-11T05:29:47.854Z] 	at com.launchableinc.ingest.commits.CommitIngester.run(CommitIngester.java:133)
[2022-08-11T05:29:47.854Z] 	at com.launchableinc.ingest.commits.CommitIngester.main(CommitIngester.java:74)
[2022-08-11T05:29:47.854Z] Can't get commit history from `/data/jenkins/jenkins-1/ci_vm_wolf-106vm1-9/workspace/ker-ofed-images-PR-2-release-2.0`. Do you run command root of git-controlled directory? If not, please set a directory use by --source option.
f
[2022-08-11T05:29:47.854Z] fatal: Not a git repository (or any parent up to mount point /data)
[2022-08-11T05:29:47.854Z] Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
[2022-08-11T05:29:47.854Z] Can't get commit hash. Do you run command under git-controlled directory? If not, please set a directory use by --source option.
[2022-08-11T05:29:47.854Z] Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
[2022-08-11T05:29:47.854Z] Command '['java', '-jar', '/localhome/lcl_builder/.local/lib/python3.6/site-packages/launchable/jar/exe_deploy.jar', 'ingest:commit', '-endpoint', 'https://api.mercury.launchableinc.com/intake/', '-max-days', '30', '-scrub-pii', '/data/jenkins/jenkins-1/ci_vm_wolf-106vm1-9/workspace/ker-ofed-images-PR-2-release-2.0']' returned non-zero exit status 1.

Error handling and reporting

Time estimate way understated

When I try to create a subset I get a time estimate that is entirely unreasonably low and a large number of tests. It's orthogonal, but I wonder if the large number of tests is because Launchable can request 423 tests thinking they will take less than a second in total.

I.e.:

[2022-10-28T17:25:25.462Z] + launchable record build --name jenkins-daos-stack-daos-PR-10701-1 --source src=.
[2022-10-28T17:25:28.742Z] Launchable recorded 1 commit from repository /data/jenkins/jenkins-1/ci_vm_wolf-106vm1-9/workspace/daos-stack_daos_PR-10701
[2022-10-28T17:25:28.742Z] Launchable recorded build jenkins-daos-stack-daos-PR-10701-1 to workspace daos-stack/daos with commits from 2 repositories:
[2022-10-28T17:25:28.742Z] 
[2022-10-28T17:25:28.742Z] | Name             | Path           | HEAD Commit                              |
[2022-10-28T17:25:28.742Z] |------------------|----------------|------------------------------------------|
[2022-10-28T17:25:28.742Z] | src              | .              | 23b728a5dc225a743020390a3d57ed16c962a1df |
[2022-10-28T17:25:28.742Z] | src/src/rdb/raft | ./src/rdb/raft | c18bcb8ef18d5937558e881cd48a7568982aa954 |
[2022-10-28T17:25:28.742Z] + launchable subset --target 80% --build jenkins-daos-stack-daos-PR-10701-1 pytest
[2022-10-28T17:25:43.649Z] dfuse/daos_build.py::test_dfuse_daos_build_data
[2022-10-28T17:25:43.649Z] pool/destroy.py::test_destroy_wrong_group
[2022-10-28T17:25:43.649Z] dfuse/daos_build.py::test_dfuse_daos_build_wt
...
[2022-10-28T17:25:43.656Z] object/integrity.py::test_invalid_tx_commit_close
[2022-10-28T17:25:43.656Z] harness/advanced.py::test_launch_failures
[2022-10-28T17:25:43.656Z] cart/ghost_rank_rpc/ghost_rank_rpc_one_node.py::test_cart_ghost_rank_rpc
[2022-10-28T17:25:43.656Z] Launchable created subset 361113 for build jenkins-daos-stack-daos-PR-10701-1 (test session 1526834) in workspace daos-stack/daos
[2022-10-28T17:25:43.656Z] 
[2022-10-28T17:25:43.656Z] |           |   Candidates |   Estimated duration (%) |   Estimated duration (min) |
[2022-10-28T17:25:43.656Z] |-----------|--------------|--------------------------|----------------------------|
[2022-10-28T17:25:43.656Z] | Subset    |          423 |                  79.9622 |                 0.00705    |
[2022-10-28T17:25:43.656Z] | Remainder |          106 |                  20.0378 |                 0.00176667 |
[2022-10-28T17:25:43.656Z] |           |              |                          |                            |
[2022-10-28T17:25:43.656Z] | Total     |          529 |                 100      |                 0.00881667 |
[2022-10-28T17:25:43.656Z] 
[2022-10-28T17:25:43.656Z] Run `launchable inspect subset --subset-id 361113` to view full subset details

I believe the timing data we are submitting for tests is being specified properly in the junit submissions. Here's an example:

<testsuite errors="0" failures="0" name="job-2022-10-27T21.20-fa88ba3" skipped="0" tests="1" time="53.097" timestamp="2022-10-27T21:21:00.283826">
<testcase classname="SimpleCreateDeleteTest" name="test_container_basics" file="./container/simple_create_delete.py" time="53.097"/>
...
</testsuite>

Any ideas why Launchable thinks our tests run so quickly?

Launchable CLI verify command does not work on windows agents

Context

Testing Launchable on a Monolithic .Net 4.7.2 Application that can only be build using windows agents.

Current Behaviour

The launchable verify command returns the error [WinError 2] The system cannot find the file specified when attempting to check the java version, on a windows environment. This is because in java.py, "which" is being used to locate Java res = subprocess.run(["which", "java"], stdout=subprocess.DEVNULL) this does not work on windows.

Expected Behaviour

The launchable verify command to work successfully on a windows environment

Possible Solution

Inside java.py make a check on the current system, and if windows use 'where' to locate java and check the version
res = subprocess.run(["where", "java"], stdout=subprocess.DEVNULL)

Record Test Results fails on windows

Context

Testing Launchable on a Monolithic .Net 4.7.2 Application that can only be build using windows agents.

Current Behaviour

launchable record tests command fails on windows environment returning the following error:

Traceback (most recent call last): File "c:\users\kevin.gunn\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\users\kevin.gunn\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\Scripts\launchable.exe\__main__.py", line 7, in <module> File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 829, in __call__ return self.main(*args, **kwargs) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 1256, in invoke Command.invoke(self, ctx) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\core.py", line 610, in invoke return callback(*args, **kwargs) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\click\decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\launchable\commands\record\tests.py", line 60, in tests session_id = find_or_create_session(context, session, build_name) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\launchable\commands\helper.py", line 23, in find_or_create_session session_id = read_session(build_name) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\launchable\utils\session.py", line 34, in read_session f = _session_file_path(build_name) File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\launchable\utils\session.py", line 19, in _session_file_path return _session_file_dir() / (hashlib.sha1("{}:{}".format(build_name, _get_session_id()).encode()).hexdigest() + ".txt") File "C:\Users\kevin.gunn\AppData\Roaming\Python\Python38\site-packages\launchable\utils\session.py", line 23, in _get_session_id id = os.getsid(os.getpid()) AttributeError: module 'os' has no attribute 'getsid'

Expected Behaviour
The launchable record tests command to work successfully on a windows environment.

Trying to subset yields UnboundLocalError: local variable 'case' referenced before assignment

When I try to subset with:

$ launchable subset --session builds/jenkins-daos-stack-daos-PR-10182-5/test_sessions/1364476 --target 100% pytest < tests.txt 
Traceback (most recent call last):
  File "/home/brian/.local/bin/launchable", line 8, in <module>
    sys.exit(main())
  File "/home/brian/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/brian/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/brian/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/brian/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/brian/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/brian/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/brian/.local/lib/python3.10/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/brian/.local/lib/python3.10/site-packages/launchable/test_runners/pytest.py", line 60, in subset
    client.run()
  File "/home/brian/.local/lib/python3.10/site-packages/launchable/commands/subset.py", line 347, in run
    self.output_handler(_output, _rests)
  File "/home/brian/.local/lib/python3.10/site-packages/launchable/commands/subset.py", line 181, in _default_output_handler
    self.print(output)
  File "/home/brian/.local/lib/python3.10/site-packages/launchable/commands/test_path_writer.py", line 49, in print
    click.echo(self.separator.join(self.formatter(t)
  File "/home/brian/.local/lib/python3.10/site-packages/launchable/commands/test_path_writer.py", line 49, in <genexpr>
    click.echo(self.separator.join(self.formatter(t)
  File "/home/brian/.local/lib/python3.10/site-packages/launchable/test_runners/pytest.py", line 109, in _pytest_formatter
    return "{}::{}".format(file, case)
UnboundLocalError: local variable 'case' referenced before assignment

where tests.txt contains:

./pool/destroy.py
./pool/create_all_vm.py
./control/ms_resilience.py
./mpiio/llnl_mpi4py.py
./control/ms_failover.py
./control/dmg_server_set_logmasks.py
./pool/evict.py
./control/dmg_telemetry_basic.py
./daos_test/dfuse.py
./datamover/posix_preserve_props.py
./datamover/serial_small.py
./container/simple_create_delete.py
./network/cart_self_test.py

launchable record tests fail with "Invalid format" message

I'm testing launchable CLI in https://github.com/trinodb/trino repository, but I faced the below error.

$ launchable record build --name GITHUB_RUN_ID
$ ./mvnw install -pl :trino-test-jdbc-compatibility-old-server
$ launchable record tests --build GITHUB_RUN_ID maven ./testing/trino-test-jdbc-compatibility-old-server/target/surefire-reports/
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/launchable/commands/record/tests.py", line 288, in run
    raise Exception(exceptions)
Exception: [Exception([Exception('Failed to process a report file: ./testing/trino-test-jdbc-compatibility-old-server/target/surefire-reports/testng-results.xml', JUnitXmlError('Invalid format.'))])]

Attached the surefire reports. This is my first use, so sorry if my setup is wrong.
surefire-reports.zip

click.echo not an exception

cli/launchable/commands/record/tests.py

Line 331 in 54c740a

 raise click.echo(click.style("Unable to get subset information from subset id {}".format( 

https://sonarcloud.io/project/issues?id=launchableinc_cli&issues=AXtA0_Hv5z-0hzjkVX3T&open=AXtA0_Hv5z-0hzjkVX3T

Should this be raised, and if so, does it need to derive from Exception?

Consider changing this to sha256

cli/launchable/utils/session.py

Line 18 in 54c740a

 return _session_file_dir() / (hashlib.sha1("{}:{}".format(build_name, _get_session_id()).encode()).hexdigest() + ".txt") 

Reason: even though this is not an insecure use of this hash, the fact that we use sha1 is a thing that would be easier to not to have to explain to a prospect/customer, especially if they're running this on their premises. It may make the infosec story easier to explain if we don't have to talk about it.

TypeError when file/glob does not match any file

Running launchable record with a glob that does not match any file throws a TypeError:

$ launchable record tests --session /intake/organizations/one-two-eight/workspaces/128T_Pipeline/builds/jenkins-128T_Pipeline-11994/test_sessions/2411 googletest 'latest/gtest_*_results.xml'

Traceback (most recent call last):
  File "/usr/share/virtualenvs/ci_py36/bin/launchable", line 8, in <module>
    sys.exit(main())
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/usr/share/virtualenvs/ci_py36/lib/python3.6/site-packages/launchable/test_runners/launchable.py", line 72, in record_tests
    click.echo("No matches found: " % root, err=True)
TypeError: not all arguments converted during string formatting

Rename variable

cli/launchable/commands/record/build.py

Line 71 in 54c740a

hash = matched.group('hash')

hash is a builtin, so we're shadowing it here. Suggest renaming this variable to something else.

Traceback when running `launchable record tests maven --help`

Steps to reproduce

Run launchable record tests maven --help.

Expected results

Detailed usage information, no traceback, and an exit status of 0.

Actual results

No detailed usage information, a traceback, and an exit status of 0 (!) despite the traceback.

$ launchable record tests maven --help
Traceback (most recent call last):
  File "lib/python3.10/site-packages/launchable/commands/record/tests.py", line 200, in tests
    session_id = str(find_or_create_session(
  File "lib/python3.10/site-packages/launchable/commands/helper.py", line 54, in find_or_create_session
    raise click.UsageError(
click.exceptions.UsageError: No saved build name found.
To fix this, run `launchable record build`.
If you already ran this command on a different machine, use the --session option. See https://docs.launchableinc.com/sending-data-to-launchable/managing-complex-test-session-layouts
Usage: launchable record tests maven [OPTIONS] SOURCE_ROOTS...

Options:
  --help  Show this message and exit.
$ echo $?
0

launchableinc / cli Goto Github PK

cli's People

Contributors

Stargazers

Watchers

Forkers

cli's Issues

Context

Current Behaviour

Expected Behaviour

Possible Solution

Context

Current Behaviour

Steps to reproduce

Expected results

Actual results

Recommend Projects

Recommend Topics

Recommend Org

Jobs