Comments (4)
Doesn't look like run_e2e_workflow.py has changed in a while. But I do think we upgraded Argo recently.
from testing.
My conjecture is that wait_for_workflows raises an exception (after calling log_status).
This doesn't show up in the log uploaded to prow because it we don't catch it and in the finally blog we upload the current log to prow. So the exception will get propogated and cause the process to exit but it won't be captured in the log uploaded to prow.
try:
results = argo_client.wait_for_workflows(api_client, NAMESPACE,
workflow_names,
status_callback=argo_client.log_status)
for r in results:
phase = r.get("status", {}).get("phase")
name = r.get("metadata", {}).get("name")
workflow_phase[name] = phase
if phase != "Succeeded":
success = False
logging.info("Workflow %s/%s finished phase: %s", NAMESPACE, name, phase)
except util.TimeoutError:
success = False
logging.error("Time out waiting for Workflows %s to finish", ",".join(workflow_names))
finally:
success = prow_artifacts.finalize_prow_job(args.bucket, success, workflow_phase, ui_urls)
# Upload logs to GCS. No logs after this point will appear in the
# file in gcs
file_handler.flush()
util.upload_file_to_gcs(
file_handler.baseFilename,
os.path.join(prow_artifacts.get_gcs_dir(args.bucket), "build-log.txt"))
from testing.
Yup that's the problem
If we get the logs for the pod run by prow see here
INFO|2018-05-10T18:32:34|/src/kubeflow/testing/py/kubeflow/testing/util.py|522| Uploading file /tmp/tmpRunE2eWorkflowbjXGWwlog to gs://kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/787/kubeflow-presubmit/1462/build-log.txt.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 281, in <module>
final_result = main()
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 271, in main
return run(args, file_handler)
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 168, in run
status_callback=argo_client.log_status)
File "/src/kubeflow/testing/py/kubeflow/testing/argo_client.py", line 51, in wait_for_workflows
GROUP, VERSION, namespace, PLURAL, n)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/custom_objects_api.py", line 697, in get_namespaced_custom_object
(data) = self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/custom_objects_api.py", line 797, in get_namespaced_custom_object_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 342, in request
headers=headers)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 231, in GET
query_params=query_params)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 10 May 2018 18:32:33 GMT', 'Audit-Id': 'd6977031-5f33-4bfa-8548-c5f3c1276ca2', 'Content-Length': '332', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"workflows.argoproj.io \"kubeflow-presubmit-kubeflow-gke-deploy-787-f893ce6-1462-040e\" not found","reason":"NotFound","details":{"name":"kubeflow-presubmit-kubeflow-gke-deploy-787-f893ce6-1462-040e","group":"argoproj.io","kind":"workflows"},"code":404}
INFO|2018-05-10T18:32:34|/src/kubeflow/testing/py/kubeflow/testing/util.py|522| Uploading file /tmp/tmpRunE2eWorkflowbjXGWwlog to gs://kubernetes-jenkins/pr-logs/pull/kubeflow_kubeflow/787/kubeflow-presubmit/1462/build-log.txt.
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 281, in <module>
final_result = main()
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 271, in main
return run(args, file_handler)
File "/src/kubeflow/testing/py/kubeflow/testing/run_e2e_workflow.py", line 168, in run
status_callback=argo_client.log_status)
File "/src/kubeflow/testing/py/kubeflow/testing/argo_client.py", line 51, in wait_for_workflows
GROUP, VERSION, namespace, PLURAL, n)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/custom_objects_api.py", line 697, in get_namespaced_custom_object
(data) = self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/custom_objects_api.py", line 797, in get_namespaced_custom_object_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 342, in request
headers=headers)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 231, in GET
query_params=query_params)
File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 10 May 2018 18:32:33 GMT', 'Audit-Id': 'd6977031-5f33-4bfa-8548-c5f3c1276ca2', 'Content-Length': '332', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"workflows.argoproj.io \"kubeflow-presubmit-kubeflow-gke-deploy-787-f893ce6-1462-040e\" not found","reason":"NotFound","details":{"name":"kubeflow-presubmit-kubeflow-gke-deploy-787-f893ce6-1462-040e","group":"argoproj.io","kind":"workflows"},"code":404}
from testing.
/close
from testing.
Related Issues (20)
- Alternative solution to removal of test on optional-test-infra HOT 31
- Deprecate ECR repo provided by optional-test-infra HOT 17
- Image Scanning for CVs HOT 8
- Image Scanning HOT 2
- IAM as Code HOT 5
- [GCP] Migrate machine type to e2 family to save costs HOT 8
- [AWS] Configure dependabot for new-built image PR
- [AWS] Optional-Test-Infra Migration HOT 2
- Go license tools no longer returning licenses for k8s libraries like apimachinery, controller-runtime, etc HOT 4
- The Optional-test infra should run presubmit jobs for kubeflow/kubeflow
- [AWS] Infrastructure as Code HOT 3
- Improve unit tests for kubeflow/testing repo codebase
- Let optional test infra manage kubeflow/testing presubmit/postsubmit HOT 6
- Migrate to CDK-deployed AWS Resources HOT 2
- rebuild test-worker image HOT 4
- Postsubmit link formatting error
- tekton cluster has been deleted in AWS Optional Test Infrastructure? HOT 3
- eksctl latest release will break cluster setup HOT 3
- Optional Test Infra Deprecation Notice HOT 11
- Support AWS EKS cluster version 1.22 in CI HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from testing.