zowe / jobs Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 5.0 1.33 MB

Repo for the jobs api controller and code

License: Eclipse Public License 2.0

Java 92.05% Shell 7.11% Batchfile 0.25% Dockerfile 0.59%

jobs's People

Contributors

Stargazers

Watchers

Forkers

raymondyanli vvvlc mehmetalpsumer shiv-patel-08

jobs's Issues

As a user of the Zowe, I would like Jobs API service to use the Zowe SSO solution for login.

See zowe/data-sets#18

Acceptance Criteria

Scenario 1: username API SSO
Given I'm a valid zowe user with username and have authenticated to the gateway (eg the catalog)
When I do GET /api/v1/jobs/username
Then I receive the response:

{
  "username": "STEVENH"
}

Scenario 2: Jobs GET API SSO
Given I'm a valid zowe user and have authenticated to the gateway (eg the catalog)
When I do a GET on /api/v1/jobs/
Then I'm not prompted for my logon and get the correct response for my user

Implementation details:

Atlas can use /login on API Gateway to get the JWT token (stored in HttpOnly cookie)
The REST endpoints are specified in https://github.com/gizafoundation/api-layer/wiki/Zowe-Authentication-and-Authorization-Service (will be migrated to Zowe GitHub)
Atlas gets the JWT token in a HttpOnly cookie in a header from UI in browser
Atlas is using /query on API Gateway to check the validity of the token
Atlas sends this token to other services in HTTP Header (Authentication Bearer)
Atlas server accesses the z/OSMF via the API Gateway so the gateway can use the LTPA token for z/OSMF

Questions: Do we want to support standalone and zowe SSO?

Remove of SHA-1 cyphers

As a Zowe contributor I need SHA-1 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA ciphers to be removed from the default (pre-configured) cipher suites so the build is green again.

Additionally, documentation should be updated to reflect that certain levels of Java will not work with the default and anyone on those levels would need to take some action (update their JDK, change their JDK configuration to use GCM ciphers, add back SHA-1 ciphers for compatibility at their own risk...).

Provide a GET job files which concatenates all files

From zowe/explorer-server#24
An API endpoint which would return all job output files in a concatenated form has been requested by customers. The implementation of this would require an update in the JES viewer to exploit such a feature

401 Authentication Error

With the latest Zowe build, the Jobs API can be started but on one of testing system, the API always failed to authenticate. It will keep prompting username and password.

Here is the error stack logged:

 [explorer-jes][Tue, 08 Jan 2019 18:30:07 GMT] /ui/v1/explorer-jes/ 200 /u/zowesvr/jes_explorer/app/index.html
 [explorer-jes][Tue, 08 Jan 2019 18:30:07 GMT] /ui/v1/explorer-jes/css/responsive-grid.css 200 /u/zowesvr/jes_explorer/app/css/responsive-grid.css
 [explorer-jes][Tue, 08 Jan 2019 18:30:08 GMT] /ui/v1/explorer-jes/css/styles.css 200 /u/zowesvr/jes_explorer/app/css/styles.css
 [explorer-jes][Tue, 08 Jan 2019 18:30:08 GMT] /ui/v1/explorer-jes/css/react-contextmenu.css 200 /u/zowesvr/jes_explorer/app/css/react-contextmenu.css
 [explorer-jes][Tue, 08 Jan 2019 18:30:08 GMT] /ui/v1/explorer-jes/app.min.js 200 /u/zowesvr/jes_explorer/app/app.min.js
 �[2m2019-01-08 18:30:31.669�[0;39m �[31mERROR�[0;39m �[35m33554528�[0;39m �[2m---�[0;39m �[2m[nio-8545-exec-7]�[0;39m �[36mo.z.a.c.s.CustomAuthenticationProvider  �[0;39m �[2m:�[0;39m authenticate
 
 java.lang.NullPointerException: null
 	at org.zowe.api.common.security.CustomAuthenticationProvider.authenticate(CustomAuthenticationProvider.java:48) ~[explorer-api-common-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
 	at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174) [spring-security-core-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199) [spring-security-core-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]
 	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:155) [spring-boot-actuator-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
 	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:123) [spring-boot-actuator-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
 	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108) [spring-boot-actuator-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) [na:1.8.0]
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [na:1.8.0]
 	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.32.jar!/:8.5.32]
 	at java.lang.Thread.run(Thread.java:812) [na:2.9 (09-15-2018)]

@stevenhorsman @NayerNajafi

Support Single Sign on SSO jobs-api

The jobs API server should be compatible with the SSO solution provided by the api layer

Improve error messaging from zosmf

We have lots of places with errors from zosmf that might not be ideal. Do you want to sort out a error messaging strategy to include codes, so we can do globalisation and translate them on the UI?

MVS File Search API looking through files for strings

From: zowe/explorer-server#8
Allow the ability to search through fiels in a data set (e.g. find strings).

New API - submit JCL and wait till completion

Requirement is to have the ability to submit a job and wait till its completion.
Issues to consider:
How long is the wait for - can a timeout value be passed in and what would the default be ?
What if the job goes onto the input queue ?
What is the response to the API ? Is a particular job file content to be returned and is this an input parm to specifiy which one, or should the list of all of the files be returned ?

Fix code coverage instrumentation for powermock code

I believe all the zowe java code uses powermock for testing.
I'm guessing the that the pipeline's code coverage is based on jacoco live instrumentation at the moment. This doesn't work with powermock: https://github.com/powermock/powermock/wiki/Code-coverage-with-JaCoCo

We could look at the offline instrumentation, or try switching to another tool (I have used cobertura with powermockito in the past successfully). Otherwise we won't get proper coverage stats: eg https://wash.zowe.org:8443/job/Explorer-Jobs/job/master/8/Java_20Coverage_20Report/

Migrate Atlas explorer-server liberty based jobs apis over to springboot

Basic apis, maybe resulting in minor function loss where we don't know of any users.
Re-write, no enhancements - see other issues for these

Sort out response caching in Zosmf*Service

The error messages are currently not great - because we can't read an HTTP response object twice (as it's a stream) we end up returning objects, rather than nice messages. We could cache these responses so we can read them in a number of ways.

Archive Jobs

Is your feature request related to a problem? Please describe.
The jobs aren't used anymore and the functionality was superseded by zOSMF

Describe the solution you'd like
Properly archive jobs

Embed OWASP dependency scan into pipeline

As sub-issue of zowe/zowe-install-packaging#406:

The changes should allow the pipeline:

to run OWASP dependency scan per commit on every branch,
to run OWASP dependency scan on daily basis if the repository is not actively developing.

Improve integration test

Rewrite the poor regex and file based integration tests with the jsoncompare to rest assured based.
Then update IntegrationTestResponse ones to rest assured as well

Manually test the jobs-api-server ‘install’ on z/OS using the apiml ssl certificates

At install time the tomcat Jobs server should use the api mediation layer ssl certificates.
This needs testing

Add new API to Return Compressed ZIP of all jobs spool file

We are looking to add this feature to explorer, where user right click on job and get all spool file downloaded as ZIP, it may provide convenience to user for filing support issue.

Zowe Jobs service not passing custom headers

Sending request to /jobs/dataset doesn't pass custom headers forward. Easily tested by X-IBM-Symbol-name. (https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.izua700/IZUHPINFO_API_PutSubmitJob.htm).

I believe problem is in https://github.com/zowe/jobs/blob/master/jobs-api-server/src/main/java/org/zowe/jobs/services/zosmf/SubmitJobFileZosmfRequestRunner.java

protected RequestBuilder prepareQuery(ZosmfConnector zosmfConnector) throws URISyntaxException {
        String urlPath = String.format("restjobs/jobs"); //$NON-NLS-1$
        URI requestUrl = zosmfConnector.getFullUrl(urlPath);
        JsonObject body = new JsonObject();
        body.addProperty("file", "//'" + fileName + "'");

        StringEntity requestEntity = new StringEntity(body.toString(), ContentType.APPLICATION_JSON);
        RequestBuilder requestBuilder = RequestBuilder.put(requestUrl).setEntity(requestEntity);
        return requestBuilder;
    }

The code cares only about body of the request (adding // so z/OSMF Jobs API gets correct syntax) and content type.

Add ability to cancel a job without purging output

We currently only provide an endpoint to cancel and purge a jobs output, we should introduce an endpoint that only cancels the job.

performance of GET concatenated output files of a job is restricted by sequential fetch of job files

Currently the GET concatenated output files of a job gets each file from z/OSMF sequentially, if there are a number of large files there could be great benefits to sending these requests to z/OSMF in parallel

Original issue for adding the endpoint: #31

Support synchronous purge

With Zowe build 414 (also see similar issue on v0.9.5):
Purge Job fails with CIM Connection Failure. In the system, CIM is not active and running. I tried also
curl -X DELETE -k -u "ibmuser:tivmvs" -i "https://tvt7050.svl.ibm.com/zosmf/restjobs/jobs/BPXAS/STC07161"
I got the same error. But when I use HTTP Header "X-IBM-Job-Modify-Version: 2.0", the z/osmf REST API for Purge Job works correctly:
curl -X DELETE -k -H "X-IBM-Job-Modify-Version: 2.0" -u "ibmuser:tivmvs" -i "https://tvt7050.svl.ibm.com/zosmf/restjobs/jobs/BPXAS/STC07161"
Note: Header 2.0 processes the request synchronously, and synchronous processing is supported for JES2 only.

I think we need to include header "X-IBM-Job-Modify-Version: 2.0" in request headers.

Review logging/serviceability of jobs app

Add more logging in - check output
Make sure FFDC is done for all errors.

Create build for new standalone springboot jobs app

Update https://wash.zowe.org:8443/job/Explorer-Jobs
it’s not really doing much at the moment, so needs test reporting, publishing pushing to artifactory and then a subsequent step/pipeline to deploy to docker and run the integration tests.

Expand Purge/Cancel Job to select multiple Job Ids

We are planning to enhance JES Explorer , where user can select multiple jobs and purge all of them together.

This enhancement can help providing above functionality on UI side.

Update Job models

Update the job model to match the data set apis
#depends on #11

Add validation to jobs apis

Add in validation framework for requests. Eg checking JCL, dataset etc

Handle content-encoding vs etag issue

The spring dependencies are outdated and should be upgraded. However, in Spring 2.1.12 an updated tomcat version is used which fixes how the HTTP spec is enforced. When strong etags are included in the response, the response will not be compressed. This is because the strong etag is a hash of the content, and compressing the content would then change the hash. Strong etags are required for the If-Match header functionality currently in use.

Gzip content-encoding and strong e-tags (therefore If-Match) are mutually exclusive. They should both be independently supported, however, where the calling client can specify whether they want gzip or strong etag in the response based on a request header.

Add zowe APIML discovery enablers

Make the jobs apis dynamic, rather than static api definition

Submit job not working for dataset member

POST /api/v1/jobs/dataset

For example body of {"file": "'HLQ.TEST.JCL(CMPBATCH)'"} always returns:
{
"status": "BAD_REQUEST",
"message": "Submit input data does not start with a slash"
}

zowe jobs submit REST API error

zowe files list ds "h*" --responseTimeout 600 is OK, so credentials are OK
zowe jobs submit ds "z83677.JCL(CBL0001)" --wfo is KO :

`Command Error:

z/OSMF REST API Error:

Rest API failure with HTTP(S) status 500

rc: 8

reason: 5

category: 6

message: Error opening input data set: //z83677.JCL(CBL0001)`

`Error Details:

HTTP(S) error status "500" received.

Review request details (resource, base path, credentials, payload) and ensure correctness.

Host: 192.86.32.250

Port: 10443

Base Path:

Resource: /zosmf/restjobs/jobs

Request: PUT

Headers: [{"Content-Type":"application/json"},{"X-CSRF-ZOSMF-HEADER":true}]

Payload: { file: "//'z83677.JCL(CBL0001)'" }`

So what can I do ?

OUTPUT filter not working properly

@stevenhorsman commented on Thu Oct 03 2019

On Plex2, when I request all output jobs:
https://winmvs28.hursley.ibm.com:8545/api/v1/jobs?prefix=*&owner=*&status=OUTPUT

There are no jobs, but if I add a prefix there are some, so the filtering is not working properly

Error icon/button in swagger UI from failed validation request when inside private network

Describe the bug
When accessing the Swagger UI a red error button shows in the bottom right corner, upon clicking this button the user is taken to a page detailing a failed file load error when trying to validate the API schema.
This does not affect API behaviour it and is not reproducible when running locally.

To Reproduce
Steps to reproduce the behavior:

Go to swagger UI
Click on red error button
See error

Expected behavior
This message should not appear

Desktop (please complete the following information):
All browsers

Additional context
The work around for this is currently documented here: springfox/springfox#2201

File to be changed is: zowe/data-sets/data-sets-api-server/src/main/java/org/zowe/spring/SwaggerConfig.java

I suggest adding below code to configure the request url to an empty string, i.e don't check

@Bean
    UiConfiguration uiConfig() {
        return UiConfigurationBuilder.builder()
            .validatorUrl("")
            .build();
    }

zowe / jobs Goto Github PK

jobs's People

Contributors

Stargazers

Watchers

Forkers

jobs's Issues

Acceptance Criteria

Implementation details:

Recommend Projects

Recommend Topics

Recommend Org

Jobs