mikefultondev / jclparser

JCL to script generate

Synopsis

This project enables z/OS developers to parse JCL and to then walk the parse tree. The default build creates a jcl2jcl program that can be used to just generate JCL from JCL. This is meant for testing and understanding of the code. A real use case would be to walk the parse trees to do some sort of analysis.

The altsrc and alttest and alttestsrc directories are historical and were used for testing the parser was working well. I am changing over to now just generate JCL from JCL so that I can 'diff' the output.

Code Example

Here is an example of jcl2jcl being used to parse orig.jcl and print it out.

jcl2jcl --input=testsrc/commented.jcl

The code works on z/OS or off. It has been tested on z/OS and on MacOS using clang.

Motivation

I am not a big fan of JCL, even though I have worked on z/OS forever. I also try to use Unix System Services as much as possible to develop software. jcl2jcl generates JCL from JCL - which can be handy if you want to 'normalize' or 'clean up' JCL.

Installation

To install jcl2jcl:

• copy the files to z/OS Unix System Services directory. For this example, we assume it is /u/ibmuser/JCLParser
• cd to the directory (/u/ibmuser/JCLParser)
• install CMake, Ninja, GNU Make, clang if you haven't already and use the provided CMake code.
• Alternately, just compile all the C code in src and link it.
• The resultant program should be called jcl2jcl to use the sample parser.

Building on z/OS

• Use z/OS Open Tools to download cmake, ninja, gnu make, and install clang.
• export CLANG_ROOT to the root directory where you installed the clang compiler.
• Generate build files if first build:

cd JCLParser
mkdir build
cd build
export CC=${CLANG_ROOT}/usr/lpp/IBM/oelcpp/v2r0/bin/clang
export CXX=${CLANG_ROOT}/usr/lpp/IBM/oelcpp/v2r0/bin/clang++ 
cmake ../

• Re-build code:

cmake --build .

This will create a jcl2jcl binary in your build directory

Building on MacOS

• Tailor the files in the .vscode directory to suit your needs.
• cd /u/ibmuser/JCLParser
• mkdir build
• cd build
• cmake ..
• cmake --build .

API Reference

To get started reading the code, begin in jcl2jcl.c, which has main and drives all the functions in the other files.

Tests

Basic regression testing can be done after building jcl2jcl with:

• cd tests
• ./regression.sh

There should be no output. As bugs fixed, files with a filetype of skipjcl should be renamed to jcl to get included in regression testing.

Work to be done

The parser is complete as far as I know. Bug reports welcome. Providing interesting tools with the parser is TBD.

Contributors

Mike Fulton (IBM Canada) is the sole contributor at this point. I am happy to change this :)

The code still needs work. Error messages can (always) be improved and there are missing features people may need. If anyone wants to contribute, please reach out to [email protected] (Mike Fulton)

License

The code uses the Apache license

jclparser's Issues

the parms in jcl2sh

Mike, good day, a question, the sample in the README.MD:
jcl2sh <//'IBMUSER.TEST.JCL(UPDATE) >./update.sh
It don't send any message and neither create the update.sh ...
could you comment some action to verify the shell?

INCLUDE statement loses information in display

See https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/include.skipjcl for an example

Print out a multi-line string properly

See: multiLine.jcl which has a multi-line string on it.

'Then' part of conditional not being written out in parsed output

Here is a diff:

< //DRVRCHKF IF (DRVRCHK.RC ^=0) THEN SEND FAILURE NOTE
---
> //DRVRCHKF IF (DRVRCHK.RC ^=0) THEN
52c52
< //COPY1OK  IF (DRVRCHK.RC = 0) THEN COPY FIRST PACK
---
> //COPY1OK IF (DRVRCHK.RC = 0) THEN
62c62
< //COPY1F   IF (COPY1.RC ^= 0) THEN SEND FAILURE NOTE
---
> //COPY1F IF (COPY1.RC ^= 0) THEN
120c120
< //CTIMEOUT IF (RESUBEX.RC ^=0) THEN SEND FINAL FAILURE NOTE
---
> //CTIMEOUT IF (RESUBEX.RC ^=0) THEN

Need to write the generation part of JCL2SH

right now, jcl2sh only has the scanner written to scan the JCL and create an internal representation. Generators need to be written for the output side.

Comment at end of conditionals is not retained

205c205
< // ENDIF /*ENDIF FOR THE IF STARTED AT STEPN007*/
---
> //  ENDIF

and

51c51
< //DOIT     ELSE    /*DG 04092012 FOR REQ C243
---
> //DOIT ELSE
56c56
< //DOIT   ENDIF     /*DG 04092012 FOR REQ C243

are two examples

Sequence numbers are not retained in parsed output

simple fix is to cut everything to the right of column 71, but 'information' is lost.

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/longcard.skipjcl
and https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/multilinejob.skipjcl for examples

Comment line in middle of parameters not treated as comment

Run jcl2jcl on https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/parmcomments.skipjcl with -v and you will see:

000000001 //IBMUSERC JOB 'CRTHFS'     Job Card                                            |
-> JOB Statement
000000002 //STEP1  EXEC PGM=IEFBR14,                                                      |
-> EXEC Statement
--> continued parameter
000000003 //* Comment                                                                     |
000000004 //       PARM='Hello World'                                                     |
//IBMUSERC JOB 'CRTHFS'  Job Card
//STEP1 EXEC PGM=IEFBR14,
Comment
// PARM='Hello World'

The comment is not being properly printed. Note that for this case, the comment is in the tree - here's a screen shot:

but note the newline at the start... So the 'printer' in genJCL could likely deal with this (should validate multi-line comments too).

Also: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/odd.skipjcl and https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/commented.skipjcl

Problems in the conversion process for certain characters

From Arturo:
I found some mistakes with the characters conversion,... for example the compiler is not accepting “!” , “[“, “]”

Multi-line conditional is reduced to one line in display

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/conditional.skipjcl for an example

JECL commands such as 'C' for cancel are not printed properly

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/cancel.skipjcl for an example

Redundant line with /* at the start is not retained by parser

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/extraslashstar.skipjcl

JCL to script generate

Hi @mikefultonbluemix ,

Please advice to me how to fix and run your source code?

Some info for your reference as below:
step 1: clone source and save my server.
step 2: create 2 folder

JCL2SH_SOURCE: save source code here
JCL2SH: empty
step 2:
cd to the directory (/u/ibmuser/JCL2SH) = /home/JCL2SH_SOURCE
edit setenv.sh to point to the various MVS programs that will be tested, and to specify where your code was copied to
=>export JCL2SH_ROOT=/home/JCL2SH
step 3:
create folder bin into folder /home/JCL2SH_SOURCE
step 4:
sh build.sh
please see result as below error:
[root@cock-01 JCL2SH]# sh build.sh
Build Default Version
gcc: error: unrecognized command line option ‘-Wc,FLAG(I)’
gcc: error: unrecognized command line option ‘-Wc,xplink,gonum,offset,langlvl(extended),list(./)’
gcc: error: unrecognized command line option ‘-Wc,FLAG(I)’
gcc: error: unrecognized command line option ‘-Wc,xplink,gonum,offset,langlvl(extended),list(./)’
gcc: error: unrecognized command line option ‘-Wc,FLAG(I)’
gcc: error: unrecognized command line option ‘-Wc,xplink,gonum,offset,langlvl(extended),list(./)’
gcc: error: unrecognized command line option ‘-Wc,FLAG(I)’
gcc: error: unrecognized command line option ‘-Wc,xplink,gonum,offset,langlvl(extended),list(./)’
gcc: error: unrecognized command line option ‘-Wc,FLAG(I)’
gcc: error: unrecognized command line option ‘-Wc,xplink,gonum,offset,langlvl(extended),list(./)’
gcc: error: jcl2sh.o: No such file or directory
gcc: error: scanjcl.o: No such file or directory
gcc: error: gensh.o: No such file or directory
gcc: error: jclargs.o: No such file or directory
gcc: error: jclmsgs.o: No such file or directory

Please advice to how to run your source and document guilde step by step. Thanks for the support.

Thanks and best regards,
Nho Luong.

@nluong9

Trailing text after a delimiter is not retained in display

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/inlinedd.skipjcl for an example, specifically:

zzStuff Here Ignored

is displayed as:

zz

PARM statement is printed on one line instead of 2

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/subparm.skipjcl

Control statements are not properly displayed

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/cntl.skipjcl for an example

sintax error on line 707 of scanjcl.c program

The compilation is generating a CCN3046 "sintax error" in line 707 of scanjcl.c program->
KeyValuePair_T* kvp = cur->kvphead;

to eliminate this error, I fixed it adding an->"else {", in line 606 and other->"}" just before the line 743
and then the compilation was ok! and it generates the module jcl2sh in the bin directory ,... but
The problem is at execution time of the jcl2sh to generate the shell, it does not do anything!...

jcl2sh <//'IBMUSER.TEST.JCL(UPDATE) >./update.sh

do you know, what is happening? or an idea or suggestion to fix this situation?
Is it correct the sintax of the execution jcl2sh? is it missing something else?
Thanks a lot for your help!

parser does not handle a key 'name' being a string

Not sure what other examples there are, but the jobcard in: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/longcard.jcl

has:

//IXCLARM JOB 'D7103P,D71,?,S=I' 

which confuses the parser because it isn't expecting a string for the 'name', e.g. it is normally something like:

//STEP EXEC PGM=FRED,PARM='<string>'

so the key value is a string (but in the first case the key name is a string).

segmentation fault for inlineGeneratedSYSIN.jcl sample

% ../bin/jcl2sh --input=./inlineGeneratedSYSIN.jcl --output=./inlineGeneratedSYSIN.jcl.sh
zsh: segmentation fault ../bin/jcl2sh --input=./inlineGeneratedSYSIN.jcl

Provide ability to suppress generated SYSIN card in output

For the following test: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/gensysin.skipjcl and
https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/inlineGeneratedSYSIN.skipjcl

the output has an extra line - it explicitly shows the otherwise implicit SYSIN DD card:

//IBMUSER JOB ,MSGLEVEL=(1,1)
//STEP EXEC PGM=IEFBR14
//SYSIN  DD *  (Generated)
Line A
Line B

Print out a multi-line comment properly

See: https://github.com/MikeFultonDev/JCLParser/blob/main/testsrc/multiLine.skipjcl which has a multi-line comment at the end of the JCL