openmainframeproject / tersedecompress

TerseDecompress is a java program that can be used to decompress files which have been previously compressed on an IBM Mainframe using the TERSE / AMATERSE program (on IBM z/OS or IBM z/VM).

tersedecompress

TerseDecompress is a java program that can be used to decompress files which have been previously compressed on an IBM Mainframe using the TERSE / AMATERSE program (on IBM z/OS or IBM z/VM).

Purpose & benefit

As java programs can virtually run on any platform / operating system with a JVM, this java decompression program can be very useful if you don't have access to an IBM mainframe but need to analyze or process a file which had been compressed on a IBM mainframe using TERSE on IBM z/OS. With this java version of TerseDecompress, you can decompress those tersed IBM Mainframe files on any workstation or laptop etc. that supports Java.

Updates

Version 5: March 2021

• Support for variable length binary records. Variable length records processed in binary mode will be prefixed with a 4 byte field in the same format as the IBM RDW i.e. 2 byte record length field (including RDW length, big-endian) followed by 2 bytes of zeros.

How to run it

For execution, TerseDecompress needs a JVM runtime environment.

Usage:

java -jar tersedecompress-5.0.0.jar [-b] tersed-file output-file

Default mode is text mode, which will attempt EBCDIC -> ASCII conversion.

The -b flag turns on binary mode, no conversion will be attempted.

Recommendation: Use binary mode if possible. EBCDIC->ASCII conversion is not a lossless process, unless the data strictly contains only characters that are common to both code pages used for the translation, and the original data does not contain line ending characters.

How to build it

To build tersedecompress you need the Java JDK and Apache Maven.

In the project directory (the directory containing pom.xml):

mvn clean package

Unit Tests

The project contains unit tests to verify that the decompression is correct.

Due to the size of the test data, it is stored in a separate git repository and referenced via a submodule. The test data is not required to build tersedecompress, unless you want to run the unit tests.

The test data can be found here: https://github.com/openmainframeproject/tersedecompress-testdata

Descriptions of the data are here: https://github.com/openmainframeproject/tersedecompress-testdata/tree/master/tests

Building with Unit Tests

1. Initialize (download) the submodule containing the test data: git submodule update --init
2. Build with unit tests: mvn -DskipTests=false clean package

How to report problems and get support/help

If you have a problem / need help with TerseDecompress.java, please create a GitHub issue.

There is also a Slack channel: https://slack.openmainframeproject.org channel #tersedecompress

tersedecompress's Issues

Can Tersedcompress output the data set type of the original dataset?

Hi,
Tersed compress is great, I was wondering if it'd be possible to display the orginal data set type ?
i.e. was it a PS / PDS or PDSE ?

Thanks

add please at least one release

hello, you great product. Please add ready to use zip file. I have wrote small bat to make simple unterse faster.

@echo OFF
java -jar tersedecompress-5.0.0.jar %1 %1.txt

Exception has occurred: java.lang.ArrayIndexOutOfBoundsException

Exception has occurred: java.lang.ArrayIndexOutOfBoundsException
"java.lang.ArrayIndexOutOfBoundsException: Index -2 out of bounds for length 256"

for an input file we are getting uncaught exception.

Terse header version not recognized : 81

Hi,
tried to unterse a File for Windows from the IBM download page. Trows an error.
C:\Temp\IBM-Terse>java -jar tersedecompress-5.0.0.jar Updatemanager-WinX64.exp Updatemanager-WinX64.exe
Exception in thread "main" java.io.IOException: Terse header version not recognized : 81
at org.openmainframeproject.tersedecompress.TerseHeader.CheckHeader(TerseHeader.java:119)
at org.openmainframeproject.tersedecompress.TerseDecompresser.create(TerseDecompresser.java:32)
at org.openmainframeproject.tersedecompress.TerseDecompress.process(TerseDecompress.java:100)
at org.openmainframeproject.tersedecompress.TerseDecompress.main(TerseDecompress.java:113)
What is the problem here?
BR, Nick

Can you please add support to decompress to ZipOutputStream ?

It would be great if we had option to output a compressed zip file instead of a huge SMF file.
Then people that process the data using Java can read using ZipInputStream

tool crashes with large input files in binary mode

The following error occurs when large input files (>2GB) are to be decompressed in binary mode:

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:135)
	at org.openmainframeproject.tersedecompress.TerseDecompresser.PutChar(TerseDecompresser.java:108)
	at org.openmainframeproject.tersedecompress.NonSpackDecompresser.decode(NonSpackDecompresser.java:81)
	at org.openmainframeproject.tersedecompress.TerseDecompress.process(TerseDecompress.java:104)
	at org.openmainframeproject.tersedecompress.TerseDecompress.main(TerseDecompress.java:113)

The problem can be fixed by limiting the maximum record size in PutChar(TerseDecompresser.java:108):

record.write(X-1);  
final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 10;  
if (record.size() >= MAX_ARRAY_SIZE) {  
   endRecord();  
}