adelosa / mciutil

Collection of MasterCard batch file utilities. Work with MasterCard IPM and MPE file formats.

MasterCard file utilities

Set of command line utilities to work with various MasterCard files.

• Free software: BSD license
• Documentation: https://mciutil.readthedocs.org.

Warning

THIS PACKAGE HAS BEEN DEPRECATED AND WILL NOT BE UPDATED GOING FORWARD

This package was created when I first started learning python. I have learned a lot over the last 4 years and I now see the error in my ways.

Some of the issues with this module that prompted me to rewrite it:

• memory efficiency - loads entire file into memory for processing. Very ineffient and not very scalable
• programming interface - mciutil did not consider the developer experience. You have to hack to use the logic elsewhere
• dependencies - Too many third party modules, with ones that required a compilation. New version is compile free
• bloat - I used a cookie cutter template when I started and it has stuff I don't like, value or use.
• just mastercard - The old module was for mastercard only but I think it makes sense to have a library for all card utils
• python 2 guff - mciutil works on py2 and 3. There is a lot of gunk in the code to make this work. We live in a py3 world now!

The replacement module is cardutil - see https://cardutil.readthedocs.io It addresses all of the above issues.

why not just update mciutil?

Thats a good question. I think because the new codebase as developed from scratch rather than via changes to the existing one (there is some borrowed code from mciutil). If I just released a new version, anyone leaning on the internal API's would definetly be in trouble as they are not the same.

Features

Provides the following command line utilities:

• paramconv: Utility for working with MasterCard MPE parameter extract files
• mideu: Utility for working with MasterCard IPM files

mciutil's Issues

Bit 11 issue while creating the MC IPM file

I am facing no bit 11 while converting the IPM file from ASCII to EBCDIC

No config found for bit 11 Traceback (most recent call last): File "/usr/local/bin/mideu", line 9, in <module> load_entry_point('mciutil==0.4.1', 'console_scripts', 'mideu')() File "/usr/local/lib/python2.7/site-packages/mciutil-0.4.1-py2.7.egg/mciutil/cli/mideu.py", line 26, in cli_entry _main(args) File "/usr/local/lib/python2.7/site-packages/mciutil-0.4.1-py2.7.egg/mciutil/cli/mideu.py", line 48, in _main args.func(args) File "/usr/local/lib/python2.7/site-packages/mciutil-0.4.1-py2.7.egg/mciutil/cli/convert.py", line 46, in convert_command ) for record in input_data File "/usr/local/lib/python2.7/site-packages/mciutil-0.4.1-py2.7.egg/mciutil/mciutil.py", line 171, in flip_message_encoding raise Exception("Config missing for bit {}".format(bit)) Exception: Config missing for bit 11

I know there is no bit 11 in the IPM file. I have added the mandatory bit only.

SAMPLE ASCII IPM Content

164410000000000000000000000100000000000000000000000100000000000000000000001000000000000000000000000000000000000000000000000000000000697080010502500216072500000012123087601100250021607250000001212308760122001P01910012000000011240111100000001000000000101010000101000000000100001100000000000000000000010000000000000000000000100000000000000000000000000000000001552100000011111110020000000000200016020110593561090101900120014002305210006207000001600826521000620700000160082110000001212340MUMBAI EMV ATM - 2 MUMBAI AE030014800478420165004001M019100127840000000211000000121231644100000000000000000000001000000000000000000000001000000000000000000000010000000000000000000000000000000000000000000000000000000006950780105025002160700000012123000121230876v0122001P0191001200000001019100120301016000000000000030003060080000000300000032

I am not able figure out where I am doing wrong.
Can you please help me?

Unit tests broken - wrong fields in csv output

Build working but broken on local machine. Correct to build correctly

• test_with_ascii_file_only
• test_with_ebcdic_file_only

Check if expected x'40' characters at end of 1014 blocks

When unblocking a file, check the blocking characters to make sure they are x'40' otherwise throw a log warning. This is an indication that the file is not blocked correctly and will likely result in an error processing the file. Don't want to throw exception until I have some view as to what characters could actually be there.
Suggest that the user try using the flag to disable block file processing - see #26

Can't Convert Some IPM Files to CSV

I use mideu to extract IPM files to CSV, about 75% of the files are extracted successfully while about 25% are not, I get the error message below for those that do not extract

C:\Data\pyenv\Scripts>mideu extract TT112T0.2020-05-29-01-83-63.001
c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\extract.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(config_file)
Traceback (most recent call last):
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Data\pyenv\Scripts\mideu.exe_main.py", line 9, in
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\mideu.py", line 27, in cli_entry
_main(args)
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\mideu.py", line 53, in _main
args.func(args)
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\extract.py", line 48, in extract_command
output_list = [
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\extract.py", line 49, in
get_message_elements(
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\mciutil.py", line 326, in get_message_elements
_process_element(bit,
File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\mciutil.py", line 369, in _process_element
field_length = int(field_length_string)
ValueError: invalid literal for int() with base 10: b'\x90\x94\x90'

How can I resolve the above error?

mideu extract giving error

OS Ubuntu 14.04LTS
TT068T0.2015-10-16-10-42-18.001 is data file MasterCard IPM
converted the above file to EBCDIC ,assuming my file is Ascii
mideu extract TT068T0.2015-10-16-10-42-18.001.out

"Traceback (most recent call last):
File "/usr/local/bin/mideu", line 9, in
load_entry_point('mciutil==0.4.1', 'console_scripts', 'mideu')()
File "/usr/local/lib/python2.7/dist-packages/mciutil-0.4.1-py2.7.egg/mciutil/cli/mideu.py", line 26, in cli_entry
_main(args)
File "/usr/local/lib/python2.7/dist-packages/mciutil-0.4.1-py2.7.egg/mciutil/cli/mideu.py", line 48, in _main
args.func(args)
File "/usr/local/lib/python2.7/dist-packages/mciutil-0.4.1-py2.7.egg/mciutil/cli/extract.py", line 52, in extract_command
) for record in input_file
File "/usr/local/lib/python2.7/dist-packages/mciutil-0.4.1-py2.7.egg/mciutil/mciutil.py", line 275, in get_message_elements
bit_config[bit],
KeyError: 8"

Remove mongo support

Original version of the tool had mongo support but has never been fleshed out.
In retrospect, a seperate tool would be more suitable to load the csv file of transactions into the data tool of choice.
Remove all logic related to mongo.

Ensure that record data all consumed, otherwise throw error

When processing files, the tools do not recognize where there is too much data in the record.
For example, say that a record had a length of 100, but the bitmap processing only consumes 88. When this happens, you generally end up with a bit error on the next record because the next record does not align with the next records length or bitmap.
This change will provide better errors for users and require less debugging to determine the issue.

One of the MFE file can't extract

Dear sir,
thanks so much for your tool as it help me so much.There is some issue when I try extract the file which generate from Mastercard at last week.

I am thinking is that have any new format coming from Mastercard, as I can extract this file by using the MFE simulator after update the simulator to latest version.

There is the issue MFE folder, hope it can help you to handle this issue.

TT112T0.zip

Version display broken in 0.4.8

Revert change in config to overwrite _version.py file with fixed version details.

ValueError: invalid literal for int() with base 10: '@@'

Hi adelosa,

I am facing following issue while extracting the IPM file.

Traceback (most recent call last):
File "/usr/local/bin/mideu", line 11, in
sys.exit(cli_entry())
File "/usr/local/lib/python2.7/site-packages/mciutil/cli/mideu.py", line 27, in cli_entry
_main(args)
File "/usr/local/lib/python2.7/site-packages/mciutil/cli/mideu.py", line 53, in _main
args.func(args)
File "/usr/local/lib/python2.7/site-packages/mciutil/cli/convert.py", line 49, in convert_command
) for record in input_data
File "/usr/local/lib/python2.7/site-packages/mciutil/mciutil.py", line 178, in flip_message_encoding
source_format)
File "/usr/local/lib/python2.7/site-packages/mciutil/mciutil.py", line 225, in _flip_element_encoding
field_length = int(field_length_string)
ValueError: invalid literal for int() with base 10: '@@'

Can you please let me know where I am going wrong?
I have attached the ipm file below.

mcsample1.ipm.txt

I NEED TO CONVERT AN ASCII FILE TO EBCDIC 1014 BLOCKING, HOW DO I GO ABOUT IT WITH MCIUTIL

1014 blocked file format
This is the same as VBS format with 1014 blocking applied.

The VBS data is blocked into lengths of 1012, and an additional 2 x’40’ characters are appended at each block.

Finally, the total file length is made a multiple of 1014 with the final incomplete record being filled with the x’40’ character

File block warning and missing DE43 breakdown

First, thank you for the great library.

This is not an issue it is just a kind request of an enhancement, unfortintly I am not a python developer to handle it.

Is it possible to extract the csv file from IPM even if it contain a corrupted row or transaction for investigation purposes.

Thank you again.
Abdullah

Creating MC IPM file Test sample file and process

How to create EBCDIC IPM file from csv file? Help is appreciated.

extract DE55 fields into readable format

Currently there is no way for the mideu tool to extract de55 as it is a binary format. The convert format leaves the DE55 field as-is, but we should change the extract command to break down and output DE55 elements.

Option to read non 1014 blocked files

The utility currently only works with 1014 blocked files. Some users do not use this format.
Add a command line option to bypass or add the blocking option. In this case, the file will be parsed without any unblock or block.

Test cases using default mideu.yml config

The reason that I did not realise that the install process was not installing the mideu.yml file was because the test cases use an inline config embedded in the test cases.
To detect this issue, I need to add a test case that uses the default package config rather than an embedded one. This issue is to add a test case that checks this.

Creating MC IPM file Test sample file and process using Java

How to create an EBCDIC IPM file from a txt file using Java? Help is appreciated.