• Introduction
• Data Augmentation Pipeline
• Model Download
• Evaluation Results
  • COBOL Code Generation
  • COBOL-Java Translation
• Getting Started
  • Installation
  • Training
  • Inference
  • Evaluation
• COBOL-JavaTrans Benchmark
• Repository Structure
• Acknowledgements
• Citation

COBOL-Coder is a family of domain-adapted LLMs specialized for COBOL code generation and bidirectional COBOL-Java code translation. Built on top of Qwen2.5-Coder, COBOL-Coder addresses the critical gap in LLM capabilities for legacy programming languages.

COBOL remains a critical language for mainframe systems in banking, insurance, and government, yet existing LLMs struggle to generate and translate COBOL code correctly. COBOL-Coder bridges this gap through domain-specific data curation and fine-tuning.

Key highlights:

• 73.95% compilation success rate and 49.33 Pass@1 on COBOLEval, compared to 41.8% and 16.4 for GPT-4o
• 34.93 Pass@1 on Java-to-COBOL translation, where all other LLMs (including GPT-4o) achieve near-zero scores
• The only model with non-trivial performance on COBOLCodeBench (26.09% CSR, 4.35 Pass@1)
• An automated data augmentation pipeline combining compiler-guided validation with multi-stage similarity-based filtering
• COBOL-JavaTrans, the first benchmark for bidirectional COBOL-Java translation

We construct COBOL-specific training data through three complementary sources and a multi-stage validation pipeline:

Source 1: Public COBOL Code from GitHub - 40,829 unique COBOL files mined from public repositories, cleaned via MinHash deduplication, then validated through a compiler-based self-debugging loop (GnuCOBOL + LLM repair, K=3 iterations), yielding 31,492 compilable programs.

Source 2: Synthetic COBOL via Code Translation - Java programs from The Stack v2-dedup are translated to COBOL using GPT-4o, then validated through:

• Stage 1: Compiler-based Validation - Iterative compilation and LLM-guided repair
• Stage 2: Similarity-based Validation - LLM-based pair scoring and AST-based back-translation scoring with CodeBERTScore

Source 3: COBOL and Mainframe Knowledge Sources - Licensed textbooks and technical websites, segmented and transformed into instruction-style QA pairs.

• Stage 3: Instruction Generation - For both Source 1 and Source 2, multiple LLMs (GPT-4, GPT-4o-mini, GPT-oss-120B, CodeLlama-70B) generate candidate problem descriptions, with GPT-4o selecting the best via LLM-as-a-judge.

All evaluations are performed in a zero-shot setting with temperature=0.0. Results are averaged over three runs.

Performance on COBOLEval (146 tasks) and COBOLCodeBench (46 tasks). Bold = best in block; Bold + Underline = overall best.

Performance on the COBOL-JavaTrans benchmark (143 pairs). C2J = COBOL-to-Java, J2C = Java-to-COBOL.

conda create -n cobol-coder python=3.10 && conda activate cobol-coder

git clone https://github.com/COBOL-Coder/COBOL-Coder.git
cd COBOL-Coder
pip install -r requirements.txt

For evaluation, you also need:

• GnuCOBOL compiler (cobc, version 2.0.0) for COBOL compilation
• Java JDK (javac, version 17.0.18) for Java compilation (C2J evaluation)

Fine-tune Qwen2.5-Coder on COBOL-specific data using DeepSpeed ZeRO-3:

bash sft.sh

Generate COBOL code completions for the COBOLEval benchmark:

python evaluation/generate_coboleval.py \
    --model_path path/to/COBOL-Coder-14B \
    --output_dir evaluation/output \
    --data_path evaluation/data/CobolEval.jsonl

Generate translations for COBOL-JavaTrans:

# COBOL-to-Java
python evaluation/generate_cobol_javatrans.py \
    --model_path path/to/COBOL-Coder-14B \
    --direction c2j \
    --output_dir evaluation/output

# Java-to-COBOL
python evaluation/generate_cobol_javatrans.py \
    --model_path path/to/COBOL-Coder-14B \
    --direction j2c \
    --output_dir evaluation/output

Evaluate COBOL code generation (COBOLEval) and Java-to-COBOL translation (J2C):

python evaluation/evaluate_coboleval.py \
    --samples_file evaluation/output/<model>_coboleval_samples.jsonl \
    --data_path evaluation/data/CobolEval.jsonl

Evaluate COBOL-to-Java translation (C2J):

python evaluation/evaluate_translation_c2j.py \
    --samples_file evaluation/output/<model>_javatrans_c2j.jsonl \
    --data_path evaluation/data/COBOL-JavaTrans.jsonl

Example output files from COBOL-Coder are provided in evaluation/output/.

COBOL-JavaTrans is the first benchmark specifically designed for bidirectional COBOL-Java code translation, derived from HumanEval. It contains 143 task pairs (out of 164 HumanEval tasks) with both COBOL and Java implementations, along with test cases for both languages. Tasks were constructed using a vibe-coding-inspired workflow, with all programs manually reviewed and validated for compilability and functional correctness.

The benchmark data is included in evaluation/data/.

COBOL-Coder/
├── src/                          # LLaMA-Factory source (training framework)
├── evaluation/
│   ├── data/                     # Benchmark datasets
│   │   ├── CobolEval.jsonl
│   │   └── COBOL-JavaTrans.jsonl
│   ├── output/                   # Example outputs from COBOL-Coder
│   ├── generate_coboleval.py     # Inference for COBOLEval
│   ├── generate_cobol_javatrans.py  # Inference for COBOL-JavaTrans
│   ├── evaluate_coboleval.py     # Evaluation for COBOL code (CSR + Pass@1)
│   ├── evaluate_translation_c2j.py  # Evaluation for C2J translation
│   └── utils.py                  # Shared utilities
├── examples/                     # DeepSpeed configs and training examples
├── scripts/                      # Utility scripts
├── docker/                       # Docker configurations (CUDA, NPU, ROCm)
├── sft.sh                        # Training launch script
├── requirements.txt              # Python dependencies
└── cobol_reserved_words.txt      # COBOL vocabulary for tokenizer

• The authors gratefully acknowledge the support and resources provided by Dr. Phong Xuan Nguyen of the FPT Software AI Center, Vietnam
• This codebase is built on LLaMA-Factory for efficient fine-tuning
• COBOLEval benchmark by BloopAI
• COBOLCodeBench by Kumar (2025)
• Base model: Qwen2.5-Coder by the Qwen team

If you find this work useful, please cite our paper:

@article{dau2026cobol,
  title={COBOL-Coder: Domain-Adapted Large Language Models for COBOL Code Generation and Translation},
  author={Dau, Anh TV and Tan, Shin Hwei and Yang, Jinqiu and Bui, Nghi DQ and Nguyen, Anh Tuan},
  journal={arXiv preprint arXiv:2604.03986},
  year={2026}
}
}

For our earlier work on mainframe modernization:

@article{dau2024xmainframe,
  title={XMainframe: A Large Language Model for Mainframe Modernization},
  author={Dau, Anh TV and Dao, Hieu Trung and Nguyen, Anh Tuan and Tran, Hieu Trung and Nguyen, Phong X and Bui, Nghi DQ},
  journal={arXiv preprint arXiv:2408.04660},
  year={2024}
}