This project implements an OpenEnv-compatible reinforcement learning environment that simulates a real-world document formatting task:

Convert structured content into HTML such that the rendered PDF matches a reference document in content, structure, and layout.

This environment evaluates practical agent capabilities such as:

• Structured generation
• Formatting correctness
• Layout reasoning
• Iterative improvement

Most RL environments are toy problems. This environment focuses on:

• Real-world document workflows (resumes, reports, structured docs)
• HTML → PDF rendering correctness
• Semantic + structural + layout evaluation
• Dense feedback for learning agents

DocumentAction:
    html_code: str

Agent outputs HTML representing the document.

DocumentObservation:
    task_description
    render_success
    render_log
    extracted_text
    score
    pdf_score_breakdown
    html_score_breakdown
    best_score
    attempt_number

Agent receives:

• Rendering feedback
• Extracted PDF content
• Score + breakdown
• Progress signal

DocumentState:
    episode_id
    task_description
    step_count
    best_score
    task_id
    difficulty
    reference

Tracks task + progress across steps.

Each step() executes:

• Sanity checks on generated HTML

• Converts HTML to PDF

• Text extraction
• Layout extraction (bounding boxes, fonts, spacing)

• Keyword matching (exact + fuzzy)
• Section detection
• Section ordering
• Anti-spam penalty

• Hierarchy validation (heading → paragraph)
• Alignment consistency
• Spacing correctness
• Content density checks

• Structural validation (partial)

Dense reward in range [0.0, 1.0]

reward =
    0.3 * html_score +
    0.4 * text_score +
    0.3 * layout_score

• Continuous reward (not sparse)
• Encourages partial correctness
• Penalizes spam / bad formatting
• Supports iterative refinement

3 task levels:

• Keyword + section presence

• Structure + ordering

• Full layout + formatting quality

Each task includes:

• Prompt
• Reference content
• Expected keywords
• Expected sections

reset() → new task
step(action) → evaluate HTML
state() → current state

Episode ends when:

• Max steps reached OR
• Score threshold achieved

Deterministic and reproducible.

• Keyword coverage
• Section presence
• Ordering correctness
• Spam penalty

• Hierarchy correctness
• Alignment
• Spacing
• Density

Includes inference.py:

• Uses OpenAI-compatible API
• Outputs:

[START]
[STEP]
[END]

• Computes:
  • Step rewards
  • Final normalized score (0–1)

uvicorn server.app:app --host 0.0.0.0 --port 8000

docker build -t document-env .
docker run -p 8000:8000 document-env

env.reset()
env.step(DocumentAction(html_code="<html>...</html>"))

• Hugging Face Spaces compatible
• WebSocket-based sessions
• Supports concurrent environments

This environment evaluates:

Structured document generation with layout correctness

A capability where current LLMs are weak.

• Improve HTML grading
• Add multi-page layout support
• Expand task diversity
• Add stricter layout constraints

• Real-world RL environment
• Dense, meaningful reward
• Deterministic evaluation
• OpenEnv compliant
• Scalable deployment ready

Meta x PyTorch OpenEnv Hackathon Submission 🚀