100% Pure Swift AI Agent MCP Server, Dual-Tier Visual Canvas & UI Audit CLI for Native Apple Platforms

ViewLens is a single, standalone native macOS CLI and Model Context Protocol (MCP) server that connects AI coding agents (Claude Code, Cursor, Windsurf, Xcode AI) and software engineers—including blind and low-vision developers—to native Apple UI layouts, W3C WCAG 2.2 accessibility validation, and Apple HIG compliance.

100% Pure Swift: The MCP server runs directly inside the native viewlens binary (viewlens mcp) using in-process Swift JSON-RPC 2.0.

By combining an ultra-fast dual-tier headless canvas renderer with the fine-tuned CoreML vision models trained in NativeUIAuditKit, ViewLens detects UI elements, evaluates Apple Human Interface Guidelines (HIG), and performs programmatic Auto Layout introspection without burning multimodal LLM tokens on raw image uploads or waiting for multiple iOS simulator boots.

• ♿ Pioneering Nonvisual Authoring for Blind and Low-Vision Developers: Transforms visual Xcode canvas previews and simulator output into accessible hierarchies, VoiceOver-oriented speech streams, 40-cell braille simulations, and semantic spatial queries.
• 🎯 W3C WCAG 2.2 Level A/AA/AAA Auditing: Enforces programmatic Name/Role/Value (4.1.2), level-aware Touch Target sizing (2.5.8 AA / 2.5.5 AAA), Light/Dark contrast (1.4.3 / 1.4.6), and Dynamic Type AX1–AX5 reflow.
• 🧠 Token-Free UI Audits: LLMs receive structured JSON coordinate geometry (x, y, width, height in normalized top-left space) and semantic issue classifications rather than raw image pixels.
• 🏎️ Dual-Tier Headless Canvas (Zero Simulator Boot Matrix):
  • SwiftUI Views: Rendered in-process on macOS via ImageRenderer in $<5\text{ms}$ with injected device dimensions, Dynamic Type sizes, and color schemes.
  • UIKit Views & ViewControllers: Rendered via a Mac Catalyst harness running the real Apple UIKit and Auto Layout engine directly on macOS.
• 🔍 Root-Cause + Visual Symptom Diagnostics:
  • Structural Introspection: Programmatically detects ambiguous layout via UIView.hasAmbiguousLayout and constraint conflicts.
  • Visual Intelligence: YOLO11n CoreML vision model catches sub-44pt touch targets, boundary clipping, text truncation, and cross-element overlaps.
• 📐 Direct SwiftUI / UIKit Alignment: Bounding boxes match SwiftUI .frame(x:y:width:height:) and UIKit coordinate conventions directly—no Y-axis inversion needed.
• 🚦 CI/CD Quality Gates: Enforces strict layout verification in CI pipelines with --strict exit codes.

flowchart TD
    subgraph Clients["1. Agent & Developer Clients"]
        Agent["🤖 AI Agent (Claude Code / Cursor)"]
        Dev["👨‍💻 Developer Terminal / CI Pipeline"]
        MacApp["🖥️ ViewLens macOS App"]
    end

    subgraph NativeBinary["2. Standalone Swift Binary (`viewlens`)"]
        MCPServer["⚡ In-Process Swift MCP Server<br/>(`viewlens mcp` stdio JSON-RPC)"]
        CLI["⚡ CLI Commands<br/>(`viewlens doctor / scan / batch / render`)"]
    end

    subgraph RenderingHarness["3. Dual-Tier Rendering Harness"]
        T1["⚡ Tier 1: Pure SwiftUI Canvas<br/>(In-process macOS ImageRenderer + Virtual Traits)"]
        T2A["🍏 Tier 2A: Mac Catalyst Harness<br/>(Real UIKit & Auto Layout on macOS via Subprocess IPC)"]
        T2B["📱 Tier 2B: Pre-warmed Simulator Host<br/>(Fallback reference for iOS-exclusive notch/font fidelity)"]
    end

    subgraph DualDiagnostics["4. Dual-Layer Diagnostics Engine"]
        direction LR
        subgraph Structural["Structural Introspection"]
            AL["📐 UIView.hasAmbiguousLayout<br/>(Window-attached)"]
            A11y["♿ Accessibility & Truncation Probes"]
        end
        subgraph VisionEngine["Visual Intelligence"]
            CoreML["🧠 NativeUIAuditKit CoreML<br/>(YOLO11n on Apple Neural Engine)"]
            HIGRules["📏 Deterministic HIG Rules<br/>(44pt touch targets, overlaps, clipping)"]
        end
    end

    Agent -->|MCP stdio JSON-RPC| MCPServer
    Dev -->|Shell commands| CLI
    MacApp -->|Direct Link| CLI

    MCPServer --> RenderingHarness
    CLI --> RenderingHarness
    RenderingHarness --> DualDiagnostics
    DualDiagnostics -->|Structured Diagnostic JSON| MCPServer
    DualDiagnostics -->|Structured Diagnostic JSON| CLI

Pre-flight readiness probe verifying that the CoreML model is located, passes size validation, and performs a cold-load warmup test.

viewlens doctor --json

{
  "status": "ready",
  "checks": [
    { "name": "model_found", "status": "confirmed", "detail": "/path/to/best.mlpackage" },
    { "name": "model_size", "status": "confirmed", "detail": "4.8MB (< 25MB threshold)" },
    { "name": "model_loads", "status": "confirmed", "detail": "Cold load: 0.84s" }
  ],
  "recommendedNextCommand": "viewlens scan <image-path>"
}

Audits one or more screenshot images. In multi-image mode, the CoreML model is loaded once into memory to process all targets efficiently.

# Human-friendly visual table output
viewlens scan ./screenshots/HomeScreen.png

# Agent-friendly structured JSON with annotated bounding box overlay output
viewlens scan ./screenshots/Login.png --format json --overlay ./reports/Login_annotated.png

# Strict CI mode (exits 1 if any layout or accessibility violations are discovered)
viewlens scan ./screenshots/Profile.png --strict

Audits entire directory trees containing test artifacts or simulator outputs, producing a consolidated JSON audit report.

viewlens batch ./DerivedData/Screenshots --pattern "png" --output ./audit_report.json

Renders and audits a SwiftUI/UIKit template across a multi-device matrix in memory without simulators:

# Audits template across iPhone SE and iPhone 16 Pro in Light and Dark mode
viewlens render --template LoginForm --devices iPhoneSE,iPhone16Pro --dt large,accessibility3 --scheme light,dark

# Output structured JSON report
viewlens render --template LoginForm --format json

Resolves the bounded local source, resource, package-pin, and preview-scenario closure for a Swift view without building or downloading dependencies:

viewlens context --workspace . --root-symbol ProfileView --missing-resource-policy request
viewlens context --workspace . --source-file Sources/ProfileView.swift --emit-harness

Synthetic resource and scenario substitutions are explicitly marked and are never eligible for visual-baseline approval.

Maps changed Swift or resource files to affected top-level views and recommends a conservative accessibility test matrix:

viewlens impact --workspace . --changed-files Sources/ProfileView.swift Assets.xcassets/avatar.imageset/Contents.json

Impact confidence is lexical by default and upgrades to build_plan_confirmed when an existing SwiftPM build description or Xcode Swift file list proves target membership. Full compiler_confirmed confidence remains reserved for future Index Store symbol evidence.

Modern MCP impact calls also return a private viewlens://impact-capsules/{capsuleId} resource link. Capsules are retained in a bounded in-memory catalog and contain relative affected paths, root views, audit recommendations, and evidence summaries—but no source bodies or workspace-root path.

Launches the interactive full-screen Terminal User Interface (TUI) or headless ASCII layout dashboard:

# Interactive full-screen terminal dashboard with live keyboard shortcuts
viewlens tui

# Headless snapshot output for CI or SSH sessions
viewlens tui --headless --template LoginForm

Executes automated quality gates enforcing Apple HIG, Touch Targets (≥44pt), Dynamic Type reflow, and Dark Mode:

# Fast pre-commit audit on staged views (<1s)
viewlens hook pre-commit

# Strict CI / PR check generating GitHub Markdown step summaries
viewlens hook pull-request --output-markdown reports/pr_summary.md

# Install git pre-commit hook in local repository
viewlens install-hook --type pre-commit

# Generate starter .viewlens.json configuration
viewlens init-config

Runs a level-aware WCAG 2.2 audit. Template audits evaluate programmatic semantics, 24pt AA or 44pt AAA target sizing, Light/Dark contrast, AX1/AX3/AX5 reflow, and portrait/landscape rendering. UIKit semantics come from live hierarchy introspection; headless SwiftUI templates use registered semantic snapshots. Screenshot-only and unregistered-template audits explicitly mark checks that cannot be established as not evaluated.

viewlens accessibility --template LoginForm --level AA
viewlens accessibility --image ./screenshots/Login.png --level AAA --json

Compares a rendered template with a Figma or baseline reference using SSIM and an optional visual heatmap, with accessibility verification enabled by default.

viewlens design-diff --reference ./designs/Login.png --template LoginForm --heatmap ./reports/Login_diff.png

Launches the 100% pure Swift Model Context Protocol (MCP) server over standard I/O (stdio):

viewlens mcp

Blind and low-vision Apple platform developers face a persistent barrier: Xcode's SwiftUI Canvas previews, Interface Builder, and simulator-rendered screens communicate essential layout information visually without providing an equivalent, efficient nonvisual representation.

ViewLens addresses this by converting visual UI layouts, device permutations, and accessibility trees into a structured, VoiceOver-oriented nonvisual authoring environment and automated WCAG 2.2 / Apple HIG auditing engine.

For full workflows, keyboard maps, and agent prompts, see the ViewLens Nonvisual Authoring Guide.

flowchart LR
    subgraph Input["1. Layout Source"]
        SwiftUI["SwiftUI / UIKit Code"]
        Shot["Screenshot / Simulator"]
    end

    subgraph NonvisualEngine["2. Nonvisual Authoring Engine"]
        Model["🧠 NonvisualScreenModel<br/>(Scalar Stable IDs)"]
        VOPredict["🗣️ VoiceOverPredictor<br/>(Reading Order & Rotor)"]
        Braille["⠃⠗⠇ Braille Simulation<br/>(40-Cell Pin Map)"]
        Correlate["🔬 Visual-Semantic Correlator<br/>(CoreML vs AX Hierarchy)"]
        Focus["🕸️ Focus Graph Engine<br/>(Trap & Cycle Detection)"]
    end

    subgraph Output["3. VoiceOver-Accessible Interfaces"]
        MacApp["🖥️ macOS Dual-Pane Outline (⌥⌘O)"]
        TUI["📟 Terminal Dashboard (viewlens tui)"]
        MCP["🤖 AI Coding Agents (16 MCP Tools)"]
    end

    Input --> NonvisualEngine
    NonvisualEngine --> Output

ViewLens transforms any SwiftUI/UIKit view or screenshot into a structured, scalar NonvisualScreenModel with deterministic identifiers (screen_..., elem_..., finding_...). Developers can inspect bounds, hierarchical parents, and spatial relationships without relying on visual inspection or sighted assistance.

• Spoken Speech Streams: Produces an API-derived prediction of VoiceOver phrases and traversal order. It does not replace testing with VoiceOver on the target platform.
• Simulated 40-Cell Braille: Formats refreshable braille pin displays for braille users.
• Rotor Inventory: Catalogs heading levels, form controls, and custom accessibility actions.
• Announcement Storm Detection: Flags rapid speech bursts ($<0.5\text{s}$) and duplicate announcements.

Cross-references on-device YOLO CoreML computer vision (rendered pixels) with the native accessibility hierarchy (programmatic semantics available to assistive technologies):

• Catches Missing Semantics: Alerts when an interactive visual button lacks .accessibilityAddTraits(.isButton) or .accessibilityLabel.
• Catches Invisible / Obscured Nodes: Alerts when an accessibility node exists in code but is clipped off-screen or renders with 0x0 pixels.
• Flags Role Conflicts: Detects when visual styling (e.g. primary button) conflicts with programmatic accessibility traits.

Constructs the directed graph of sequential keyboard and switch-control navigation:

• Automatically detects focus traps (loops where focus cannot escape a modal or card).
• Identifies unreachable elements that cannot be focused via keyboard or assistive switches.

• macOS Desktop App (ViewLens.app): Features a VoiceOver-accessible dual-pane outline with dedicated keyboard shortcuts (⌥⌘O to toggle outline, ⌥⌘L for landmarks, ⌥⌘S for spatial relationships, ⌥⌘I for findings).
• Interactive Terminal UI (viewlens tui): Provides a full-screen, keyboard-navigable ASCII layout and semantic list that works over SSH and in terminal environments.

{
  "mcpServers": {
    "viewlens": {
      "command": "/path/to/ViewLens/.build/release/viewlens",
      "args": ["mcp"],
      "env": {
        "VIEWLENS_MODEL_PATH": "/path/to/NativeUIAuditKit/models/best.mlpackage"
      }
    }
  }
}

{
  "mcpServers": {
    "viewlens": {
      "command": "/path/to/ViewLens/.build/release/viewlens",
      "args": ["mcp"]
    }
  }
}

Or run the automated setup script:

./scripts/install_mcp.sh

ViewLens offers unique native Apple platform capabilities that set it apart from standard web testing and general multimodal AI tools:

Traditional AI agents burn thousands of multimodal vision tokens uploading raw high-res screenshots to external APIs. ViewLens runs YOLO11n CoreML inference locally on the Apple Neural Engine (ANE) in $<10\text{ms}$. It passes back compact, structured JSON bounding boxes [x, y, w, h] and classified layout defects—saving 95%+ in token costs and eliminating API latency.

Forget waiting 30 seconds for iOS simulators to boot. ViewLens renders SwiftUI templates directly in-process on macOS via ImageRenderer:

• Permutes across iPhone SE (3rd gen), iPhone 16 Pro, iPad Pro (13-inch), and Landscape in under $5\text{ms}$ per frame.
• Simulates Dynamic Type sizes from Default (100%) to AX5 (312%), instantly exposing text clipping and button overflows before committing code.

ViewLens correlates computer vision bounding boxes with programmatic accessibility nodes:

• Catches Missing Semantics: Identifies visual buttons that look like buttons but lack .accessibilityAddTraits(.isButton) or .accessibilityLabel.
• Catches Obscured Elements: Identifies accessibility elements that exist in the hierarchy but are off-screen, clipped, or have 0-pixel bounds.
• Flags Role Conflicts: Alerts you when a visual element classification disagrees with programmatic accessibility roles.

AI coding agents don't have human spatial vision, but with viewlens_query_spatial they can ask:

• "What element is at coordinate (0.5, 0.35)?"
• "What is the nearest interactive control to the user's touch point?"
• "Is the Header .above the Save Button, or are they .overlapping?"
• "Is the Icon .inside the Container view?"

Verify implementation fidelity against Figma exports using Structural Similarity (SSIM):

• Generates pixel-delta heatmaps highlighting layout drifts, color shifts, and padding discrepancies.
• Computes exact match percentages and classifies whether design deltas violate HIG spacing or WCAG contrast.

ViewLens builds a complete nonvisual model of your UI:

• Simulates natural VoiceOver reading order and rotor navigation sequences.
• Generates spoken speech strings and simulated refreshable braille display outputs.
• Detects announcement bursts ($<0.5\text{s}$) and duplicate speech messages.

With viewlens_accessibility_graph, ViewLens builds a directed graph of sequential keyboard and switch-control navigation:

• Automatically detects focus traps (circular focus cycles where a user cannot tab out of a modal or card).
• Identifies unreachable elements that cannot be navigated to via sequential keyboard or assistive switches.

When an AI agent interacts with your app via viewlens_ui_perform or viewlens_flow_replay:

• Automatically rejects and redacts Bearer tokens, API keys, and sensitive patterns.
• Prevents typing credentials into password fields.
• Replays declared multi-step user workflows deterministically and verifies step assertions (element_exists, element_contains_text).

When you fix an issue, ViewLens compares your before-and-after audit runs and categorizes findings into:

• resolvedIssues: Violations successfully eliminated.
• remainingIssues: Unresolved existing issues.
• introducedIssues: Regressions! Any new defects created by your code change.

Run viewlens hook pre-commit for sub-second pre-commit checks, or generate gorgeous GitHub Markdown step summaries in your CI/CD pipelines with viewlens hook pull-request --output-markdown pr_summary.md.

ViewLens outputs coordinates in normalized top-left space [0.0, 1.0], matching SwiftUI .frame() and UIKit coordinate systems.

(0,0) ────────────────────────── (1,0)
  │                                │
  │    ┌──────────────────┐        │
  │    │  (x, y)          │        │
  │    │   BoundingBox    │ height │
  │    │                  │        │
  │    └────── width ─────┘        │
  │                                │
(0,1) ────────────────────────── (1,1)

{
  "sourceMode": "screenshot",
  "image": "screenshots/checkout_screen.png",
  "dimensions": { "width": 1179, "height": 2556, "scale": 3.0 },
  "elements": [
    {
      "type": "primaryButton",
      "confidence": 0.962,
      "boundingBox": { "x": 0.05, "y": 0.88, "width": 0.90, "height": 0.048 }
    }
  ],
  "issues": [
    {
      "kind": "tappableTargetTooSmall",
      "severity": "error",
      "description": "primaryButton height 38pt (114px @3x) is below Apple HIG requirement of 44x44pt.",
      "confidence": 0.962,
      "elementIndex": 0,
      "remediation": {
        "description": "Increase frame dimensions to at least 44x44pt.",
        "codeSnippet": ".frame(minWidth: 44, minHeight: 44)"
      }
    }
  ],
  "passed": false,
  "summary": {
    "totalElements": 1,
    "totalIssues": 1,
    "errorCount": 1,
    "warningCount": 0,
    "worstIssue": "primaryButton height 38pt (114px @3x) is below Apple HIG requirement of 44x44pt."
  }
}

# Fast unit tests (default) — no simulator required
swift test

# Also run the gated live-simulator integration suite (requires a booted simulator)
VIEWLENS_ENABLE_INTEGRATION_TESTS=1 swift test --filter RuntimeBackendIntegrationTests

The gated suite is skipped by default so swift test stays fast and deterministic in CI; it exercises the real xcrun simctl io screenshot capture path in ProcessController.

ViewLens is open source software released under the MIT License.