GitHubHelp

EthanCornell/mini-migration

Mini-Migration — Cross-platform resumable file-transfer tool. C++17 core, Objective-C++ macOS layer; built for Apple Backup & Migration workflows.

★ 1Forks 0C++GitHub ↗Compare
apfsbackupclidata-integritymacosmigration

README

Mini-Migration (v0.31)

Cross-platform proof-of-concept for Apple Backup & Migration workflows.

  • Core — C++17 resumable chunk copy with SHA-256 integrity
  • macOS layer — Objective-C++ bridge (MigrationEngine.mm) ready for NSProgress / os_log
  • Linux / WSL — Stub engine that re-uses the core (builds without Cocoa)
  • CLI — migrate -s <src> -d <dst>
  • CI — GitHub Actions matrix: ubuntu-22.04 + macos-14
  • Tests — smoke, functional, perf; fault-inject runs only on macOS

Status: MVP complete (green builds on both platforms). v0.2 is green on both platforms and now exposes progress + telemetry. v0.3 green on both platforms. Progress, telemetry, adaptive throughput, and hardened fault tests are now in place.
Next: APFS zero-copy, .resume.meta hardening, XPC daemon.

What’s new in v 0.31

Area Details
Incremental backup --delta builds 64 KiB CityHash index, copies only changed ranges; 50 MB file with 16 KiB edit patches in <200 ms.
copyRange helper Byte-range copier reused by delta path; atomically renames patched temp file.
Tests tests/scripts/incremental_flow.sh proves full → mutate → delta → cmp ok.
Run-tests harness Prints ✔️ / ❌ per script and exits non-zero on any failure.

Incremental (delta) backup NEW in v 3.1 🎉

Mini-Migration now supports block-level patching.
Instead of re-copying the entire file, the tool:

  1. Builds a 64 KiB block index (CityHash-64) for src and existing dst
  2. Detects the byte-ranges that changed
  3. Copies only those ranges into a temporary *.patching file
  4. Atomically renames it over the original

When should I use it?

  • Any time you re-export VM images or disk archives that change only in small areas (logs, headers, etc.)
  • Daily incremental backups where < 10 % of the data mutates.

CLI examples

# initial full copy (creates backup.img)
migrate -s src.img -d backup.img

# later… after modifying src.img:
migrate --delta -s src.img -d backup.img

Typical speed-up on an NVMe SSD:

File size Edited bytes Full copy Delta (--delta)
4 GB 40 MB (≈1 %) 20 s ≈ 0.6 s

Note
Delta mode automatically falls back to a full copy if the destination file does not exist. No extra metadata is required beyond the standard *.resume.meta used for crash-recovery.

1 . Features implemented

Area Details
Resumable copy Chunked I/O (4 MiB default). If process dies, resume from *.resume.meta.
Integrity Streaming SHA-256 via EVP_Digest (OpenSSL ≥ 3) or CommonCrypto on macOS.
CLI Minimal POSIX-style flags -s/-d, plus --help.
Cross-platform Builds on Linux/WSL (GCC 13 / Clang 17) and macOS (Xcode 15).
Automated tests tests/run_tests.sh orchestration: smoke → functional → perf → (mac only) fault-inject.
CI Workflow in .github/workflows/ci.yml; fast green checks on every push / PR.

2 . Quick start

Prerequisites

Platform Dependencies
Ubuntu 22.04+ / WSL2 clang-17 or g++-12+, cmake ≥ 3.22, libssl-dev, git, dos2unix
macOS 14+ Xcode Command-Line Tools, Homebrew cmake (OpenSSL optional — CommonCrypto is used by default)

Build & test

git clone https://github.com/<you>/mini-migration.git
cd mini-migration
./scripts/build.sh          # cmake configure + build
./tests/run_tests.sh        # all script tests → ✅

You should see something like:

skip fault test on non-mac
perf: 114 ms for 128 MB
✅ all script tests passed

(On macOS you will also see the fault-inject test; it must succeed.)

Example CLI session

# copy a 4 GB disk image to an external APFS volume
migrate -s ~/Downloads/BigSur.iso -d /Volumes/Backup/BigSur.iso

macOS runtime feedback

[00:00] starting…  0 % (0 B/4 GB)
[00:12] copying…  47 % (1.9 GB/4 GB)
[00:25] copying…  97 % (3.9 GB/4 GB)
Migration succeeded    bytesCopied=4294967296   retries=0
  • Linux / WSL shows the same stderr progress; no Console.app telemetry.
  • In Console.app › subsystem = com.demo.mini-migration you’ll see a sign-posted Migration interval with start/end times and byte count.

Resume scenario

Interrupt the transfer (e.g., Ctrl-C) and run again:

migrate -s ~/Downloads/BigSur.iso -d /Volumes/Backup/BigSur.iso
# prints “Resuming at offset 2.3 GB…”

*.resume.meta is detected, the file seeks to the offset, and the copy finishes checksum-verified without re-sending earlier chunks.

3 . Project layout

.
├── CMakeLists.txt
├── include/
├── src/                # core + platform engines + CLI
├── scripts/            # build & hook installers
├── tests/
│   ├── run_tests.sh
│   ├── scripts/        # smoke, perf, fault, net_stall
│   └── manual/large_file.sh
└── .github/workflows/ci.yml

4 . Roadmap

Milestone Description
v0.2 – macOS integration (completed) Add NSProgress callbacks, os_log signposts, Instruments template.
v0.3 – Fault-tolerance hardening (completed) More aggressive fault tests (network stall, SIGSTOP); verify 100 % recovery rate.
v0.31 – Incremental diff (completed) Mini-Migration can now run lightning-fast incremental backups. Patch-only mode fingerprints each file, copies just the changed blocks, and re-assembles the destination atomically.
v0.4 – APFS clone path Use copyfile(…, COPYFILE_CLONE) for zero-copy inside APFS.
v0.5 – XPC service Split engine into a background daemon; CLI becomes a thin client.
v1.0 – Windows ingress C# helper to seed files over SMB; full NTFS → APFS demo.

5 . License

MIT — see LICENSE for details.

Contributors

EthanCornell

Issues