Emmanuel-Tech-Dev/nexus

★ 0Forks 0JavaScriptGitHub ↗Compare

README

Nexus V1 — Infrastructure Orchestrator

Stop configuring. Start coding.

Nexus is a local infrastructure control plane for Node.js + MySQL projects. It automates the repetitive friction of setting up new backend projects — provisioning databases, scaffolding folder structures, generating config files, and monitoring connection health — so you can focus on business logic from minute one.


Table of Contents


What Nexus Does

For every project you provision, Nexus:

  1. Creates a MySQL schema — nexus_<slug>
  2. Creates a scoped MySQL user with DML + DDL privileges on that schema only
  3. Grants SELECT on performance_schema for query monitoring
  4. Scaffolds the project folder structure from the template manifest
  5. Generates .env with provisioned DB credentials
  6. Generates package.json with template dependencies
  7. Generates src/config/db.js — a mysql2 connection pool reading from .env
  8. Generates src/index.js — an Express entry point with a /health endpoint
  9. Generates .gitignore
  10. Runs npm install in the background
  11. Registers everything in the internal registry
  12. Starts monitoring the connection via scheduled pings and slow query polls

What Nexus Does Not Do

Nexus never writes your business logic. It does not generate routes, controllers, models, middleware, or any application code. After scaffolding, you own everything inside /src except the three generated config files.


System Architecture

┌─────────────────────────────────────────────────────┐
│                    NEXUS SERVER                      │
│                                                     │
│  ┌──────────────┐      ┌─────────────────────────┐  │
│  │  REST API    │      │   Monitor (always-on)   │  │
│  │  (Express)   │      │   Ping + Slow Query     │  │
│  └──────┬───────┘      └──────────┬──────────────┘  │
│         │                         │                  │
│  ┌──────▼─────────────────────────▼──────────────┐  │
│  │           Pipeline / Registry                  │  │
│  │          (nexus_internal DB)                   │  │
│  └───────────────────────┬────────────────────────┘  │
└──────────────────────────┼──────────────────────────┘
                           │
           ┌───────────────┼───────────────┐
           │               │               │
    ┌──────▼──────┐ ┌──────▼──────┐ ┌─────▼──────┐
    │  MySQL       │ │  Filesystem │ │  npm       │
    │  (schemas)   │ │  (projects) │ │  (install) │
    └─────────────┘ └─────────────┘ └────────────┘

Nexus operates in two distinct modes simultaneously:

Provisioner Mode — activated on demand via API or CLI. Runs an atomic pipeline to set up a new project from scratch.

Monitor Mode — always running in the background from server boot. Pings every registered connection on a schedule and polls performance_schema for slow query data.


Technical Stack

Backend

  • Runtime: Node.js v18+ (ESM modules)
  • Framework: Express.js
  • Database: MySQL 8.0+
  • Driver: mysql2/promise
  • File uploads: multer (multipart form support)

Frontend

  • Framework: React 19 + Vite
  • UI Library: Ant Design v6
  • Styling: Tailwind CSS v4
  • Routing: React Router DOM v6
  • Server State: TanStack React Query v5
  • UI State: Zustand v4
  • HTTP: Axios
  • Charts: Recharts
  • Dates: Day.js

Prerequisites

  • Node.js v18 or higher
  • MySQL 8.0 or higher
  • npm

Installation

Backend

git clone <repo>
cd nexus
npm install

Frontend

cd nexus-ui
npm install

Configuration

Create src/config/nexus.env inside the backend folder:

# Nexus Server
NEXUS_PORT=7700

# MySQL Root Access
# Nexus needs root (or equivalent) to CREATE DATABASEs and CREATE USERs
NEXUS_MYSQL_HOST=localhost
NEXUS_MYSQL_PORT=3306
NEXUS_MYSQL_ROOT_USER=root
NEXUS_MYSQL_ROOT_PASSWORD=

# Provisioning
NEXUS_SCHEMA_PREFIX=nexus

# Encryption key for remote connection credentials (any 32+ char string)
NEXUS_ENCRYPTION_KEY=your32characterencryptionkeyhere

# Filesystem browser root (restrict browsing to this path)
NEXUS_FS_ROOT=F:/dev

# npm install timeout in ms
NEXUS_NPM_TIMEOUT_MS=120000

Important: NEXUS_MYSQL_ROOT_PASSWORD can be empty if your MySQL instance has no root password. Only NEXUS_MYSQL_ROOT_USER and NEXUS_ENCRYPTION_KEY are strictly required.


Database Setup

Run the registry schema against your MySQL instance before starting Nexus:

mysql -u root < src/db/schema.sql

This creates the nexus_internal database with six tables:

Table Purpose
templates Cached template manifests loaded from filesystem
projects All provisioned projects and their status
connections DB connections per project (local + remote)
db_users Scoped MySQL users created per project
events Full audit log of every pipeline step
metrics Time-series ping and slow query data

Running Nexus

Backend

# Development (with nodemon)
npm run dev

# Production
npm start

On startup Nexus:

  1. Verifies registry DB connection
  2. Loads template manifests from src/templates/ into DB cache
  3. Starts the monitor scheduler — picks up all active connections
  4. Starts the HTTP server on NEXUS_PORT (default 7700)

Frontend

cd nexus-ui
npm run dev

Opens on http://localhost:3000. All /api requests are proxied to http://localhost:7700 via Vite's proxy config — no CORS issues.

CLI

# Provision a new project
node bin/nexus.js spam NomadSync F:/dev/nomadsync

# List all projects
node bin/nexus.js list

# View project events
node bin/nexus.js status <projectId>

If Nexus is not already running, the CLI auto-starts it as a background process before executing the command.


Backend Architecture

Registry Schema

Six tables power the entire system. Key design decisions:

connections — one project, many connections. In V1 every project gets one MySQL connection created automatically. In V2, Redis and Postgres connections are added as additional rows — no schema change needed.

credential vs password_ref. Local connections store the env var name (DB_PASSWORD) in the credential column — the actual password only lives in the project .env on disk. Remote connections store the AES-encrypted password. The credential_type column distinguishes them.

events — high-resolution timestamps. Uses DATETIME(3) for millisecond precision so the CLI can stream events in the correct order even when multiple steps complete within the same second.

metrics — separate from events. Ping data is high-frequency append-only. Keeping it separate from lifecycle events means you can query latency history without scanning provisioning noise.


Pipeline System

The Nexus pipeline is an implementation of the saga pattern. Each provisioning operation is a sequence of discrete handlers. If any handler fails, compensation handlers run in reverse order to undo everything that succeeded — leaving the system in a clean state.

Handler Array (forward)          Compensation Array (reverse on failure)
─────────────────────            ──────────────────────────────────────
1. validateInput                 dropDatabase
2. registerProject               removeDirectory
3. provisionDatabase     ──────► deregisterProject
4. scaffoldFilesystem
5. mergeUserImports
6. generateEnv
7. generateProjectFiles
8. runNpmInstall

The runner (src/pipeline/runner.js) executes handlers in order. Each handler is an async function that receives and mutates the shared context object. Handlers return a payload object that gets logged to the events table.

// Handler signature
export async function myHandler(ctx) {
  // read from ctx
  // mutate ctx
  // return payload for events log
  return { key: "value" };
}

Context object (src/pipeline/context.js) carries all shared state through the pipeline:

{
  name, slug, path, templateName,   // input
  template, project, projectId,     // resolved during pipeline
  connection, dbUser,               // set by provisionDatabase
  mergedEnvVars, mergedPackageJson, // set by mergeUserImports
  userImports,                      // from API request
  startedAt, error,                 // pipeline tracking
}

Pipeline Handlers

validateInput — validates name, path, template. Checks uniqueness. Attaches resolved template to context. Uses parseIfString() helper because mysql2 auto-parses JSON columns on read.

registerProject — inserts the project row first so all subsequent pipeline steps have a valid project_id FK to log events against. Status starts as provisioning.

provisionDatabase — connects as root, creates the schema, creates a scoped MySQL user, grants DML + DDL on the schema and SELECT on performance_schema. Stores the connection row with credential_type: 'env_ref' — the real password is attached to ctx.connection.password temporarily for generateEnv to write, then discarded.

scaffoldFilesystem — creates all directories defined in the template manifest's directories array using fs.mkdir with recursive: true.

mergeUserImports — resolves user-provided imports (from importPath on disk or direct file uploads). Uploads take precedence over importPath. Attaches parsed results to ctx.mergedPackageJson and ctx.mergedEnvVars.

generateEnv — calls buildEnvContent() from the env generator service. Merges user env vars with Nexus provisioned vars. Nexus vars always win on conflicts. Annotates overwritten keys with comments in the file.

generateProjectFiles — iterates the template manifest's files array. Each file entry has a generator key that maps to a function in src/services/fileGenerators.js. Detects ESM vs CJS from package.json type field and generates appropriate syntax.

runNpmInstall — spawns npm install as a child process. Uses npm.cmd on Windows. Awaits completion — if it times out or exits non-zero, throws so the runner records the failure. Returns stdout tail for the events log.


Compensation System

Three compensation handlers in src/pipeline/handlers/compensations.js:

dropDatabase — drops the provisioned schema and MySQL user if they were created.

removeDirectory — removes the scaffolded project directory with fs.rm({ recursive: true, force: true }).

deregisterProject — deletes the project row from nexus_internal. The ON DELETE CASCADE foreign keys automatically clean up connections, db_users, and events rows.


Monitor System

The monitor starts at boot and runs independently of the provisioner. It maintains two Maps:

pingIntervals:      Map<connectionId, IntervalHandle>
slowQueryIntervals: Map<connectionId, IntervalHandle>

pingWorker.js — opens a fresh connection per ping (not a pool — pool would mask real latency), runs SELECT 1, records round-trip time to the metrics table. Updates connections.status to active or unreachable.

slowQueryWorker.js — queries performance_schema.events_statements_summary_by_digest scoped to the project schema. Records top 20 slowest queries by average time, including exec count, max latency, and no-index usage flag.

credentialResolver.js — single responsibility: given a connection row, return a usable password string. Handles env_ref (reads from project .env on disk) and encrypted (AES decrypts using NEXUS_ENCRYPTION_KEY).

scheduler.js — exports scheduleConnection() and unscheduleConnection() so the connections controller can register/deregister connections at runtime without restarting Nexus.


Template System

Templates live in src/templates/<name>/manifest.json. The loader reads them at startup and upserts into the templates table (DB cache). Version comparison prevents unnecessary re-inserts.

Manifest structure:

{
  "name": "basic-express",
  "version": "1.0.0",
  "description": "...",
  "type": "module",
  "engines": ["mysql"],
  "directories": [
    "src",
    "src/routes",
    "src/controllers",
    "src/models",
    "src/middleware",
    "src/config"
  ],
  "files": [
    { "path": ".env", "generator": "envGenerator" },
    { "path": "package.json", "generator": "packageJsonGenerator" },
    { "path": "src/config/db.js", "generator": "dbConfigGenerator" },
    { "path": "src/index.js", "generator": "appEntryGenerator" },
    { "path": ".gitignore", "generator": "gitignoreGenerator" }
  ],
  "dependencies": {
    "express": "^4.18.2",
    "mysql2": "^3.6.5",
    "dotenv": "^16.3.1"
  },
  "devDependencies": { "nodemon": "^3.0.2" },
  "scripts": { "start": "node src/index.js", "dev": "nodemon src/index.js" }
}

To add a new template: create a folder in src/templates/, add a manifest.json, restart Nexus. The loader handles the rest.


Import Merge System

When creating a project, you can optionally provide existing files to merge:

Via API (multipart):

POST /api/projects
Content-Type: multipart/form-data

name        = "NomadSync"
path        = "F:/dev/nomadsync"
template    = "basic-express"
importPath  = "F:/dev/old-project"   (optional — reads package.json + .env from disk)
packageJson = <file upload>           (optional — takes precedence over importPath)
envFile     = <file upload>           (optional — takes precedence over importPath)

Merge rules:

File Conflict Resolution
package.json deps Nexus template deps win — logged as warning in events
package.json scripts Your scripts preserved, template scripts added if missing
package.json type Your type field wins — CJS or ESM respected
.env DB keys Nexus provisioned values always win — annotated with comment
.env other keys Your keys preserved in a separate labeled section

npm install always runs regardless of whether node_modules already exists — ensures merged deps are always installed.


Credential System

Remote connection passwords are AES-256-CBC encrypted using a key derived from NEXUS_ENCRYPTION_KEY via crypto.scryptSync. The encrypted value is stored in connections.credential. The key never leaves nexus.env.

Local connection passwords are never stored in the registry — only the env var name (DB_PASSWORD) is stored as the credential reference. The actual password lives in the project .env on disk.


Filesystem Browser

GET /api/fs/browse?path=F:/dev returns a directory listing. Browsing is restricted to paths within NEXUS_FS_ROOT. Dotfiles are hidden by default (NEXUS_FS_SHOW_HIDDEN=true to show them). All paths are normalized to forward slashes before comparison to handle Windows path inconsistencies.


API Reference

Health

Method Route Description
GET /health Nexus server health — used by CLI for server detection

Templates

Method Route Description
GET /api/templates List all available templates

Projects

Method Route Description
POST /api/projects Provision a new project
GET /api/projects List all projects
GET /api/projects/events/recent Recent events across all projects
GET /api/projects/:id Single project detail
DELETE /api/projects/:id Delete project (drops schema + registry)
PUT /api/projects/:id/sync Regenerate .env from registry connections
GET /api/projects/:id/events Project audit log
GET /api/projects/:id/metrics Ping history + slow query data
GET /api/projects/:id/health On-demand live ping of all connections
GET /api/projects/:id/connections List connections
POST /api/projects/:id/connections Add remote connection
DELETE /api/projects/:id/connections/:cid Remove connection
PUT /api/connections/:id/ping-interval Update ping interval

Filesystem

Method Route Description
GET /api/fs/browse Browse filesystem — ?path=F:/dev

POST /api/projects — Request Body

{
  "name": "NomadSync",
  "path": "F:/dev/nomadsync",
  "template": "basic-express",
  "importPath": "F:/dev/old-project"
}

Or multipart/form-data with packageJson and envFile file fields.

POST /api/projects/:id/connections — Request Body

{
  "engine": "mysql",
  "label": "primary",
  "host": "your-remote-host.com",
  "port": 3306,
  "db_name": "myremotedb",
  "username": "dbuser",
  "password": "secret"
}

If db_name does not exist on the target MySQL server, Nexus creates it.

GET /api/projects/:id/metrics — Query Params

Param Default Description
limit 50 Max records to return (max 500)
type all connection_ping or slow_query_poll

CLI Usage

The CLI entry point is bin/nexus.js. It detects whether Nexus is running on port 7700 — if not, it spawns the server as a background process and waits for it to become ready before executing the command.

Commands

# Provision a new project — streams live events to terminal
node bin/nexus.js spam <name> <path>

# List all registered projects
node bin/nexus.js list

# Show event log for a project
node bin/nexus.js status <projectId>

Terminal Output Example

Provisioning NomadSync...

  [OK]  validateInput                  12ms
  [OK]  registerProject                8ms
  [OK]  provisionDatabase              43ms
  [OK]  scaffoldFilesystem             5ms
  [OK]  mergeUserImports               2ms
  [OK]  generateEnv                    3ms
  [OK]  generateProjectFiles           6ms
  [..]  npm_install_started            ...
  [OK]  npm_install_complete           8432ms
  [OK]  pipeline_complete

  Ready at F:/dev/nomadsync

The CLI polls GET /api/projects/:id/events every second and streams each new event as it appears.


Frontend Architecture

Frontend Tech Stack

Library Version Purpose
React 19 UI framework
Vite latest Build tool + dev server
Ant Design 6 Component library
Tailwind CSS 4 Utility styling
React Router DOM 6 Client-side routing
TanStack React Query 5 Server state + caching + polling
Zustand 4 UI state (theme, sidebar, provisioning IDs)
Axios 1.7 HTTP client
Recharts 2 Ping latency line chart
Day.js 1.11 Date formatting

Frontend Folder Structure

nexus-ui/src/
├── main.jsx                  ← QueryClient + RouterProvider
├── App.jsx                   ← ConfigProvider (Ant Design theme)
├── index.css                 ← @import "tailwindcss"
├── router/
│   └── index.jsx             ← All routes
├── layout/
│   ├── AppLayout.jsx         ← Persistent sidebar shell
│   └── Sidebar.jsx           ← Navigation + contextual project nav
├── pages/
│   ├── Overview/             ← Dashboard home
│   ├── Projects/
│   │   ├── index.jsx         ← Projects list
│   │   ├── NewProject.jsx    ← Multi-step provision form
│   │   ├── ProjectDetail.jsx ← Project detail with tabs
│   │   └── tabs/
│   │       ├── OverviewTab.jsx
│   │       ├── ConnectionsTab.jsx
│   │       ├── MetricsTab.jsx
│   │       └── EventsTab.jsx
│   └── Templates/            ← Template browser
├── components/
│   ├── filesystem/
│   │   └── FolderBrowser.jsx ← Modal filesystem navigator
│   └── shared/
│       ├── PageHeader.jsx
│       ├── StatusBadge.jsx
│       └── EmptyState.jsx
├── hooks/
│   ├── useProjects.js
│   ├── useEvents.js
│   ├── useConnections.js
│   └── useMetrics.js
├── services/
│   └── api.js                ← All HTTP calls — components never use axios directly
└── store/
    └── nexusStore.js         ← Zustand store

State Management

React Query handles all server state — projects, events, connections, metrics, templates. It provides automatic caching, background refetching, loading/error states, and polling.

Zustand handles UI state only:

{
  appTheme:             'light' | 'dark',  // from system preference
  sidebarCollapsed:     boolean,
  provisioningIds:      Set<number>,       // projects currently being provisioned
}

provisioningIds drives the polling behaviour. When createProject succeeds, the returned projectId is added to the set. The EventsTab polls every 1 second while the project ID is in the set. When pipeline_complete or pipeline_error appears in the events, the ID is removed and polling stops.

API Service Layer

All HTTP calls go through src/services/api.js. The response interceptor unwraps res.data on success — components receive data directly without .data access. On error the interceptor rejects with the server error body.

Vite proxies /api and /health to http://localhost:7700 in development — no CORS configuration needed.

Pages

Overview (/) — stats bar (total, active, errors, provisioning), recent events table across all projects, connection health panel with status dots per project.

Projects (/projects) — stats bar, full projects table with status, template, path, created date. Actions: view, sync .env, delete.

New Project (/projects/new) — three-step form. Step 1: name, path (type or browse), template. Step 2: optional imports (importPath, package.json upload, .env upload). Step 3: confirmation summary before provisioning.

Project Detail (/projects/:id) — quick info bar (path, template, created), four tabs:

  • Overview tab — project details, metric summary cards (avg latency, uptime, slow query count, total pings), latest ping reading, connections summary grid
  • Connections tab — stats bar, connection cards with engine badge, host:port/db, status dot, ping interval; add connection drawer; remove connection
  • Metrics tab — terminal-style console with traffic light buttons; ping latency line chart (Recharts); slow query analysis table in monospace grid with avg_ms, max_ms, exec_count, no_index flag; limit selector
  • Events tab — terminal-style console; events in monospace grid (status symbol, type, duration/error, timestamp); live polling spinner during provisioning

Templates (/templates) — template cards showing name, version, engines, dependencies, scaffold structure, load time.


Project Scaffold Output

After running nexus spam NomadSync F:/dev/nomadsync:

nomadsync/
├── .env                      ← DB credentials + NODE_ENV + PORT
├── .gitignore                ← node_modules, .env, dist, logs
├── package.json              ← merged deps + scripts
└── src/
    ├── index.js              ← Express entry + /health endpoint
    ├── config/
    │   └── db.js             ← mysql2 pool from .env
    ├── routes/               ← empty — yours to fill
    ├── controllers/          ← empty — yours to fill
    ├── models/               ← empty — yours to fill
    ├── middleware/            ← empty — yours to fill
    └── config/

Run npm run dev inside the scaffolded project — the server starts and GET /health confirms the DB connection is live before you write a single line of business logic.


Version Roadmap

V1 — Current

  • MySQL provisioning + scoped users
  • Project scaffolding (3 generated files + folder structure)
  • Import merge (package.json + .env)
  • Connection-level monitoring (ping latency + uptime)
  • Slow query analysis via performance_schema
  • Local + remote connections
  • REST API + CLI + React dashboard

V2 — Planned

  • Redis connection support + monitoring
  • SSE streaming for live log tailing (replaces polling)
  • Visual folder structure editor in the dashboard
  • Webhook support for continuous metrics updates
  • Remote SSH tunnel for production DB monitoring
  • Dashboard observability charts with historical trends

V3 — Future

  • Custom folder structure templates
  • Query builder scaffolding (select().where() pattern)
  • PostgreSQL support
  • Multi-engine templates (MySQL + Redis in one project)

Design Decisions

Why the saga pattern for the pipeline? Each provisioning step has a side effect (DB creation, directory creation, file writing). If any step fails mid-way, you end up with orphaned schemas, half-built directories, and stale registry rows. Compensation handlers undo exactly what was done — the system either fully succeeds or fully cleans up.

Why separate events and metrics tables? Events are lifecycle records — one per pipeline step, a handful per project lifetime. Metrics are high-frequency time-series — one row every 30 seconds per connection, indefinitely. Mixing them would make both harder to query efficiently.

Why scoped MySQL users instead of root credentials in .env? A leaked .env from one project should not give access to every database on your MySQL instance. Each project user has SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER on its own schema only. Root credentials never leave nexus.env.

Why credential_type instead of just storing the password? Local provisioned connections should not have their passwords stored in the registry at all — they live on disk in .env. Remote connections need encrypted storage because there is no .env to read from. The credential_type column makes the resolution logic explicit and auditable.

Why React Query instead of Redux or plain useState? The dashboard is almost entirely server state — projects, connections, metrics, events. React Query handles caching, background refetch, stale-while-revalidate, and polling with a fraction of the boilerplate. The polling behaviour for the events log (refetchInterval: isProvisioning ? 1000 : false) is one line.

Why no TypeScript? V1 is a personal development tool built for speed. TypeScript would add overhead without meaningful benefit at this stage. The codebase is structured to make V2 adoption straightforward — discrete modules, clear interfaces, no global mutable state.

Why Tailwind v4 + Ant Design? Ant Design owns components. Tailwind owns layout, spacing, and composition. Tailwind v4 requires no config file — just @import "tailwindcss" in the CSS entry. corePlugins: { preflight: false } prevents Tailwind's base reset from conflicting with Ant Design's component styles.

Contributors

Emmanuel-Tech-Dev

Issues