Complete guide and automation for deploying OpenShift on AWS with multiple network interfaces per pod using Multus CNI and macvlan.

• Overview
• Architecture
• Prerequisites
• Quick Start
• Detailed Documentation
• Configuration Files
• Automation Scripts
• Examples
• Troubleshooting
• Cleanup

This project provides a complete solution for deploying an OpenShift cluster on AWS with:

• Primary CNI: OVN-Kubernetes (default OpenShift networking)
• Secondary CNI: Multus with macvlan plugin
• Cluster Size: 3 control plane + 3 worker nodes
• Installation Method: IPI (Installer-Provisioned Infrastructure)
• Cloud Platform: AWS

✅ Fully automated OpenShift installation on AWS ✅ Multus CNI configured for secondary networks ✅ macvlan network attachment definitions ✅ Secondary ENI support for dedicated network interfaces ✅ Whereabouts IPAM for cluster-wide IP coordination ✅ Example pods/VMs with multiple network interfaces ✅ Validation scripts to verify configuration ✅ Comprehensive documentation

┌─────────────────────────────────────────────────────────────┐
│                         AWS Cloud                            │
│  ┌───────────────────────────────────────────────────────┐  │
│  │              VPC (Auto-created by IPI)                │  │
│  │                                                        │  │
│  │  ┌──────────────────┐      ┌──────────────────┐     │  │
│  │  │  Control Plane   │      │  Worker Nodes    │     │  │
│  │  │  - Master 1      │      │  - Worker 1      │     │  │
│  │  │  - Master 2      │      │  - Worker 2      │     │  │
│  │  │  - Master 3      │      │  - Worker 3      │     │  │
│  │  └──────────────────┘      └──────────────────┘     │  │
│  │                                                        │  │
│  │  ┌────────────────────────────────────────────────┐  │  │
│  │  │         Networking Stack                       │  │  │
│  │  │  ┌──────────────┐  ┌──────────────────────┐  │  │  │
│  │  │  │ OVN-K (eth0) │  │ Multus + macvlan     │  │  │  │
│  │  │  │ Primary CNI  │  │ (net1) Secondary CNI │  │  │  │
│  │  │  └──────────────┘  └──────────────────────┘  │  │  │
│  │  └────────────────────────────────────────────────┘  │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Pod with Multiple Interfaces
├── eth0 (OVN-Kubernetes)
│   ├── Cluster IP: 10.128.x.x
│   ├── Service networking
│   ├── Pod-to-pod communication
│   └── Internet egress
│
└── net1 (macvlan)
    ├── Secondary IP: 192.168.1.x
    ├── Direct L2 connectivity
    ├── Storage network access
    └── Management network access

• AWS CLI (v2.x): Configured with appropriate credentials
• OpenShift Installer (4.14+): Download from Red Hat
• oc CLI: OpenShift command-line tool
• kubectl: Kubernetes command-line tool
• jq: JSON processor for scripts
• git: Version control

1. AWS Account with appropriate permissions
2. Service Quotas sufficient for:
   • 6 EC2 instances (3 m5.xlarge control plane + 3 m5.xlarge workers)
   • VPC resources (subnets, route tables, NAT gateways)
   • Elastic Load Balancers (2)
   • Elastic IPs (3+)
3. DNS Domain for cluster (e.g., example.com)
4. Red Hat Pull Secret from https://console.redhat.com/openshift/install/pull-secret

Your AWS user/role needs permissions for:

• EC2 (full access)
• VPC (full access)
• ELB/ALB (full access)
• Route53 (full access)
• S3 (read/write for bootstrap)

This project supports two deployment approaches for Multus networking:

Uses the primary network interface for Multus secondary networking.

Pros:

• ✅ Simpler setup, no additional AWS resources
• ✅ No additional costs
• ✅ Suitable for testing and light workloads

Cons:

• ⚠️ Shares bandwidth with primary network
• ⚠️ No physical network isolation

Quick Start:

./scripts/setup-multus-for-existing-cluster.sh my-cluster

Attaches dedicated secondary ENIs (eth1) to worker nodes for Multus networking.

Pros:

• ✅ Dedicated network bandwidth
• ✅ Physical network isolation
• ✅ Better performance for network-intensive workloads
• ✅ Hardware-level network separation

Cons:

• ⚠️ Additional AWS ENI costs (~$0.36/day for 3 workers)
• ⚠️ More complex setup (requires node reboots)

Quick Start:

./scripts/setup-multus-with-secondary-eni.sh my-cluster 192.168.1.0/24

See: Secondary ENI Setup Guide for detailed instructions.

• IAM (limited for instance profiles)

See docs/01-prerequisites.md for detailed IAM policy.

# Clone this repository
git clone <repository-url>
cd openshift-aws-multus

# Make scripts executable
chmod +x scripts/*.sh

# Set required environment variables
export AWS_REGION="us-east-1"
export CLUSTER_NAME="ocp-multus"
export BASE_DOMAIN="example.com"
export PULL_SECRET_FILE="$HOME/pull-secret.txt"

# Optional: Customize cluster size
export CONTROL_PLANE_REPLICAS=3
export WORKER_REPLICAS=3
export INSTANCE_TYPE="m5.xlarge"

# Run the installation script
./scripts/install-openshift.sh

# This will:
# - Download OpenShift installer
# - Generate install-config.yaml
# - Create the cluster (takes 30-45 minutes)
# - Save kubeconfig to ./auth/kubeconfig

# Deploy Multus network configurations
./scripts/configure-multus.sh

# This will:
# - Verify Multus is installed
# - Create NetworkAttachmentDefinition for macvlan
# - Deploy test pods

# Run validation script
./scripts/validate-networking.sh

# This will:
# - Check pod interfaces
# - Test connectivity
# - Verify routing
# - Generate validation report

# Use example manifests
kubectl apply -f examples/deployment-with-multus.yaml

# Or annotate existing pods
kubectl annotate pod <pod-name> k8s.v1.cni.cncf.io/networks=macvlan-conf

Comprehensive guides for each phase:

1. Prerequisites - AWS setup, IAM permissions, service quotas
2. Installation - OpenShift IPI installation step-by-step
3. Multus Configuration - Multus and macvlan setup
4. Secondary ENI Setup - Dedicated network interfaces for Multus
5. Validation - Testing and verification procedures
6. Troubleshooting - Common issues and solutions
7. AWS Networking Constraints - AWS-specific networking details

All configuration files are in the config/ directory:

# Copy template
cp config/install-config.yaml.template install-config.yaml

# Edit with your values
vi install-config.yaml

# Key fields to update:
# - baseDomain: your-domain.com
# - metadata.name: cluster-name
# - platform.aws.region: us-east-1
# - pullSecret: '<your-pull-secret>'
# - sshKey: '<your-ssh-public-key>'

All scripts are in the scripts/ directory:

setup-multus-for-existing-cluster.sh - Configure Multus on existing cluster (uses eth0)

./scripts/setup-multus-for-existing-cluster.sh <cluster-name> [secondary-network-cidr] [network-name]

# Example
./scripts/setup-multus-for-existing-cluster.sh my-cluster 10.200.0.0/16 vm-bridge-network

setup-multus-with-secondary-eni.sh - Complete setup with dedicated secondary ENIs

./scripts/setup-multus-with-secondary-eni.sh <cluster-name> [secondary-subnet-cidr] [network-name] [use-whereabouts]

# Example
./scripts/setup-multus-with-secondary-eni.sh my-cluster 192.168.1.0/24 vm-macvlan-eth1 true

attach-secondary-enis.sh - Attach secondary ENIs to worker nodes

./scripts/attach-secondary-enis.sh <cluster-name> [secondary-subnet-cidr]

# Example
./scripts/attach-secondary-enis.sh my-cluster 192.168.1.0/24

install-openshift-cluster.sh - Install new OpenShift cluster on AWS

destroy-cluster.sh - Destroy OpenShift cluster and cleanup resources

Example manifests are in the examples/ directory:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: test-container
    image: registry.access.redhat.com/ubi8/ubi:latest
    command: ["sleep", "infinity"]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-network-app
spec:
  replicas: 3
  template:
    metadata:
      annotations:
        k8s.v1.cni.cncf.io/networks: macvlan-conf
    spec:
      containers:
      - name: app
        image: nginx:latest

See examples/ for more complete examples including:

• Deployments with Multus
• StatefulSets with Multus
• DaemonSets with Multus
• Multiple network attachments

Symptoms: Pod only has eth0, no net1

Solutions:

1. Check NetworkAttachmentDefinition exists:

   kubectl get network-attachment-definitions

2. Verify annotation is correct:

   kubectl get pod <pod-name> -o yaml | grep annotations -A 5

3. Check Multus logs:

   kubectl logs -n openshift-multus -l app=multus

Symptoms: net1 interface exists but no connectivity

Solutions:

1. Verify IP assignment:

   kubectl exec <pod-name> -- ip addr show net1

2. Check routing table:

   kubectl exec <pod-name> -- ip route

3. Verify AWS security groups allow traffic

4. Check if source/destination checks are disabled on EC2 instances

Symptoms: OpenShift installer errors

Solutions:

1. Check AWS service quotas
2. Verify IAM permissions
3. Review installer logs in .openshift_install.log
4. Ensure DNS domain is properly configured

See docs/05-troubleshooting.md for comprehensive troubleshooting guide.

# Using cleanup script (recommended)
./scripts/cleanup.sh --cluster-name ocp-multus

# Manual cleanup
cd <installation-directory>
openshift-install destroy cluster --dir .

# Check for remaining resources
aws ec2 describe-instances --filters "Name=tag:kubernetes.io/cluster/ocp-multus,Values=owned"
aws ec2 describe-vpcs --filters "Name=tag:Name,Values=ocp-multus*"
aws elb describe-load-balancers | grep ocp-multus

After deployment, verify:

• OpenShift cluster is accessible via oc CLI
• All nodes are in Ready state
• OVN-Kubernetes pods are running
• Multus daemonset is running on all nodes
• NetworkAttachmentDefinition is created
• Test pod has multiple interfaces (eth0 + net1)
• Primary interface (eth0) has cluster IP
• Secondary interface (net1) has macvlan IP
• Pod can reach cluster services via eth0
• Pod can communicate on secondary network via net1
• DNS resolution works
• No routing conflicts

# Check cluster status
oc get nodes
oc get clusterversion
oc get co  # cluster operators

# Check networking
oc get network.config/cluster -o yaml
oc get pods -n openshift-ovn-kubernetes
oc get pods -n openshift-multus

# Check network attachments
kubectl get network-attachment-definitions -A
kubectl describe network-attachment-definition macvlan-conf

# Debug pod networking
kubectl exec <pod-name> -- ip addr
kubectl exec <pod-name> -- ip route
kubectl exec <pod-name> -- ping -c 3 <target-ip>

# View logs
oc logs -n openshift-multus -l app=multus
oc logs -n openshift-ovn-kubernetes -l app=ovnkube-node

• OpenShift Documentation
• Multus CNI Documentation
• OVN-Kubernetes Documentation
• AWS OpenShift Installation Guide
• macvlan CNI Plugin

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Test thoroughly
5. Submit a pull request

This project is provided as-is for educational and deployment purposes.

For issues and questions:

1. Check Troubleshooting Guide
2. Review OpenShift Documentation
3. Open an issue in this repository