SPDX-License-Identifier: Apache-2.0       
Copyright (c) 2020 Intel Corporation

Introduction
Overview
Deployment
Use case
Post-deployment Verification
SRIOV Network Device Plugin and SRIOV CNI plugin
Known issues
Requirements
References

Introduction

Common Integration Repository(CIR) aims to supply easy orchestration and application deployment in cloud network and edge environment on Intel architecture platform. It provides tarball package of ansible playbooks for automating installation and configuration of ingredients from single source which can be deployed on supported Intel BKC platforms. With CIR, users can setup network cluster environment to verify application deployment easily, and internal ingredients can verify the system robustness and compatibility with easy installtion and scaling management.

Related basic technology includes but not limited to: Docker Container, Kubernetes, Ansible, BMRA, OpenNESS, OVC, OpenVINO, VCMTS.

Overview

CIR integrates well-known field network cluster orchestration framework of both cloud core and multi-access edge, and deploys typical cloud use case apllicaitons on the cluster with variety configuration to demonstrate Intel advanced network techonologies.

It takes usage of ansible playbooks and roles to supply customer an easy way to setup environment on Intel BKC platforms.

BMRA deployment is used to install container based bare metal NFV platform;
OpenNESS deployment is used to setup edge orchestration for edge service in multiple environment;
Use case deployment is used to deploy applications and services on cloud network or edge cluster:

Figure 1 - Common Integration Main Components Overview

Ansible is an agentless configuration management tool that uses playbooks to perform actions on remote machines. CIR provides a set of playbooks to setup cloud native or edge orchestration and deploy related user applications on them. Following chart shows the high level ansible playbook topology:

Figure 2 - Common Integration High level Ansible playbooks

BMRA: Build a bare metal container based cloud cluster setup on Intel platforms with accelerators. Multiple profiles are supplied for different users. Users can select profiles to config different advanced network techonology and device plugins, ex. SRIOV, QAT, DPDK, VPP etc. This is the basis for application deployment on cloud. Following is the architecture topology:

Figure 3 - BMRA Architecture Topology
OpenNESS: Setup edge cluster for edge services deployment and network functions across diverse network platform. OpenNESS supports both on-premises and network modes. CIR deploys with network mode, which leverages Kubernetes based cloud-native container orchestration stack to deploy edge cloud that can host applications, services and network functions. This is the basis for application deployment on edge cluster. Following is the OpenNESS network deployment topology:

Figure 4 - OpenNESS Network Deployment Topology
Use Case: Some high profile cloud usage models are integrated here to demonstrate Common Integration capability for network orchestration and easy scale and manage the cluster with K8S, including CDN, ADI, SmartCity, VCMTS.

These components can be combined with variety recipes according to customer focus:

User can setup bare matel NFV cluster with BMRA[1] only.
User can setup pure edge sub system with Openness[2] only.
User can deploy sample Use case[3] on cloud cluster with BMRA[1] .
User can deploy sample Use case[3] on edge system with Openness[2]
User can setup cloud+edge orchestration and deploy use case on it to emulate reality usage with BMRA[1] + Openness[2] + Use case[3]

Typical repices will be shown in following Use Case in details.

Deployment

Ansible Host Prerequisites

CIR uses python 3 as default ansible interpreter. Please change it to python2.7 on CentOS7.6 manually with step1 in below:

Edit and remove or comment "ansible_python_interpreter" in example/inventory.ini.
```
[all:vars]
ansible_python_interpreter=/usr/bin/python3
```

Prepare ansible host environment with following commands:

# sudo yum install epel-release
# sudo yum install ansible
# easy_install pip
# pip2 install jinja2 –upgrade
# sudo yum install python36 –y
# pip install netaddr

Enable passwordless login between all nodes in the cluster.
```
# ssh-keygen
# ssh-copy-id root@node-ip-address
```

Get Playbooks and Config Variables

Get CIR repo to get ansible playbooks:

 # git clone https://github.com/intel/common-integration-repository.git
 # cd common_integration_repository

Copy example inventory file to the CIR home location:
```
 # cp examples/<usecase>/*.ini ./
```
Edit the inventory.ini to reflect the requirement with correct machines ip.

Copy group_vars and host_vars directories to the CIR home location:

 # cp -r examples/<usecase>/<profile>/group_vars examples/<usecase>/host_vars .

Update variables in group_vars/all/all.yml and usecase.yml and host_vars/nodeX.yml (Detailed info will be touched in below deployment details)

 # vi group_vars/all/all.yml
 # vi host_vars/node1.yml

CIR leverages BMRA method to configure intel capabilities or device plugins with variables which are defined in yaml files. Please refer to Container BMRA Architecture for details.

Additional key CIR variables are defined in below:

Variable	Choice,Defaults	Description
openness_enabled	Boolean, default “false”	Enable or Disable OpennNESS setup.If "true", OpenNESS cluster will be deployed.
cdn_transcode_enabled	Boolean, default “false”	Enable or Disable Use case CDN deployment. If "true", CDN will be deployed with correct inventories config.
ad_insertion_enabled	Boolean, default “false”	Enable or Disable Use case ADI deployment. If "true", ADI will be deployed with correct inventories config.
smtc_enabled	Boolean, default “false”	Enable or Disable Use case Smart city deployment. If "true", Smart city will be deployed with correct inventories config.
vcmts_enabled	Boolean, default “false”	Enable or Disable Use case VCMTS deployment. If "true", VCMTS will be deployed with correct inventories config.

Update and initialize git submodule. This git repository has nested submodules to support kubespray installation and OpenNESS if enabled.
```
 # git submodule update --init --recursive
```

BMRA Deployment

Edit inventory.ini to reflect correct master and nodes ip.

Execute ansible playbooks: BMRA includes 3 parts in the setup: Infrastructure setup, K8S setup and Intel capability setup. It supports basic, remote_fp, on_prem and full_nfs profiles. User can run it in one kick:

 # ansible-playbook -i inventory.ini playbooks/cir.yml --extra-vars "profile=<profile>"
 Or
 # ansible-playbook -i inventory.ini playbooks/orchestration/bmra/playbooks/<profile>.yml

Or User can run them one by one:

 # ansible-playbook -i inventory.ini playbooks/orchestration/bmra/playbooks/infra/<profile>.yml
 # ansible-playbook -i inventory.ini playbooks/orchestration/bmra/playbooks/k8s/k8s.yml
 # ansible-playbook -i inventory.ini playbooks/orchestration/bmra/playbooks/intel/<profile>.yml
 # ansible-playbook -i inventory.ini playbooks/usecase/usecase.yml

OpenNESS Deployment

Edit openness_inventory.ini to reflect correct controller and edge nodes ip.
Enable openness_enabled and openness_current_enabled or openness_next_enabled in group_vars/all/orchestration.yml.

Execute ansible playbook:

 # ansible-playbook -i openness_inventory.ini playbooks/cir.yml --extra-vars "profile=<profile>"

You can also run independently commands as following:

 # ansible-playbook -i openness_inventory.ini playbooks/orchestration/orchestration.yml --extra-vars "profile=<profile>"
 # ansible-playbook -i openness_inventory.ini playbooks/usecase/usecase.yml

Use Case Deployment

Edit ***_inventory.ini to reflect correct master and nodes ip.
Enable selected use case variable in group_vars/usecase.yml.

Execute ansible playbook: User can complete the deploy with both BMRA and use case in one command:

 # ansible-playbook -i ***_inventory.ini playbooks/cir.yml --extra-vars "profile=<profile>"

You can also run independently commands as following:

 # ansible-playbook -i ***_inventory.ini playbooks/orchestration/orchestration.yml --extra-vars "profile=<profile>"
 # ansible-playbook -i ***_inventory.ini playbooks/usecase/usecase.yml

  Notes: `***` above can be replaced with selected Use case shortname, ex. "adi", "cdn" or "smtc" or "vcmts"

Use Case

CDN Deployment on BMRA Basic Infrastructure

CDN(Content Delivery Transcode) network deployment is high profile visual cloud usage. It supplies live streaming and VOD service to client based on media multi-transcode in network. Please refer to CDN Source code for details. CIR provides playbooks to deploy CDN based on Flannel CNI with BMRA:

Figure 5 - CDN on BMRA Flannel Deployment

Deployment steps are below:

copy cdn_inventory.ini to CIR folder and edit it with correct ips:
```
cp examples/cdn/cdn_inventory.ini ./
vi cdn_inventory.ini 
```

copy group_vars and host_vars to CIR folder, update proxy and node configation file name:

cp examples/cdn/basic/group_vars examples/cdn/basic/host_vars -rf ./
vi group_vars/all/all.yml
  - http_proxy: "http://proxy-example.com"
  - https_proxy: "http://proxy-example.com"
  - additional_no_proxy: "127.0.0.1,all master and minion's ip list which is seperated with comma" 
cp host_vars/node1.yml host_vars/node_name_in_inventory.yml (run multiple times if you have multiple nodes defined)

Setup CDN based on BMRA flannel CNI:

ansible-playbook -i cdn_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

ADI Deployment on BMRA Basic Infrastructure

AD insertion supplies advertise add-in to video content at VOD or streaming based on video content analysis. It contains CDN service, msedia analytics and AD insertion. Please refer to ADI Source code for details. Similar as CDN, CIR provides playbooks to deploy ADI based on Flannel CNI with BMRA:

Figure 6 - ADI on BMRA Flannel Deployment

Deployment steps are below:

copy adi_inventory.ini to CIR folder and edit it with correct ips:
```
cp examples/adi/adi_inventory.ini ./
vi adi_inventory.ini 
```

copy group_vars and host_vars to CIR folder, update proxy and node configation file name:

cp examples/adi/basic/group_vars examples/adi/basic/host_vars -rf ./
vi group_vars/all/all.yml
  - http_proxy: "http://proxy-example.com"
  - https_proxy: "http://proxy-example.com"
  - additional_no_proxy: "127.0.0.1,all master and minion's ip list which is seperated with comma" 
cp host_vars/node1.yml host_vars/node_name_in_inventory.yml (run multiple times if you have multiple nodes defined)

Setup ADI based on BMRA flannel CNI:

ansible-playbook -i adi_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

ADI Deployment on OpenNESS with VCAC-A

AD insertion supplies advertise add-in to video content at VOD or streaming based on video content analysis. It contains CDN service, msedia analytics and AD insertion. Please refer to ADI Source code for details. ADI also supports to deploy with VCAC-A accelerator, CIR provides playbooks to deploy ADI based on VCAC-A with OpenNESS:

Figure 7 - ADI on OpenNESS Deployment

Deployment steps are below:

copy openness_adi_inventory.ini to CIR folder and edit it with correct ips:

cp examples/openness/openness_adi_inventory.ini ./
vi openness_adi_inventory.ini

copy group_vars and host_vars to CIR folder, update proxy, VCAC-A, OpenNESS option and node configation file name:

cp examples/openness/basic/group_vars examples/openness/basic/host_vars -rf ./
vi group_vars/all/all.yml
  - http_proxy: "http://proxy-example.com"
  - https_proxy: "http://proxy-example.com"
  - additional_no_proxy: "127.0.0.1,all master and minion's ip list which is seperated with comma" 
  - vcaca_enabled: true
vi group_vars/all/orchestration.yml
  - openness_enabled: true
  - openness_next_enabled: true
vi group_vars/all/usecase.yml
  - ad_insertion_enabled: true
  - adi_platform: VCAC-A

Setup ADI based on OpenNESS:

ansible-playbook -i adi_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

Smart City Deployment on BMRA Basic Infrastructure

Smart city is another high profile cloud usage model. SMTC implements aspects of smart city sensing, analytics and management feature. Please refer to Smart City Source code for details. CIR provides 2 kinds of deployment for SMTC, on cloud cluster or boto cloud and edge clusters. This seciton depicts how to deploy SMTC based on Flannel CNI with BMRA on cloud cluster:

Figure 8 - SMTC on BMRA Flannel Deployment

Deployment steps are below:

copy smtc_inventory.ini to CIR folder and edit it with correct ips:
```
cp examples/smtc/smtc_inventory.ini ./
vi smtc_inventory.ini 
```

copy group_vars and host_vars to CIR folder, update proxy and node configation file name:

cp examples/smtc/basic/group_vars examples/smtc/basic/host_vars -rf ./
vi group_vars/all/all.yml
  - http_proxy: "http://proxy-example.com"
  - https_proxy: "http://proxy-example.com"
  - additional_no_proxy: "127.0.0.1,all master and minion's ip list which is seperated with comma" 
cp host_vars/node1.yml host_vars/node_name_in_inventory.yml (run multiple times if you have multiple nodes defined)

Setup SMTC based on BMRA flannel CNI:

ansible-playbook -i smtc_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

Smart City Deployment on Cloud And Edge

Smart City is a sample applications that is built on top of the OpenVINO & Open Visual Cloud software stacks for media processing and analytics. The application is deployed across multiple regional offices (OpenNESS edge nodes). Each office is an aggregation point of multiple IP cameras (simulated) with their analytics. The media processing and analytics workloads are running on the OpenNESS edge nodes for latency consideration.

The full pipeline of the Smart City sample application on OpenNESS is distributed across three regions:

Client-side Cameras Simulator
OpenNESS Cluster
Smart City Cloud Cluster

The Smart City setup with OpenNESS should typically deployed as shown in this Figure. The drawing depicts 2 offices according to the sample playbooks, but there is no limitation to the number of offices.

Figure 9 - Smart City Setup with OpenNESS

CIR setups 2 central offices in one edge node, and simulates 2 IP cameras in cloud cluster with OpenNESS 20.03. External camera dataplane is supported with next OpenNESS. Please see the sample deployment topology as following:

Figure 10 - SMTC on BMRA and Openness Deployment

Deployment steps are below:

copy openness_smtc_int_camera_inventory.ini to CIR folder and edit it with correct ips and hostnames:

cp examples/openness/openness_smtc_int_camera_inventory.ini ./
vi openness_smtc_int_camera_inventory.ini

copy group_vars and host_vars to CIR folder, update proxy, SMTC, OpenNESS options and node configation file name:

cp examples/openness/basic/group_vars examples/openness/basic/host_vars -rf ./
vi group_vars/all/all.yml
  - http_proxy: "http://proxy-example.com"
  - https_proxy: "http://proxy-example.com"
  - additional_no_proxy: "127.0.0.1,all master and minion's ip list which is seperated with comma" 
  - vcaca_enabled: true (if you want to deploy SMTC with VCAC-A)
vi group_vars/all/orchestration.yml
  - openness_enabled: true
  - openness_next_enabled: true
vi group_vars/all/usecase.yml
  - smtc_enabled: true
  - smtc_platform: VCAC-A (if you want to deploy SMTC with VCAC-A)
cp host_vars/node1.yml host_vars/node_name_in_inventory.yml (run multiple times if you have multiple nodes defined)

Setup SMTC based on BMRA and Openness:

ansible-playbook -i openness_smtc_int_camera_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

Notes: Considering simplification, All use cases will be depicted with 1 master and 1 minion. Users can easily scale up the node numbers with inventory files.

VCMTS Deployment on BMRA Remote Forwarding Platform Infrastructure

VCMTS is virtual cable modem termination system, which contains DOCSIS MAC functionality and may be deployed at the Cable Service Provider Head-End. Please refer to vcmts for details. VCMTS includes DPDK, QAT, sriov features and CIR provides playbooks to deploy them based on BMRA remote_fp profile automatically. CIR supports 2 deployment meothods for VCMTS, baremetal and k8s. User can config them in group_vars/all/usecase.yml easily:

Deployment steps are below:

copy vcmts_inventory.ini to CIR folder and edit it with correct ips:

cp examples/vcmts/vcmts_inventory.ini ./
vi vcmts_inventory.ini

copy group_vars and host_vars to CIR folder, update proxy, vcmts optins and node configation file name::

vi group_vars/all/all.yml
  - http_proxy: "http://proxy-example.com"
  - https_proxy: "http://proxy-example.com"
  - additional_no_proxy: "127.0.0.1,all master and minion's ip list which is seperated with comma" 
vi group_vars/all/usecase.yml
  - vcmts_enabled: true
  - vcmts_env_type: baremetal (or kubernetes)
  - vcmts_domain: example.com (vcmts hosts domain name)
cp host_vars/node1.yml host_vars/node_name_in_inventory.yml (run multiple times if you have multiple nodes defined)
update isolcpu option in nodex.yml
  - isolcpus: "2-21,24-43,46-65,68-87"
  - cpu_count: 88

Setup vcmts :

ansible-playbook -i vcmts_inventory.ini playbooks/cir.yml --extra-vars "profile=remote_fp"

Post-deployment Verification

Check BMRA Cluster

This section shows how to verify all the components deployed by the scripts. All following commands are executed on master:

Check the post-deployment node status of master & minion, which should match the inventory configuration:
```
 # kubectl get nodes -o wide
```

Check pod status of master & minion. All Pods should be in Running or Completed status:

 # kubectl get pods --all-namespace

Following pods should be listed (BMRA Flannel as example)

NAMESPACE	NAME
kube-system	cmk-bm7jn
kube-system	cmk-webhook-cdccd9854-872wb
kube-system	coredns-58687784f9-tkvd8
kube-system	coredns-58687784f9-x4k8n
kube-system	dns-autoscaler-79599df498-gjjd9
kube-system	kube-apiserver-master1
kube-system	kube-controller-manager-master1
kube-system	kube-flannel-brj7b
kube-system	kube-flannel-vhc8h
kube-system	kube-multus-ds-amd64-9hn5g
kube-system	kube-multus-ds-amd64-fhp6b
kube-system	kube-proxy-g75gq
kube-system	kube-proxy-l2ptb
kube-system	kube-scheduler-master1
kube-system	kubernetes-dashboard-556b9ff8f8-bh94j
kube-system	nfd-node-feature-discovery-6fb4t
kube-system	nfd-node-feature-discovery-pbmqm
kube-system	nginx-8664849564-tmh4j
kube-system	nginx-proxy-node1
kube-system	nodelocaldns-228bq
kube-system	nodelocaldns-5fjnd
kube-system	registry-c4xkf
kube-system	registry-proxy-dkllt
kube-system	tiller-deploy-58f6ff6c77-xnz2q