GithubHelp home page GithubHelp logo

azure-samples / virtual-kubelet-aci-burst Goto Github PK

View Code? Open in Web Editor NEW
15.0 24.0 19.0 252.81 MB

Run image recognition from Azure Container Service (AKS) and burst into Azure Container Instances (ACI) with the Virtual-Kubelet.

License: MIT License

Makefile 0.09% Python 5.41% CSS 23.09% JavaScript 3.31% HTML 50.54% PowerShell 0.04% Shell 0.80% Dockerfile 0.32% SCSS 16.31% Mustache 0.11%

virtual-kubelet-aci-burst's Introduction

languages page_type description products
python
sample
Run a facial recognition demo across your AKS cluster and use ACI and the ACI Connector to burst into for on-demand compute.
azure
azure-container-instances

Bursting from AKS to ACI with the Virtual Kubelet

Description: Run a facial recognition demo across your AKS cluster and use ACI and the ACI Connector to burst into for on-demand compute.

Contact [email protected] if you need help!

Steps to deploy

Part 1 Prerequisites

In order to run this demo, you will need the following:

Part 2 Setup

Replace <myResourceGroup> with your expected and run following command to create resource group. Then remember the created resource group name.

$ az group create --name <myResourceGroup> --location eastus

Note: The AKS extending to ACI only support in one location. So far AKS and ACI are both deployed to East US and West Europe. So please create your resource group in East US or West Europe only.

Replace <myResourceGroup>,<myK8sCluster> with your expected and run following command to create the AKS. Then remember the AKS name.

$ az aks create --resource-group <myResourceGroup> --name <myK8sCluster> --node-count 1 --generate-ssh-keys

Replace <myResourceGroup>, <myK8sCluster> with your created in previous steps and run following command to set the AKS as your current connected cluster.

$ az aks get-credentials --resource-group <myResourceGroup> --name <myK8sCluster>

Make sure you're connected.

$ kubectl get nodes

Make sure you have helm installed and initialize this.

$ helm init

Replace <your email> with yours and run to install kube-lego w/ helm for certs.

$ helm install stable/kube-lego --name kube-lego --namespace kube-system --set config.LEGO_EMAIL=<your email>,config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory

Install ingress controller w/ helm.

$ helm install stable/nginx-ingress --name ingress --namespace kube-system

Get the Public IP of the ingress controller. It may take some times for the IP assigned to the services

$ kubectl get services --namespace kube-system --watch

When the IP assigned, remember the IP address and press Ctrl+Q to exit

Clone this repo

$ git clone https://github.com/rbitia/aci-demos.git

Change the folder to the root of the source code

$ cd aci-demo

Replace <myResourceGroup>, <IP Address> with your created in previous steps. Replace <appName> with your expected and run following shell script to bind FQDN to your IP. Remember the return FQDN name. You will use it to update your configuration file.

$ chmod u+x assignFQDNtoIP.sh
$ ./assignFQDNtoIP.sh -g <myResourceGroup> -d <appName> -i <IP Address>

Edit the values.yaml file and replace all <host name> with the FQDN name in pervious step.

$ vim ./charts/fr-demo/values.yaml 

Start at the top of the aci-demos directory and deploy the Facial Recognition application that consists of a frontend, a backend, and a set of image recognizers.

$ helm install charts/fr-demo --name demo

Checkout the UI that's generated in the output and see the pictures start to get processed The rate will be super slow because we have a 1 node AKS cluster running 1 worker pod.

Deploy the ACI connector : Replace <myResourceGroupmy>, <myK8sCluster> with yours in previous steps and run following command

az aks install-connector --resource-group <myResourceGroup> --name <myK8sCluster> --connector-name myaciconnector

The connector has been deployed and with a kubectl get nodes you can see that the ACI Connector is a new node in your cluster. Now scale up the image recognizer to 10 using the following command

$ kubectl scale deploy demo-fr-ir-aci --replicas 10

Though we are using kubectl, the ACI Connector is dispatching pods to Azure Container Instances transparently, via the ACI connector node. This virtual node has unlimited capacity and a per-second billing model, making it perfect for burst compute scenarios like this one. If we wait a minute or so for the ACI containers to warm up, we should see image recognizer throughput increase dramatically.

Check out the dashboard to see throughput it dramatically increase...

Here we can see throughput really beginning to pick up, thanks to burst capacity provided by ACI.   This is powerful stuff.  Here we can see AKS and ACI combine to provide the best of “serverless” computing – invisible infrastructure and micro-billing – all managed through the open Kubernetes APIs.  This kind of innovation – the marriage of containers and serverless computing -- is important for the industry, and Microsoft is working hard to make it a reality.

Once you've done all the set up you just need these commands during the live demo:

$ helm install charts/fr-demo --name demo
$ az aks install-connector --resource-group <myResourceGroup> --name <myK8sCluster> --connector-name myaciconnector
$ kubectl scale deploy demo-fr-ir-aci --replicas 10

Part 3 Clean up:

Replace <myResourceGroup>, <myK8sCluster> with your created in previous steps. Then run following command to clean up resources

$ helm del --purge demo
$ az aks remove-connector --resource-group <myResourceGroup> --name <myK8sCluster> --connector-name myaciconnector
$ az group delete --resource-group <myResourceGroup> --no-wait -y

virtual-kubelet-aci-burst's People

Contributors

afengkk avatar alexchx avatar chadliuxc avatar jluk avatar lachie83 avatar ogcanviz avatar rbitia avatar samkreter avatar supernova-eng avatar technosophos avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

virtual-kubelet-aci-burst's Issues

helm install fails, same with `az aks install-connector`.

I've tried following this guide with:

  • az aks install-connector on AKS, RBAC disabled
  • manual helm install method on AKS, RBAC disabled
  • manual helm install method on cluster outside Azure, RBAC disabled

In all instances I was unable to get it to work.

I know that AKS recently added RBAC. I'm not sure whether this guide has been updated since then. I suspect my errors are because helm is tricky to install on RBAC-enabled clusters. It's not clear whether this guide is for RBAC-enabled or disabled clusters. The az aks create command doesn't specify either way.

az aks install-connector

With the az aks install-connector method:

k describe node virtual-kubelet-myaciconnector-linux-eastus

  Type     Reason               Age                  From              Message
  ----     ------               ----                 ----              -------
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 72.05301ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 60.940238ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 63.277417ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 55.728664ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 74.759ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 174.284729ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 94.766969ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 93.29422ms: instance not found
  Warning  FailedToCreateRoute  38m                  route_controller  Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 56.232981ms: instance not found
  Warning  FailedToCreateRoute  3m (x1042 over 38m)  route_controller  (combined from similar events): Could not create route 37d96542-833d-11e8-9bbb-1aca1bb112c2 10.244.1.0/24 for node virtual-kubelet-myaciconnector-linux-eastus after 62.260086ms: instance not found

And the pod itself:

k logs myaciconnector-linux-eastus-virtual-kubelet-for-aks-664b67sm6ds

time="2018-07-09T05:59:18Z" level=error msg="interface public not found"
/.kube/config
2018/07/09 05:59:18 Reading ACS credential file "/etc/acs/azure.json"
2018/07/09 05:59:18 Load ACS credential file "/etc/acs/azure.json" successfully
2018/07/09 05:59:18 Node 'virtual-kubelet-myaciconnector-linux-eastus' with OS type 'Linux' registered
2018/07/09 05:59:18 tls: failed to find any PEM data in certificate input
2018/07/09 06:00:19 Pod watcher connection is closed unexpectedly.
2018/07/09 06:00:40 Pod watcher connection is closed unexpectedly.
...
2018/07/09 06:35:12 Pod watcher connection is closed unexpectedly.
2018/07/09 06:36:17 Pod watcher connection is closed unexpectedly.
2018/07/09 06:37:22 Pod watcher connection is closed unexpectedly.
2018/07/09 06:38:27 Pod watcher connection is closed unexpectedly.

helm ls

NAME                            REVISION        UPDATED                         STATUS          CHART                           NAMESPACE
ingress                         1               Mon Jul  9 16:25:54 2018        FAILED          nginx-ingress-0.22.1            kube-system
kube-lego                       1               Mon Jul  9 15:28:37 2018        DEPLOYED        kube-lego-0.4.2                 kube-system
myaciconnector-linux-eastus     1               Mon Jul  9 15:59:05 2018        DEPLOYED        virtual-kubelet-for-aks-0.1.4   default

helm install

Steps to reproduce:

Install ingress controller w/ helm.
$ helm install stable/nginx-ingress --name ingress --namespace kube-system

Error: release ingress failed: clusterroles.rbac.authorization.k8s.io "ingress-nginx-ingress" is forbidden: attempt to grant extra privileges: [PolicyRule{APIGroups:[""], Resources:["configmaps"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["configmaps"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["endpoints"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["endpoints"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["secrets"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["secrets"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["services"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["services"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["services"], Verbs:["update"]} PolicyRule{APIGroups:[""], Resources:["services"], Verbs:["watch"]} PolicyRule{APIGroups:["extensions"], Resources:["ingresses"], Verbs:["get"]} PolicyRule{APIGroups:["extensions"], Resources:["ingresses"], Verbs:["list"]} PolicyRule{APIGroups:["extensions"], Resources:["ingresses"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["events"], Verbs:["create"]} PolicyRule{APIGroups:[""], Resources:["events"], Verbs:["patch"]} PolicyRule{APIGroups:["extensions"], Resources:["ingresses/status"], Verbs:["update"]}] user=&{system:serviceaccount:kube-system:default b474f955-8340-11e8-82af-625d75d991f6 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]

result:

I couldn't access the application

I have gone through all the steps till deploying the application, which from the command line went through.
when I tried to use the URL given by step:
$ helm install charts/fr-demo --name demo
the URL was:
https://aks-app.eastus.cloudapp.azure.com
however when I tried to access the URL, I got the following error:
default backend - 404

I didn't receive any errors in previous steps, everything went smooth
I can also see from the Azure Portal -> AKS Cluster -> Metrics, that there are number of pods initiated and in ready state!
I am not sure what went wrong?

Deployed solution does not process images

I have followed the installation instructions to set up this sample without any major issues, but when I bring up the UI no images are being processed, and the CPU and memory utilisation of the single node are both very low. There is nothing I can see in the cluster config to suggest an issue - front end, back end, and ir applications are all running.

Can you please provide some pointers in the readme to application debugging logs or similar to help diagnose the problem?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.