networkservicemesh / networkservicemesh

The Hybrid/Multi-cloud IP Service Mesh

Archived

This repo has been archived. Network Service Mesh continues to be very actively developed in multiple repos.

What is Network Service Mesh

Network Service Mesh (NSM) is the Hybrid/Multi-cloud IP Service Mesh.

For more information, have a look at our website

Getting started

Get Started with our latest release

Docs

Documentation can be found on our website

Get involved

Information on getting involved: meetings, slack, etc can be found on our commmunity page

networkservicemesh's Issues

X-Factor CNFs - Define guidance similar to 12-factor app heuristics.

Error in network service mesh when creating NetworkService CRD

Created network service CRD with this config yaml -

cat networkservice-crd.yaml
apiVersion: networkservicemesh.io/v1
kind: NetworkService
metadata:
  name: gold-network
spec:
  name: gold-network
  selector: routing
  channels: gold-ethernet%

Seeing errors in controller for this in nsm:

ERROR: logging before flag.Parse: E0601 18:20:20.252772       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: v1.NetworkService.Spec: netmesh.NetworkService.Channels: []*netmesh.NetworkService_NetmeshChannel: decode slice: expect [ or n, but found ", error found in #10 byte of ...|hannels":"gold-ether|..., bigger context ...|65c8-11e8-b86b-080027566d28"},"spec":{"channels":"gold-ethernet","name":"gold-network","selector":"r|...
ERROR: logging before flag.Parse: E0601 18:20:20.274367       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: v1.NetworkService.Spec: netmesh.NetworkService.Channels: []*netmesh.NetworkService_NetmeshChannel: decode slice: expect [ or n, but found ", error found in #10 byte of ...|hannels":"gold-ether|..., bigger context ...|65c8-11e8-b86b-080027566d28"},"spec":{"channels":"gold-ethernet","name":"gold-network","selector":"r|...
ERROR: logging before flag.Parse: W0601 18:20:20.274597       1 reflector.go:341] github.com/ligato/networkservicemesh/pkg/client/informers/externalversions/factory.go:72: watch of *v1.NetworkService ended with: very short watch: github.com/ligato/networkservicemesh/pkg/client/informers/externalversions/factory.go:72: Unexpected watch close - watch lasted less than a second and no items received
ERROR: logging before flag.Parse: E0601 18:20:21.277375       1 reflector.go:205] github.com/ligato/networkservicemesh/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1.NetworkService: v1.NetworkServiceList.Items: []v1.NetworkService: v1.NetworkService.Spec: netmesh.NetworkService.Channels: []*netmesh.NetworkService_NetmeshChannel: decode slice: expect [ or n, but found ", error found in #10 byte of ...|hannels":"gold-ether|..., bigger context ...|65c8-11e8-b86b-080027566d28"},"spec":{"channels":"gold-ethernet","name":"gold-network","selector":"r|...
ERROR: logging before flag.Parse: E0601 18:20:22.279390       1 reflector.go:205] github.com/ligato/networkservicemesh/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1.NetworkService: v1.NetworkServiceList.Items: []v1.NetworkService: v1.NetworkService.Spec: netmesh.NetworkService.Channels: []*netmesh.NetworkService_NetmeshChannel: decode slice: expect [ or n, but found ", error found in #10 byte of ...|hannels":"gold-ether|..., bigger context ...|65c8-11e8-b86b-080027566d28"},"spec":{"channels":"gold-ethernet","name":"gold-network","selector":"r|...
ERROR: logging before flag.Parse: E0601 18:20:23.282213       1 reflector.go:205] github.com/ligato/networkservicemesh/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1.NetworkService: v1.NetworkServiceList.Items: []v1.NetworkService: v1.NetworkService.Spec: netmesh.NetworkService.Channels: []*netmesh.NetworkService_NetmeshChannel: decode slice: expect [ or n, but found ", error found in #10 byte of ...|hannels":"gold-ether|..., bigger context ...|65c8-11e8-b86b-080027566d28"},"spec":{"channels":"gold-ethernet","name":"gold-network","selector":"r|...
ERROR: logging before flag.Parse: E0601 18:20:24.284469       1 reflector.go:205] github.com/ligato/networkservicemesh/pkg/client/informers/externalversions/factory.go:72: Failed to list *v1.NetworkService: v1.NetworkServiceList.Items: []v1.NetworkService: v1.NetworkService.Spec: netmesh.NetworkService.Channels: []*netmesh.NetworkService_NetmeshChannel: decode slice: expect [ or n, but found ", error found in #10 byte of ...|hannels":"gold-ether|..., bigger context ...|65c8-11e8-b86b-080027566d28"},"spec":{"channels":"gold-ethernet","name":"gold-network","selector":"r|...
ERROR: logging befo

Stop relying on $HOSTNAME to identify the pod

Hostnames can be set in the pod's spec. They are not capable of reliably identifying the current pod's metadata name or differentiating an eNSM from another.

Find all instances of HOSTNAME and exorcise them unless that specific instance really does need a host name.

creating daemonset for network mesh fails

Trying to follow instructions mentioned here to create network service - RUN.md

deploying using this command - kubectl create -f conf/netmesh.yaml

Pod failed to create, seeing following two issues-

• Labels mentioned in RUN.md are not same in node-selector label in YAML file.

• YAML file has imagePullPolicy: Never, this should be either imagePullPolicy: Always or imagePullPolicy: IfNotPresent

• Docker images for network service mesh doesn't exist -

root@node2:/media/sf_golang# docker pull ligato/networkservicemesh/nsm
Using default tag: latest
Error response from daemon: pull access denied for ligato/networkservicemesh/nsm, repository does not exist or may require 'docker login'
root@node2:/media/sf_golang# ^C
root@node2:/media/sf_golang# docker pull ligato/networkservicemesh/netmesh
Using default tag: latest
Error response from daemon: pull access denied for ligato/networkservicemesh/netmesh, repository does not exist or may require 'docker login'

netmesh process crash

There is a race condition due to the fact that crds are handled by two plugins instead of one. When crd plugin get AfterInit before the Handler plugin, then ObjectStore methods might not be ready and when it calls Handler's methods nil pointer exception happens.

time="2018-07-02 00:02:08.41343" level=info msg="LogCrdHandler.ObjectCreated: %!(EXTRA *reflect.rtype=*v1.NetworkService, *v1.NetworkService=&{{NetworkService networkservicemesh.io/v1} {gold-network  default /apis/networkservicemesh.io/v1/namespaces/default/networkservices/gold-network f5be5144-7d4d-11e8-98e0-5254004c4249 5601202 1 2018-07-01 16:44:02 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {name:\"gold-network\" namespace:\"default\" labels:<selector:<key:\"networkmesh.io/app\" value:\"gold-network\" > >  [metadata:<name:\"gold-ethernet\" > payload:\"ipv4\" ] {} [] 0} { }})" loc="handler/plugin_impl_handler.go(105)" logger=netmeshhandler
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x100a01b]
goroutine 94 [running]:
github.com/ligato/networkservicemesh/plugins/handler.(*Plugin).ObjectCreated(0xc4203613c0, 0x12c24a0, 0xc42057e000)
        /go/src/github.com/ligato/networkservicemesh/plugins/handler/plugin_impl_handler.go:106 +0x10b
github.com/ligato/networkservicemesh/plugins/crd.workforever.func1(0x141fae0, 0xc4203922e0, 0x11e7160, 0xc420596060, 0xc42022c220, 0x14, 0xc4203612a8, 0x141f1c0, 0xc4200ea2d0, 0xc42022c220, ...)
        /go/src/github.com/ligato/networkservicemesh/plugins/crd/crd_queue.go:124 +0x996
github.com/ligato/networkservicemesh/plugins/crd.workforever(0xc4203612a8, 0x141fae0, 0xc4203922e0, 0x141f1c0, 0xc4200ea2d0, 0xc420258840)
        /go/src/github.com/ligato/networkservicemesh/plugins/crd/crd_queue.go:134 +0xc5
created by github.com/ligato/networkservicemesh/plugins/crd.(*Plugin).AfterInit
        /go/src/github.com/ligato/networkservicemesh/plugins/crd/plugin_impl_crd.go:233 +0xb40

I do not see why would 2 plugins be required to deal with CRDs, single CRD plugin should be able to deal with all CRD related activities.

Convert Selector from string to map

Currently Selector is defined as string in types.go:

type NetworkService struct {
	Name     string                           `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"`
	Uuid     string                           `protobuf:"bytes,2,opt,name=uuid" json:"uuid,omitempty"`
	Selector string                           `protobuf:"bytes,3,opt,name=selector" json:"selector,omitempty"`
	Channels []*NetworkService_NetmeshChannel `protobuf:"bytes,4,rep,name=channels" json:"channels,omitempty"`
}

Based on the discussions, it appears Selector is planned to be used as a regular label selector which is map[string]string.
The proposal is to replace Selector's type from string to map[string]string

Move some scripts out of conf/sample

Both conf/sample/setup.sh and conf/sample/teardown.sh should move into the top-level scripts/ directory. The file conf/sample/README.md should be updated to reflect the move as well. Anything in docs/ referencing these should be updated as well.

Separate out the concerns for the audiences of NSM to make it more accessible

There are three principal audiences of developers/consumers for the NSM project:

1. Developers of NSM framework code and APIs.
2. Developers of NSM plugins who use the framework to create new network services.
3. Consumers of NSM plugins who leverage the plugins with their Kubernetes infrastructure to network Pods together or connect Pods to external networks.

The first audience of the NSM framework code developer is currently well served. The issue is how to enable the last two audiences to accelerate adoption of NSM. The following steps are suggested:

1. Separate out all NSM plugin code into single directory structure with instructions on how to create a NSM plugin.
2. Each NSM plugin should be self contained and adding a plugin should require minimal (no?) changes in the framework code.
3. Document how to install individual NSM plugins (assuming everyone does not want to install every NSM plugin).
4. Document how to use the plugin with a set of Kubernetes PODs
5. Put examples for each plugin into a similar directory structure to enable consumers of NSM plugins to use them.

Verify different container runtimes

We specifically want to verify namespace access when not using the Docker runtime.

Add readiness and liveness probes into Network Service Mesh

From a discussion over IRC, we should add these as we can prevent the pod from progressing to a ready state until NSM is ready. For example, until CRDs are registered and ready. Also, we can fail a pod faster if these probes become unresponsive. See below for a link to these:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes

Becoming a Kubernetes WG Member

Upstream ticket: kubernetes/community#2372

Investigate in-pod authentication

The review for #46 indicated a desire to explore in-pod authentication. This issue tracks that investigation.

When running and then stopping networkservicemesh as a daemonset, the Close() plugin function isn't called

We need to understand what signal is generated to the containers when a daemonset is stopped. Currently, our Close() function is not called when deleting a daemonset containing networkservicemesh. This may require changes in the Ligato framework should it need to handle a signal other than SIGINT [2].

[1] https://github.com/ligato/networkservicemesh/blob/master/plugins/netmesh/plugin_impl_netmesh.go#L91
[2] https://github.com/ligato/cn-infra/blob/master/core/event_loop.go#L25

Deletion of CRDs isn't working as expected

For some reason, on a CRD delete, this is triggered, meaning the factory can't find the object. An example of the error I get is as follows:

time="2018-05-30 18:51:03.41696" level=info msg="Read item 'default/yogi-bear' off workqueue. Processing..." loc="crd/crd_queue.go(67)" logger=netmeshcrd
time="2018-05-30 18:51:03.41759" level=info msg="Dequeuing interface of type networkservices" loc="crd/crd_queue.go(70)" logger=netmeshcrd
ERROR: logging before flag.Parse: E0530 18:51:03.417636       1 crd_queue.go:74] error getting object 'default/yogi-bear' from api: networkservice.networkservicemesh.io "yogi-bear" not found

Refactor any instances of shutdownChan <- struct{}{} to close(shutdownChan)

Instead of passing in a struct into a channel, e.g. shutdownChan <- struct{}{} we should close the channel instead through close(shutdownChan).

This allows the shutdown channel to be passed around internally with all listeners receiving the close event, and allows cleanup goroutines to block against the same channel until the channel is closed.

Here is an example:

https://play.golang.org/p/QDWJreCZuBb

CRD Definition validation

Since CRD definitions get created as a part of daemonset, a validation of desired schema against already existing must be added. Example, at one point in the past CRDs' definitions were created, then later some changes in API changed it in the code but the cluster still has old version of these definitions. The current logic of netmesh daemonset is to check if CRDs' definition exist and it they do, do not create them. This behaviour could cause lots of issues in the CRD controller, when a user tries to create new crd based object but the definition of that object does not match with expected.

Incorrect device plugin name

Currently NSM registers itself with kubelet using nsm.ligato.io, but the format should be:

nsm.ligato.io\{resource name}

Jun 18 12:55:46 kube-5.sbezverk.cisco.com kubelet[1356]: I0618 12:55:46.472757    1356 manager.go:290] Got registration request from device plugin with resource name "nsm.ligato.io"
Jun 18 12:55:46 kube-5.sbezverk.cisco.com kubelet[1356]: I0618 12:55:46.472787    1356 manager.go:307] Bad registration request from device plugin: the ResourceName "nsm.ligato.io" is invalid

Constrain network service manager API token access

We should work out the minimum set of API privileges necessary for the network service mesh to work.

Create/send new NSM meeting time poll

Action item: Send a follow-up poll to select the preferred NSM meeting time

Requested: during NSM Meeting on June 15th

Solves the problem(s):

• find a time that will work best for the group, including 8-9am PT (current time)
• use a voting method that allows voters to "give weight" to their preferred meeting time

Tasks:

• review results and options from previous poll https://docs.google.com/forms/d/e/1FAIpQLSdX7jjPn0AfEoCENVWLwI44fOuT7U_imLtYbsQvaGM7bqf-1Q/viewform

• create a new google form to allow each voter to use up to x votes

• use Pacific Time for meeting time options

• send URL to poll to mailing list for networkservicemesh: [email protected]

Publish images to docker hub

NSM images should be published to the docker repo on every successful merge in master.

NSM needs a mascot

Brainstorming ideas in July 13, 2018 NSM meeting:

• tunnel spider
• something that will look great as a plush, stuffed animal
• something alive preferred
• doesn't have to be an animal
• something switching connections (like telephones)
• 90's - Teenage Mutant Ninja Turtles - Master Splinter

Validate "namespace", "inline", and "omitempty" in CRD schemas

Namespaces were removed from the CRD schemas before #54 merged, and omitempty and inline were left intact. Explore if we need any of these or not post merge of #54.

Check for changed files after running script/build.sh on travis

The build script runs a code generator. We should check for modifications and fail if any are present.

Build NSM website using Hugo and Netlify

This will track the work being done create the Network Service Mesh website.

Tasks to add sidecar containers in NSM

These are the high levels tasks we need to do for adding sidecar containers.

• Create side car container image

• Add side car container spec

• Create MutatingWebhookConfiguration for API server

• Create MutatingAdmissionWebhook server for handling admission requests

• MutatingAdmissionWebhook service and pod spec

• Injector webhook configmap, containing container spec information to be injected in target pod

• TLS authentication between API server and MutatingAdmissionWebhook server

NSM Enhancement Proposal: Support CNCF CNF project

Task: Write up/KEP of how NSM could help support CNCF CNF project

Including:

• Overview of CNF project => see #165 (comment)

• Where does K8s not provide required features (for Kubecon test cases)? => see #165 (comment)

• Where is the integration of NSM with CNF?

  • CNCF CNF project folks will provide NSM team with access to example tests cases and use case information
  • NSM folks will provide steps and Q/A use NSM to provide needed functionality
  • CNCF CNF team with integrate NSM following given instructions into provisioning code and test

---

To inform:

• New NSM features that would need planned to support CNF project

Rework integration tests to allow for developers to run them

Currently it's difficult to run the integration tests in a developer environment without copying and pasting. Work should commence to make it so they are much more modular and it's easier to run them in a developer environment.

SharedInformer and API

When k8s Informer informs about k8s object deletion, it does not provide the actual object as in the case of object creation, instead it returns name/namespace. Normally it would be sufficient but with proposed API, delete method is called with the object.

type API interface {
	ObjectCreated(obj interface{})
	ObjectDeleted(obj interface{})
	ObjectUpdated(objOld, objNew interface{}) 

In order preserve this API model, prior to calling ObjectDeleted, a type of deleted object needs to be deduced and only then ObjectDeleted should be called with this object type.

It can be done by switching back to per object type Informer, in this case we will know implicitly the type of object.
For the SharedInformer case, ObjectDeleted should be changed to string, but it will be impossible to guess the type of object.

WDYT?

Add validators for CRDs

PR #54 adds in support for CRDs, but does not add in the validators into the CRD schemas. We'll likely want to take advantage of this functionality, so this issue will track adding that support.

Work out documentation infrastructure

Considered options are a docs directory in the repo or gh wiki.

Add Serguei to CNCF cluster access list

This is done, adding for tracking purposes.

Check for deprecated Kubernetes API calls as a result from 1.10.5 to 1.11.0 upgrade

I've done some preliminary checking, but a bit more needs to be done. Adding this issue for tracking.

Makefile doesn't preserve GOPATH and other variables

Executing scripts in Makefile does not preserve variables that are set in the Makefile itself.

Unit testing

The NSM code needs unit test coverage.

Refactor the work queue functions for CRD objects

Per Frederick's comment on #71, we should look at refactoring the work queue functions (networkserviceWork(), networkservicechannelWork(), and networkserviceendpointWork()) into a single function. They are all basically the same functions.

Migrate errors to go-errors

Handle errors from EventLoopWithInterrupt()

This call [1] can return errors, and we're not handling them.

[1] https://github.com/ligato/networkservicemesh/blob/master/cmd/netmesh/main.go#L26

Document how to get a privileged container

Investigate sidecar approach for NSM

Similar to Istio, we may want to use a sidecar model where we put a container into each pod which is run.

Implement Customer Resource Definitions

We have a protobuf file containing NetworkServices and NetworkServiceEndpoints, we need to implement CRD support to enable these resources to exist in the Kubernetes DB and be manipulated with kubectl.

Add shellcheck support to perform CI of bash scripts

I'm working on this as well as issue #139 and will post a patch soon.

[Proposal] Allow only squash commits

GitHub allows us to control the types of commits we can perform in the project. I recommend we use the following settings:

• Allow merge commits - Add all commits from the head branch to the base branch with a merge commit.
• Allow squash merging - Combine all commits from the head branch into a single commit in the base branch.
• Allow rebase merging - Add all commits from the head branch onto the base branch individually.

The reason for disabling merge and rebase is two fold:

• Intermediate states representing work in progress would not be present in the repo, making history easier to read
• Easier to find defects via git bisect since only states that pass the build are merged in.

Any preferences or comments?

Workaround travis not having systemd

Per this issue here [1], we'll need to figure out a workaround for the eventual dropping of localkube support in Kubernetes, as our travis runs of Minikube require localkube.

kubernetes/minikube#2704

Add readiness and liveness probes

Readiness probes prevent traffic from reaching a pod until the pod is ready.

Liveness probes kill a pod if the pod becomes unresponsive.

We should add both.

L2 forwarding with VPP example

• Create L2 forwarding example
• Document the example

Integration test coverage

We need to settle on some way to test NSM functionally.

Verify the ability to change the CNI driver in a Minikube

This came out of the 2018-06-01 weekly NSM meeting. We want to see if it's possible to change the CNI driver inside of a qinikube. Useful for testing and CI purposes.

Check out SR-IOV on packet.net

Ian Wells checked out whether SR-IOV support is available. Support appears to be present.

We may need to ask packet.net to enable in BIOS for a given instance.

• Determine if we can set BIOS SRIOV settings on demand
• Check for SR-IOV support on Mellanox system (thanks Ian)
• Check for SR-IOV support on Intel based system
• Determine if we can break down NIC on single NIC system

Set up project board

• Add issues from meeting agenda to GH
  Add labels to issues tracked in meeting (doesn't seem necessary)
• Verify project board is set up properly
• Link agenda to project board

Review protobuf usage for protobuf 1.1.0 breaks

Protobuf will now break when messages are instantiated with implicit names. We should check for the following pattern.

// Incorrect
MyMessage{"foo", 10}

// Correct
Message{Message: "foo", Status: 10}