intuit / katlas

A distributed graph-based platform to automatically collect, discover, explore and relate multi-cluster Kubernetes resources and metadata.

K-Atlas

What It Does?

K-Atlas (pronounced Cutlass), is a distributed graph based platform to automatically collect, discover, explore and relate multi-cluster Kubernetes resources and metadata. K-Atlas's rich query language allows for simple and efficient exploration and extensibility.

It addresses following problems in a large scale enterprise environment of Kubernetes.

• Discoverability
  • Find K8s objects across multiple distributed K8s clusters
  • Real-time view of discovered objects
  • Streaming APIs and UI for both programmatic and human interactions
• Advanced Exploration
  • Identify similarities and differences between objects from pods to clusters
  • Correlate different objects by performing advanced join operations
• Federated Application View
  • Applications take center stage. K-Atlas provides a unique, application-centric view, with metadata from multiple clusters
  • Single pane of glass view of the entire application - from edge to database, across all clusters, regions etc.
• Reporting
  • Provide advanced reporting on compliance, security and other organizational policies
• Policy Enforcement
  • Allow for organizational policies to be enforced across the fleet in a consistent manner

Check out more details on Motivation and Use Cases that K-Atlas is addressing.

It provides a Web Viewer that can be used to search the Kubernetes cluster data and view graphical results in real time.

Click on the thumbnail below to see a demo video.

Core Components

Collector

The collector is responsible for discovery of Kubernetes assets in Kubernetes Clusters. For details on the Collector design, please refer Design Concepts

K-Atlas Service

The K-Atlas Service exposes APIs that can be used to get details about Kubernetes entities and run queries to help diagnose issues in Kubernetes clusters. For details , please refer K-Atlas APIs

K-Atlas Browser

The Web Application exposes UI search capability to search clusters based on several criteria and provide a real time graphical view of entities. For details on usage, please Click here to see a demo

Graph Database

Dgraph is used as the graph database. To know more about our motivation to choose Dgraph, please refer Design Concepts

Deploying to a Cluster

Technical Requirements

Make sure you have the following prerequisites:

• A local Go 1.7+ development environment.
• Access to a Kubernetes cluster.

Setup Steps

How to Set Up.

Releases

Latest version (v0.6)

• QSL query support
• Dynamic search result layout based on QSL
• Graph view based on QSL query required objects & relationships
• Pagination support for both API and UI
• Custom metadata definition via new API

More details about specific K-Atlas features are at Release Notes.

Contributing

We encourage you to get involved with K-Atlas, as users or contributors and help with code reviews.

Read the contributing guidelines to learn about building the project, the project structure, and the purpose of each package.

katlas's Issues

Refine metadata API

1. separate metadata and schema creation
2. support json array and single josn input when create metadata and schema
3. initial create schema and metadata while server start

Update QSL document based on new QSL changes

https://github.com/intuit/katlas/blob/master/docs/qsl-api.md

Add conditions to validate test results and update test case data for happy path

Setup Prometheus to view Dgraph Metrics

Review and update documents with latest version

historical API design

1. user can get entity change history by specify time range
2. support history graph view

Search API could return object key with empty entries

The search API could return something like:

{
"obj17": [],
"obj18": [],
...
}

UI need to support pagination of search result

Review Dgraph Config an test with new config

Review existing config with Dgraph team and update accordingly.

Get page parameter from URL

The page param is stored in the state which will be kept after filing a new search. Better to put in the parameter. The other benefit of doing that is when sharing the URL, it will directly dive into the right page.

implement update and deletion in metadata API

Add more prometheus metrics to API service

latency
api error count
TPS
etc.

also check the metrics required by foremast

Execute sanity test from Travis CI

Browser need to retrieve edge information from metadata API instead of hard coding

Graph should not flash/reset when new data is added to network

Following the extend-graph-navigation work, the graph currently flashes and re-sets when data is added to the rendered network.

This should be able to be mitigated, even with new store data objects per change since that data only feeds the visjs.org network data object which can remain in place. currently, the network object is being cleared and re-built upon data changes. This task is to fix the issue to improve UX

QSL to support count filter on node and relationship

Need QSL to support use case like find out Application had less than three pods linked to it

"I'm Feeling Graphy!" button should show only first entity in response

currently, the direct to graph mechanism will show a tree for each item in the response list

but to better maintain the metaphor of "I'm feeling lucky", we should only show the first item in the response list

allow QSL projection into graph view for entities other than the first

following #98, only the first entity in an array is shown in the graph when using "feeling graphy" concept

but this means there is no good way to see the items, other than the first. so we should allow the user, on the results list view, to flow into the graph while using that item in the QSL response array

Transaction abort error when populate data from k8s cluster to dgraph

Found data missing when k8s collector sending data to dgraph.
The error showing ERRO[0048] Transaction has been aborted. Please retry.
This could be caused by multiple mutate requests send to dgraph at the same time, need add verify and retry when error happen.

metadata API should support mixed case queries

currently, querying the metadata API with object types including upper-case chars, leads to nearly empty response objects

the UI can't use these as it expects and this can lead to later UI issues in the results list view

here is an example of an expected request/response:

/v1/metadata/cluster

{"uid":"0x25b1ea","name":"cluster","fields":[{"fieldname":"name","fieldtype":"string","mandatory":true,"cardinality":"one"},{"fieldname":"resourceid","fieldtype":"string","mandatory":false,"cardinality":"one"},{"fieldname":"resourceversion","fieldtype":"string","mandatory":true,"cardinality":"one"},{"fieldname":"creationtime","fieldtype":"string","mandatory":true,"cardinality":"one"},{"fieldname":"k8sobj","fieldtype":"string","mandatory":true,"cardinality":"one"},{"fieldname":"objtype","fieldtype":"string","mandatory":true,"cardinality":"one"}]}

and unexpected:

/v1/metadata/Cluster

{"uid":"","name":""}

Graph should not focus/blur upon mouse over/out

Currently, the graph DOM node has a blue outline when the mouse is over the diagram.

This is a small but awkward behavior which catches the eye.

I'll add a screenshot or video of the behavior soon.

Graph view should be able to be loaded with a structured network of data (more than just root UID)

Based on meeting with @fzxu, @formuzi, @ssmails, @kianjones4

extending graph view such that it can be initialized with more than just a root UID

instead a specific QSL response or similarly structured data used in certain circumstances:

• object type is Application

this allows for a QSL request put together by the user to be immediately "projected" into a graph view with all of the nested child (chained in query) objects included

Add log files for test results or test reports

Graph view requesting `undefined` entity UID

Once the user loads the graph view, an async request is made to retrieve entity data by UID. The UI is currently requesting undefined value from the API. This likely means that a recent change caused an issue where the root node UID is not being correctly registered and maintained.

Here is an example API request URI: https://example.com/v1/entity/uid/undefined

The user experience of this bug is that the orange notifier banner appears and indicates the following error: "rpc error: code = Unknown desc = Some variables are used but not defined Defined:[] Used:[undefined]"

I am seeing this behavior in all deployed environments.

Expose prometheus api for API service

Push docker images to docker hub after success build

Abstract data access layer to support multiple database

Run docker images in Travis CI and update relevant deployment docs for open source

Run docker images in Travis CI and update relevant deployment docs for open source

Search API to support pagination

Due to grpc maximum size limitation, search API need to support limit, offset

K8s collector should only send the data schema defines

The current k8s collector controller would send everything it collects which is not necessary as the database only stores the data based on the schema.

It should send the data defines in the schema only.

Create meta & schema data in json format with collected data from developer portal service

Create meta & schema data in json format with collected data from developer portal service

dynamic table layout base on search result

UI mockup for change history

Design the mockup for object change history

Run go test cases in a structured sequence and use test fixtures

#115

Build image and push to docker hub for collector on Travis CI

Add commands to Travis CI

QSL pagination does not work properly

For query like: http://localhost:8011/v1/qsl/Namespace[$$first=10,offset=0]{*}.Deployment{*}.Application[@name="config-service"]{*} returns count 7, result 0

Add function test for search and query API

API should alway return application/json content type

It only returns application/json if everything goes fine, otherwise would be plain/txt for now. We should always return json type even with exception as long as there is body.

draft JS styleguide

ideally most of the style guide will be a community contribution or standard

create a markdown file to document those we'll adhere to

then we'll want to draft out extenstions to that base due to specifics of this project or other reasons

where possible, also add this to an eslint rc file and add that file, along with dox to project

Understand go test framework and initialize the test

Get familiar with Go test framework and how to set up the initial test suites such as Sanity test

Add converting and displaying test results to junit report for unit tests

Support search by k8s label

need a design how to support search by JSON type.

uppercase characters in query can cause UI bug

@formuzi recognized an issue in the UI related to upper/lowercase of characters in query

this needs to be further investigated, and once the cause is identified, the bugfix should be story pointed

notification is shown a blank orange box, no content

history view UI design

1. user should able to get entity history from UI
2. support history data graph view

Graph should reset data after leaving and re-entering view

Currently, if a user drill-down navigates in the graph view and then goes back to home/results views and finally back to the graph by way of a different scoped UID, the graph can be polluted/optimistic with data the user hasn't yet requested (this time).

A sequence of screen shots will explain this best:

QSL wildcard search support

I can do search like deployment[@name="*abc*"] or regex style deployment[@name="^abc"]

KAtlas Website

Create a new marketing website for KAtlas and make it available at https://katlas.io and https://k-atlas.io

QSL need to support pagination

Due to grpc maximum size limitation, qsl API need to support limit, offset

Graph nodes should have hover tooltip with full name and details

Currently, graph nodes often have long names and this makes for messy graphs when there are many of them.

Instead, the graph could show only a shortened (capped at 20-30 chars - ex: first14chars...last14chars) name and all info in a tooltip overlay

• name
• assoc. clustername and/or namespace
• status
• more?

Or perhaps no label at all

or the label is + or similar fonticon and acts as expand (contract?) button