GithubHelp home page GithubHelp logo

graph's People

Contributors

gaurav-chawla avatar melugoyal avatar srohatgi avatar

Stargazers

avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

haozhan1 etsangsplk akamble-splunk

graph's Issues

Exhaustive resource executor

Currently, createSync() returns as soon as a resource execution errors out.

This leaves out a possibility that other alternate resources and associated resource sub-trees could still execute. For deeply nested and broad sub-trees this would be ensuring that all resources that can be executed are always executed.

Cached Resource execution state

Currently, buildCache is not persisted between execution runs. This results in the entire graph being executed, with potentially long running checks. While this ensures that the graph execution is always consistent, it can be slow to execute deeply nested graphs.

If we persist the buildCache, and have an associated policy to refresh it either on demand OR on a time basis, it can avoid potentially expensive checks.

Signature simplification for Sync() method

The current implementation of Sync() takes in Resource slice, with resource names being unique. It may be better to enforce this requirement by changing the signature to take in a map of resources indexed by a resource name.

The new signature would be:

func Sync(resources map[string]Resource, toDelete bool, factory Factory) error

Temporal Dependencies can not be specified without fields

If I want to say A should happen before B, but B does not depend on the output of A, there is no way to specify this.

One needs to specify fromfield and tofield as well.

dependencies:
    - fromresource: maestro-search-cfn
      fromfield: SHIAMRoleARN
      tofield: IAMRoleARN

Remove this requirement.

Validate Resource Graphs

Currently, the graph lib synchronizes resource graphs based on a correct dependency order. Following validation is performed:

  • dependencies within resources should always resolve correctly

Missing dependencies on a resource graph is not validated. This enhancement proposes adding validation capability for a given graph kind.

Proposed api changes:

// Opts captures customizable functionality like logging
type Opts struct {
  CustomLogger func(args ...interface{})
  Decorator    func(r Resource) Resource

  // Validator captures constrained dependency graph kinds
  Validator    map[string][]Resource
}

// Sync method uses the Resource slice to generate a DAG. The DAG is verified by comparing it to the 
// `kind` parameter. The DAG is processed based on the value of toDelete flag. Resources may be 
// processed concurrently. Processed resources may return a status string and or an error. The function 
// collects these and aggregates them in respective maps keyed by resource names.
func (lib *Lib) Sync(ctxt context.Context, kind string, resources []Resource, toDelete bool) (map[string]string, error)

wrong resourceName in dependencies fromResource leads to panic

this can be reproed by changing spec and override spec in example_usage.go to the following:

const spec = `
kinesis:
- name: mykin2
  shardcount: 5
  streamname: myEventStream
deployment:
- name: mydep
  dependencies:
  - fromresource: mykin
    fromfield: Arn
    tofield: KinesisArn
dynamo:
- name: mydyn
  tablename: myDynamoTable
`

const overrideSpec = `
kinesis2:
- name: mykin2
  shardcount: 10
`

stack trace:

panic: reflect: call of reflect.Value.Elem on zero Value [recovered]
	panic: reflect: call of reflect.Value.Elem on zero Value

goroutine 1 [running]:
testing.runExample.func2(0xbf1f59ffb90317c0, 0x584079, 0x12a8cc0, 0xc00000c070, 0xc00000c018, 0xc0000707e0, 0x1189dd3, 0xd, 0x1193930, 0x119344d, ...)
	/usr/local/Cellar/go/1.11.5/libexec/src/testing/example.go:116 +0x6f8
panic(0x115c700, 0xc00000aa40)
	/usr/local/Cellar/go/1.11.5/libexec/src/runtime/panic.go:513 +0x1b9
reflect.Value.Elem(0x0, 0x0, 0x0, 0xc000016900, 0xa, 0x1151300)
	/usr/local/Cellar/go/1.11.5/libexec/src/reflect/value.go:805 +0x1bd
github.com/srohatgi/graph.checkField(0x0, 0x0, 0xc0000168e8, 0x3, 0x12c5900, 0x0)
	/Users/mehulg/go/src/github.com/srohatgi/graph/builder.go:150 +0xea
github.com/srohatgi/graph.check(0xc00000aa20, 0x2, 0x2, 0xc00000ee40, 0xc000095ac0)
	/Users/mehulg/go/src/github.com/srohatgi/graph/builder.go:136 +0x4ce
github.com/srohatgi/graph.Sync(0x11b1960, 0xc00000ee40, 0xc00000aa20, 0x2, 0x2, 0xc00000ee00, 0x11b1960, 0xc00000ee40, 0xc000095b48)
	/Users/mehulg/go/src/github.com/srohatgi/graph/builder.go:95 +0x46
github.com/srohatgi/graph_test.Example_usage()
	/Users/mehulg/go/src/github.com/srohatgi/graph/example_test.go:114 +0x289
testing.runExample(0x1189dd3, 0xd, 0x1193930, 0x119344d, 0x94, 0x0, 0x0)
	/usr/local/Cellar/go/1.11.5/libexec/src/testing/example.go:121 +0x1fd
testing.runExamples(0xc000095ef0, 0x12a1aa0, 0x1, 0x1, 0x1000101)
	/usr/local/Cellar/go/1.11.5/libexec/src/testing/example.go:45 +0x1ad
testing.(*M).Run(0xc0000b4080, 0x0)
	/usr/local/Cellar/go/1.11.5/libexec/src/testing/testing.go:1035 +0x1fb
main.main()
	_testmain.go:66 +0x13d
FAIL	github.com/srohatgi/graph	0.022s

stack trace is from v0.5.1

Wait for status before calling the next node update method

Currently, Sync method calls the next node's update method without it to be ready. To avoid that we have to add the polling inside the update method so the next method dependencies are fulfilled.
Builders should implement IsReady() bool method and Sync method should call it before calling the next node's update method.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.