openziti / fablab Goto Github PK

Implement a fablab box (or export, or...) that will take an instance and package it up for sending to another fablab user, or for downstream consumption of model data, reports, etc.

Implement a distribution stage, which can do an initial upload to a "distribution bastion" (created as part of the model), which is then used as a high-bandwidth staging point to distribute components to hosts in the model.

When on low-bandwidth links, this will drastically improve model update performance. The components will only transit the low-bandwidth link once, regardless of model host count.

Implement a detailed, per-instance "activity log".

Implement a fablab dump utility that will generate a human-readable (and possibly machine-readable (JSON)) "dump" of the current model structure.

Implement a log retrieval action, which will insert retrieved logs into the forensics tree.

Do a better job of tracking and persisting lifecycle stage information in the v0.4 version.

The zitilab/characterization model requires a lot of deep parameterization, and dynamic model building, based on several different kinds of configurability.

Explore how factory chains can be integrated into an even more dynamic model building approach. Validate this approach on the Ziti characterization implementation.

The docs need to be brought up to date for v0.3.

The zitilab/characterization project requires the ability to invoke multiple models, and/or multiple configurations of a model, to execute through a series of scenarios containing multiple tests.

Implement a very simple facility for listing configurations to be executed, and then executing them, collating their output into a higher-level result set.

Make enabling, and the snaplen configurable for the tcpdump integration in the zitilib/characterization/ziti model.

Help the core fablab engine to comprehend the state of instances… for a multiplicity of reasons.

Iterate one last time on the characterization reporting to close out the 0.3.x series and launch as 0.4.0. Report on all of the new data that's become available throughout this development cycle.

Incorporate support for:

• both TCP and UDP measurements
• UDP target bandwidth parameters
• full metrics capture across multiple iPerf3 test runs, inside of the same dataset (model activation)

In #14. we introduced the concept of "factories", alongside the existing actions and lifecycle stages. With the introduction of parenting (#18), we're now allowing those component collections to be amended in "sub-models". The current implementation works great for additive changes, or wholesale replacements of these sets. Where it breaks down is when we want to examine, or selectively remove from those component collections.

Implement better metadata and support for iterating and adjusting these collections programmatically.

Clean up the separation between components that are specific to zitilab, and the components that are core fablab. Currently, lifecycle stages and actions are commingled in a single location in the project structure. Separate. Refactor to support that separation where necessary.

Deeper dive into fablab strategy and supporting internals.

While working through upfitting the zitilib/example models to use the latest fablab idioms (#25), I'm seeing a lot of strangeness and potentially some regressions when running ziti-fabric-test loop2. It's currently unclear whether this is a ziti-fabric issue, or if this is a problem in the fablab implementation that is driving ziti-fabric-test loop2.

The current lib tree is global, and its namespace is shared across any models defined using the fablab SDK variant.

Might want to consider providing some sort of "namespacing" mechanism, allowing template variants to be scoped to specific model namespaces.

The 0.3.x series is currently hardcoded to use terraform as it's framework for expressing infrastructure. The infrastructure expression lifecycle stage is currently idempotent, because terraform apply itself is idempotent.

We're going to want to be able to swap in different infrastructure expression strategies. In a lot of cases, we may want multiple models to share a single chunk of infrastructure (characterization, for example). So, we may want to be able to express additional models on shared infrastructure.

Possibly co-develop an "inventory-based" infrastructure expression strategy. The existing terraform-based strategy could emit inventory data, which could then be re-used to express additional models on top of.

Here is an illustration of the issue:

{14:46}~/Repos/nf/github/fablab:master ✓ ➭ fablab list instances

[0] instances:

{14:46}~/Repos/nf/github/fablab:master ✓ ➭ fablab create bogus
[   0.000]    INFO fablab/cmd/fablab/subcmd.create: allocated new instance [429798997]
[   0.000]   FATAL fablab/cmd/fablab/subcmd.create: unable to create instance label [429798997] (no such model [bogus])
{14:46}~/Repos/nf/github/fablab:master ✓ ➭ fablab list instances

[1] instances:

429798997    open /Users/curt/.fablab/instances/429798997/fablab.yml: no such file or directory

{14:46}~/Repos/nf/github/fablab:master ✓ ➭ fablab clean
[   0.000] WARNING fablab/cmd/fablab/subcmd.clean: error loading label for instance [429798997] (open /Users/curt/.fablab/instances/429798997/fablab.yml: no such file or directory)
{14:46}~/Repos/nf/github/fablab:master ✓ ➭ fablab list instances

[1] instances:

429798997    open /Users/curt/.fablab/instances/429798997/fablab.yml: no such file or directory

Models often need to bind to data that can be expressed through multiple "cascades" of sources. Environment variables, command line switches, stored user preferences, specific configuration files, and others are all potential sources for model data.

These data sources end up in fablab as model "variables".

Refactor bindings, labels, and the other fixed sources for configuration into a simple, but appropriately extensible framework for injecting variables into models. The implementation should provide a single entry point, which makes writing model code that depends on variables simple and clear.

Incorporate scalability target testing and validation as part of the zitilib/characterization/ziti model:

Endpoints

• Maximum Concurrent Sessions
• Maximum Concurrent Connections
• Maximum Concurrent Services

Fabric

• Maximum Concurrent Routers
• Maximum Concurrent Sessions
• Maximum Concurrent Services

Bind the configuration of and the enablement/disablement of the various regions (local, short, medium, long) in zitilab/characterization/ziti.

Building on the 0.4.x architectural improvements, this update to performance characterization supports linked models, with linked infrastructure, and will ultimately include support for model data comparison summaries.

Continued elaboration of the underlying metrics.

Mapping directly to the previous characterization work.

Report on the full data set available in the model. This should also cover our performance characterization needs (#17).

Develop an "oh, wow!" (hello world) usage example. This likely ties to the generalized SDK model (#7).

Support both quic: and tls: protocols in the zitilib/characterization/ziti model.

The zitilib/characterization/ziti model needs to incorporate tcpdump captures from either the sender side, the receiver side, or potentially both sides. The capture should support limiting the packet size (only need the first 64 bytes for analysis).

Develop a framework for generating "compiled" reports. Given an instance and its current forensic payload situation, we should be able to invoke something akin to fablab report, and have it produce a distributable report containing standardized results from the instance.

Will probably use d3.js to produce some relatively nice-looking analysis of the data set contained in the instance.

Develop framework components that make it easy to express "cascading" data sources in the model. For example, we might want an instance type to have a default in the model, which can be overridden by a variable in the user's preferences (the binding), which is overridden by a variable set on the instance label. Rather than having to hand-code this type of cascade over and over, it should be expressed using a common framework component that makes the arrangement obvious for that data element.

Implement a simple host metrics probe (CPU, memory), and integrate it into the zitilab/characterization/ziti (#17) model.

The DevKit kitting strategy currently depends on ZITI_ROOT and is ziti-specific. This can easily become generic and still completely fulfill Ziti’s needs.

As a part of expressing infrastructure, our current models update operating system packages and restart the model instances. The models include a semaphore_0.Restart semaphore, to wait and check for model instance restart.

This is painful in those situations where you're running fablab up, or otherwise want to work through the infrastructure lifecycle stage, and you know there is no need to wait for the hosts to restart.

We should attempt to make this workflow smarter, to eliminate unnecessary waits. May require fablab to handle state tracking (#2) in a better way, first.

Develop a model for deploying and managing our MattermoZt deployment.

fablab currenty overwrites the fablab identity in ~/.ziti/identities.yml for each instance, which effectively only allows ziti-fabric to work with a single fablab model instance.

This really needs to become instance-based. Ideally we’ll add and remove identities from identities.yml as instances come and go.

• Deploy controller with edge enabled
• Deploy edge routers
• Deploy ziti endpoints (tunneler?)
• Base of characterization models

• Enable running on any OS
• Enable allowing alternate SSH authentication methods (i.e. windows does not support rsync/ssh/ssh-agent toolkits well)

fablab is intended to primarily act as a software framework, which can be extended and customized by developers (using code) to meet their needs. This means we're going to need an SDK-style workflow, which can acccomodate things like fablab new, and the static template files that are included, in addition to the framework code (and models) itself.

Refine the generalized SDK strategy.

Where the "distribution stage" is primarily concerned with delivering bits onto the model, the “forensics” framework is primarily concerned with retrieving and locally curating bits from the model (logs, databases, files, snapshots, etc.).

Update the zitilab/development models to the 0.4.x idioms.

The model.Dump functionality provides everything that operation.Persist does, plus it includes a snapshot of the model state.

We're going to want the ability to specify bindings on the command-line.

As I start running longer and longer tests, the cost of a crash far along in the testing process becomes a heavier loss in terms of data and time.

It might make sense to start "checkpointing" the model throughout the characterization run, to minimize potential data losses.

Develop components that facilitate “swapping” out various approaches, such that they can be characterized via metrics.

This generally means configuring two appropriate models, running them through a set of operations, and then comparing the observed metrics for both.

Need to be able to report on incomplete data sets:

$ fablab exec report
[   1.082]    INFO fablab/zitilib/characterization/reporting.(*report).Execute: [/home/michael/.fablab/instances/dev/dumps/1582731593823.json] => [/home/michael/.fablab/instances/dev/reports/0.html]
[   2.067]    INFO fablab/zitilib/characterization/reporting.(*report).Execute: [/home/michael/.fablab/instances/dev/dumps/1582737334394.json] => [/home/michael/.fablab/instances/dev/reports/1.html]
[   2.113]   FATAL fablab/cmd/fablab/subcmd.exec: action failed [report] (error building report data (error getting ziti iperf udp summary (error querying json path [$.regions.short.hosts.client.scope.data.iperf_udp_ziti_1m_metrics] (key error: iperf_udp_ziti_1m_metrics not found in object))))

Implement a simple inheritance model, allowing for specialization of models.

What if we could seamlessly integrate strategies developed using Node.js into our behavioral models? We're already going to end up with JSON imports for the structural model, including links to the behavioral model components (#27), what if we could also have fablab directly invoke dynamic language components in JavaScript (or other dynamic languages)?

Provide a facility for marking binding data as "sensitive", so that when models are dumped or otherwise externalized, those contents are omitted or obscured.

The tcpdump implementation should support Joiner.

Move the loop testing into the operate lifecycle stage (fablab run) in the zitilib/examples models.