anrgusc / jupiter Goto Github PK

@jasonatran: pinpoint pieces of Jupiter code that may be Ubuntu specific.

@caravansary83 and @NeveIsa: Try to run Jupiter on the new DCOMP setup. We just downgraded k8s, cni, and kubeadm so hopefully this works...

The DAG should be enhanced to allow input data to come into arbitrary task nodes from arbitrary compute locations in the network. Currently all input data comes from a home node. This enhancement is needed for sensing/IoT applications.

Once #28 is finished, a developer who codes a single task may still have dependency issues when it is finally launched along with an entire project. We need a way to allow developers to smoke test a single task by building and running a container which either fakes inputs or creates 0 inputs just to see if the task runs without error.

@MaLeiOnline we have a working version of Jupiter v5.0 in the following branch. This version is a lot easier to use, and the top-level readme provides some new instructions. It does not, however, have instructions on setting up k8s.

https://github.com/ANRGUSC/Jupiter/tree/finalccdag

Originally posted by @jasonatran in #30 (comment)

I made a copy of the old notes, gutted it, and added TODOs:

https://docs.google.com/document/d/1rPaLta-OQY6k8XjbsPqtejW1gFncDIqqK4OmEr2jRPU/edit#

We need to migrate over to ansible and setup for debian. We should also add a new section to our docs and ditch this google doc.

We need to store kubeadm.yaml files in the repo and also create a bash script to setup the master node for development.

We want to be able to do the following:

Cloning the Jupiter Orchestrator along with the Distributed Network Anomaly Detection related files, run :

git clone --recurse-submodules [email protected]:ANRGUSC/Jupiter.git cd Jupiter git submodule update --remote

Each app written in python will have a different set of dependencies. We need a way for app developers to provide a requirements.txt

Change all the .ini files to .cfg and make the necessary code changes to support this.

Repository: https://github.com/ANRGUSC/Automatic-DAG-Generator

TODO: Add the DAG degenerator in a subdirectory in this repository. Now that tools are starting to grow, we should consolidate all these tools under a tools/ directory to reduce the front page clutter.

@caravansary83 does it make sense to do this?

@gshanr please update your progress here.

Create a video to show how app developers who have finished writing code for all their tasks to launch their DAG using Jupiter.

Currently, the final output of a DAG is sent to the home node. Jupiter needs to be enhanced to allow arbitrary destinations for the final output to be sent to.

What happened (please include outputs or screenshots):
RUN: python3 auto_deploy_system.py
OUTPUT:

Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
1
Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
[8080]
['coded1']
coded1

• Serving Flask app "auto_deploy_system" (lazy loading)
• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: off
• Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)

Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
Deploy WAVE greedy mapper
Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
Exception Occurred
F1112 15:12:32.327170 114217 proxy.go:160] listen tcp 127.0.0.1:8080: bind: address already in use
Exception Occurred
Exception Occurred
Exception Occurred
Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
Exception Occurred

---

Network Profiling Information:
{}
Execution Profiling Information:
{}

---

---

WAVE mapper
Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
{'node1': ['Node1'], 'node2': ['Node2'], 'node3': ['Node3']}
Task mapper: Wave random selected
Non pricing scheme selected
CIRCE path-------------------------------------------
/home/master2/Jupiter-develop/circe/original/
Unhandled exception in thread started by <function deploy_app_jupiter at 0x7fe925bda400>
Traceback (most recent call last):
File "auto_deploy_system.py", line 361, in deploy_app_jupiter
k8s_jupiter_deploy(app_id,app_name,port)
File "auto_deploy_system.py", line 133, in k8s_jupiter_deploy
task_mapping_function(profiler_ips,execution_ips,node_names,app_name)
File "auto_deploy_system.py", line 63, in task_mapping
return f(args[0],args[3])
File "/home/master2/Jupiter-develop/mulhome_scripts/k8s_wave_scheduler.py", line 124, in k8s_wave_scheduler
ser_resp = api.create_namespaced_service(namespace, home_body)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 6820, in create_namespaced_service
(data) = self.create_namespaced_service_with_http_info(namespace, body, **kwargs)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 6905, in create_namespaced_service_with_http_info
collection_formats=collection_formats)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 321, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
_request_timeout=_request_timeout)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 364, in request
body=body)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/home/master2/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Tue, 12 Nov 2019 07:12:33 GMT', 'Content-Length': '208'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services "coded1-home" already exists","reason":"AlreadyExists","details":{"name":"coded1-home","kind":"services"},"code":409}

Environment:

• Jupiter version: 4.0

• Kubernetes version (kubectl version): v1.15.2

• OS: Ubuntu 18.04

• Python version: 3.6.9

We will be using something like https://github.com/ANRGUSC/MCPDAG as a skeleton and/or demo. We should provide a full skeleton for developers to practice creating a dummy DAG of their choosing by filling in the task() function (like in https://github.com/ANRGUSC/MCPDAG/blob/master/scripts/preprocess1.py). In the skeleton, we can guide them with comments above task().

Currently, sphinxV4 helps to maintain the documentation for Jupiter on readthedocs, in Documentation for version 5, this should be merged to develop.

Instead of checking files in for a "dummy app", we should have a very simple example app and call the folder example_multicast.