wfau / gaia-dmp Goto Github PK
View Code? Open in Web Editor NEWGaia data analysis platform
License: GNU General Public License v3.0
Gaia data analysis platform
License: GNU General Public License v3.0
Examples of how to run tasks on our Spark system using a Java program launched from a command line rather than inside a inside a notebook.
Evaluate the INDIGO IAM service and see if it can help us with things like delegated identities and registering our service as an OAuth application in 3rd party systems like GitHub (see #55 )
Learn how to deploy Kubernetes using Ansible.
Single user account, with some form of calendar to allocate who is using the system when.
Transfer copies of the Gaia DR2 parquet files into a public Swift Object Store that can be shared across projects.
Transfer a copy of the Gaia DR2 csv files into a public Swift Object Store that can be shared across projects.
Evaluate the iris-magnum examples from John Garbutt at StackHPC.
https://github.com/RSE-Cambridge/iris-magnum
Needs to be done in time for the IRIS F2F meeting at ROE in March.
https://indico.ph.qmul.ac.uk/indico/conferenceDisplay.py?confId=645
We need to allow Zeppelin users to store data somewhere.
Do we look into using HDFS for this as a first step?
The Zeppelin node in the "production" service is very small so we can't use that most likely.
If we move to a service similar to what we have for lsst with Jupyter on K8s, and spawn new Zeppelin containers per user, we can allocate storage for those containers. Making it persistent probably is a longer term task though, as it would require us to setup user management / quotas / etc..
There is a requirement for persistent storage, but as a start temporary storage would work for helping our initial users develop notebooks, though persistent storage with HDFS might be easier to setup.
Download and transfer using command line tools.
We need a Continuous Integration system that runs automated tests each time we create a new deployment. This is the first step along that route, to propose solutions and describe how they would work.
Given we are putting our code into GitHub the obvious one is the GitHub Actions CI system.
but there is a whole range of others to choose from as well
Question is .. how do we integrate the CI system with a deployment on an external OpenStack system that isn't part of the GitHub universe ?
Make config-dns use the same instance discovery process as config-ssh to get the lists of virtual machines and their IP addresses.
Remove the reliance on the partial files generated during the create process.
I think we need to split this project into three separate GitHub projects to reflect the different layers in the system. Hopefully this will make it easier to manage the source code, issues and testing for each of the layers.
This way we can be working on development versions of the Kubernetes and Spark layers, while at the same time using the current release versions of them as a stable platform to develop the Spark and Zeppelin analysis tools.
This has implications for issue #33. Each layer would have its own CI integration, using the other layers as components to build the CI test environment.
Learn how to deploy Spark on Kubernetes.
Examples of how to run tasks on our Spark system using a Python client launched from a command line rather than inside a inside a notebook.
Initial step of importing notebooks from a publicly readable GitHib repository.
Integrate OAuth login into Zeppelin.
As we add more benchmark tests we need to be sure we are being consistent.
This task is to create some kind of standard layout that we use to state the conditions of each test that we run.
As an example, Nigel needs HDBSCAN installed for the “Kounkel and Covey groups demo” notebook.
Export a notebook as an executable Python program
Future issues will address where the program can be run from, how to import all the required libraries and what happens to the original notebook text.
Publicly readable GitHub repository of example notebooks.
Needs #55 to be able to import the notebooks.
Notes on how to use the parquet-tools
from Hadoop to inspect Parquet files.
https://stackoverflow.com/questions/36140264/inspect-parquet-from-command-line
I suspect we will end up using these a lot as we evaluate different ways to partition the data into Parquet files.
The StackOverflow question is about how to do this from the command line, but as this is a Java component, we can probably use these from Zeppelin notebook or a Spark job ?
Learn how to use OpenStack Magnum to deploy Kubernetes.
Create a github/aglais
directory in the Gaia svn repository and link it to this GitHub project.
This includes a shell script suitable for adding as a cron job that pulls the latest changes from the default
branch of the GitHub project and pushes them to the svn repository.
Using a top level github
directory in svn leaves room to add links to more GitHub projects if we split the project into smaller parts later on (see #34).
Evaluate Ansible deployment playbooks:
Alternative : https://github.com/sergevs/ansible-cloudera-hadoop
Cloudera director : https://blog.cloudera.com/automated-provisioning-of-cdh-in-the-cloud-with-cloudera-director-and-ansible/
Cloudera archive : http://archive.cloudera.com/cm5/
Examples of how to run tasks on our Spark system using a Scala program launched from a command line rather than inside a inside a notebook.
Integration with the GitHub system to be able to load and save notebooks to/from the users GitHub account.
Adding a 'Save to GHitHub' button to the Zeppelin interface could become the primary route for sharing notebooks with colleagues.
Learn how to deploy Spark using Ansible.
There is a bug in the Ansible os_server module that means create can't be applied twice. It should detect that there is already an instance with the same name and update it rather than create a new one.
From what I can tell, it does detect an existing instance, but it doesn't do the update properly. If the server definition contains a security group, then it tries to apply the same group again and fails with a duplicate group error message from the server.
First part of this task is to identify where it is going wrong, and create a patch fix for us to use. Then, if we can fix it properly, we could propose our fix to the upstream project.
Source code is here:
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/openstack/os_server.py
Basic export notebook function to export and download a notebook as a text file.
The Hadoop logs directory seems to grow quite fast, which we discovered recently after out-of-space errors started appearing when trying to run Spark jobs. The cluster recovered after clearing the logs from each worker.
We need to setup Logrotate on the Hadoop workers for /logs directory.
Explore how we can use the Zeppelin webservice API to run integration tests to verify the system is working as expected.
Instructions on how to import JSON file generated by 'export as file' menu option.
Formatted as a markdown page on the project wiki, including screen shots of key steps in the process.
https://github.com/wfau/aglais/wiki/How-use-the-platform
The Ansible/OpenStack os_network module has an issue with an extra param added to the call to the OpenStack client component:
We have a patch in our Docker container that implements a short term fix to make this go away:
Longer term, we need to track this:
A Zeppellin notebook that demonstrates how to unpack a Parquet data set and extract the metadata, displaying things like number of files, file format, compression, number of columns, block size, indexing etc.
Three use cases:
This task represents the first part of #20.
A Spark job to transform the Gaia DR2 csv files into Parquet.
Written as source code for a Spark job that we can submit automatically as part of a Continuous Integration test suite.
Including an initial set of pass/fail tests to count the output files, check the data types, count the rows etc.
Include a parameter to select how much of the data to process, e.g. 10 csv files, 100 csv files etc. enabling us to run quick tests and build small deployments with partial data.
Export a notebook as a PDF, with basic formatting.
Future issues will cover more advanced formatting options.
Importing the Gaia data from the csv files needs to define a schema to set the data type of the columns.
Examples:
A single hard coded user account and password, managed manually.
A set of notebooks that demonstrate how to do basic analysis steps on the DR2 dataset, including things like calculating the mean, creating histograms etc.
Each example written in each of the three main languages used for analysis, Python, Java and Scala. Aim is to build up a teaching resource like the Apache Spark documentation with tabs showing the Python, Java and Scala version for each example.
Based on example code from ashrithr
https://gist.github.com/ashrithr/f7899fdfd36ee800f151
Java tool that reads blocks from file:// or http:// and writes to hdfs:// designed as a Java program that can be run inside Spark.
Partly because a tool to do this would be useful for deployment, and partly to learn how to write Java code that can be run distributed inside Spark.
This task cleans up and frees all the resources created by #43.
Create a SSH key file that contains keys for each of the users and install it in our OpenStack projects.
https://help.switch.ch/engines/faq/how-to-use-multiple-ssh-keys/
If not, find an alternative method to enable developers to login to virtual machines created by another member of the team?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.