ncar / xdev-projects

All past, current, and future projects for the Xdev Team

THE CONTENTS OF THIS REPO HAVE BEEN MOVED INTO THE NCAR/XDEV REPO!!! Please use that repository in the future.

xdev-projects

All past, current, and future projects and planning documents for the Xdev Team.

This repository is meant to contain team our project plans and organization efforts. First, let's define some terminology:

• Projects are concerted efforts with a few measurable goals that span only a few months of work (i.e., a quarter of a year). Projects are meant to be small enough in scope and commitment to see the "light at the end of the tunnel." That is, it should be clear when defining a project what the outcome should be before starting any work.

• Campaigns are long-term efforts that can span multiple years, potentially. Campaigns are large enough in scope and commitment that it is hard to see what the outcome will be specifically, and yet some (possibly vague) goals can be articulated with the expectation that it might take a long time (and a lot of work) to achieve those goals. Campaigns should be seen as spanning multiple projects. Campaigns are harder to define specifically than projects, and as such should be seen more as a way of defining a direction for work to take.

Our work can be categorized into one of two categories:

• Operational work pertains to day-to-day activities that do not have a defined deadline. That is, operations can include responding to questions from scientists or users of our tools and utilities. Operations can include work to stay up-to-date on technology, such as reading blogs, following issue trackers, etc.. In simple terms, operations are the stuff we do every day just because its required to do a good job on our projects, but may not directly impact or even be associated with an existing project or campaign.

• Project work is work on a specific project, tied to a measurable goal (or goals) of that project. Most of our work should be project work. Projects should be scoped with the intention that they can be fully completed in a matter of 2-3 months (i.e., quarterly), but shorter projects are acceptable.

This repository is meant to help organize and plan our project work. The intent is for each project to be described by a single document, a Project Specification document. A template for this document exists in the projects directory. Similarly, Campaign Specification documents should be created, too, though the requirements for defining a campaign as much less stringent.

Contents

• projects/: This directory contains all project-related specifications and templates.

  • projects/current/: All project specification documents for on-going projects should be placed here.
  • projects/complete/: All project specification documents for completed projects should be placed here.
  • projects/construction/: All project specification documents for projects that are still being constructed/specified should be placed here. This includes projects that are waiting to start (and, therefore, don't have milestone dates defined, yet).
  • projects/construction/projspec_template.md: A template for Project Specification documents. When a new project is defined/specified, it should be named projspec_{codename} where {codename} is replaced by the project codename.

• campaigns/: This directory contains all campaign-related specifications and templates.

  • campaigns/current/: All campaign specification documents for on-going campaigns should be placed here.
  • campaigns/former/: All campaign specification documents for obsolete campaigns should be placed here.
  • campaigns/construction/: All campaign specification documents for campaigns that are still being constructed/specified should be placed here.
  • campaigns/construction/campspec_template.md: A template for Campaign Specification documents. When a new campaign is defined/specified, it should be named campspec_{codename} where {codename} is replaced by the campaign codename.

xdev-projects's Issues

Add project template

Currently, the project template resides in a Google document. It would be useful to have it converted into a markdown file that is then added to the repo for future reference.

New Strategic Balance?

In light of our recent discussions with the GeoCAT team, I have been thinking more closely about what efforts are uniquely suited to Xdev, rather than to GeoCAT. This issue is to create a forum to discuss these strategic efforts as a group.

Campaigns

I still believe that there are two primary campaigns to which Xdev should direct attention:

• Education, Outreach & Community Building (EOCB)
• Improving Scientific Data Analysis Workflows (WORKFLOWS)

The EOCB campaign is see as vital to build up both the number of users of the Python/Pangeo stack and to increase the number of scientist-developers who contribute back to the stack codebase (from reporting issues to providing necessary feedback and design input to writing actual code). Without a growing contributor base, these open source tools will always struggle to move forward to the solutions that we need them to be. We need these contributors to provide us input on what functionality needs improvement, how they do their analysis, etc.

The WORKFLOWS campaign is also vital to build out the functionality that the scientists need to do scalable data analysis. To some extent, this campaign is "obvious," but our recent discussions with the GeoCAT team indicate that the development of "operators" (to use Matt Long's word, which is any functional operation applied to the data during analysis) is an area that most clearly falls into the GeoCAT namespace, as does visualization capabilities. We should help with these efforts, as we have promised, but this means that the rest of the scientific data analysis workflow (as aluded to in the WORKFLOWS campaign spec) with the Pangeo stack falls more clearly into the Xdev namespace. And our efforts should reflect this.

New Projects

With all of this in mind, I think we need to scope out some new projects that clearly delineate Xdev-specific work. Some projects currently exist that clearly fall into the Xdev space (though GeoCAT collaboration is always welcome on them):

• [EOCB] Tutorials & Hackathons & Examples Gallery (in collaboration with GeoCAT, obviously)
• [WORKFLOWS:Ingestion] Intake-ESM and cataloging related packages/projects

And then I can foresee other projects in the near future that take us into new territory that clearly delineates us from GeoCAT:

• [WORKFLOWS:Search&Discovery] Notebook search/sharing via JupyterLab extensions
• [WORKFLOWS:Search&Discovery] Intake catalog search and integration with DASH
• [WORKFLOWS:Checkpointing] Workflow caching and chaining workflow elements together (xpersist and beyond)
• [WORKFLOWS:Publication] Binder (or similar) on HPC and deployment on Cheyenne/Casper

And there are probably many others, but this is just a start. Those having new ideas, please post them here. Those wanting to take any idea above and flesh it out with the Project Spec, please do so.

cc: @NCAR/xdev @NCAR/geocat

Original Post from Notes

Note the “Workflows” Campaign Spec in the xdev-projects repository.

Within the context of the Workflows Campaign, “Analysis computations” (or “Operators” as Matt Long calls them) are operations performed on scientific data (possibly calculating new physical quantities) during the “data analysis” phase of the scientists Jupyter-based workflow.

In light of GeoCAT’s new growing effort, it makes sense that these “Operators” should exist in the GeoCAT package. Xdev’s role in this should be to assist in identifying, porting, testing, and developing these Operators for the GeoCAT package. But the primary responsibility for maintaining these operators will be GeoCAT (even if we help). And that leaves Xdev with an opportunity to shift responsibility to other parts of the Workflow Campaign, including “Search & Discovery”, “Ingestion”, “Check-Pointing”, and “Publication”.

To that end, I am “greenlighting” efforts to develop new JupyterLab extensions, dashboarding, prototyping, etc., that can assist scientists with their workflow, (i.e., addressing pain-points) rather than just help them perform “big data calculations.”

NOTE: Training people in the Pangeo stack is still an Xdev campaign, so we will still help people figure out how best to do their analysis. However, I just don’t think we should be devoting as much of our time to building and maintaining the “analysis toolbox” packages.

Thoughts?