esmvalgroup / esmvaltool_tutorial Goto Github PK
View Code? Open in Web Editor NEWESMValTool Tutorial
Home Page: https://tutorial.esmvaltool.org/
License: Other
ESMValTool Tutorial
Home Page: https://tutorial.esmvaltool.org/
License: Other
### Prerequisites (Should be in top-level lesson page)
- [ ] Server specific information (Jasmin, …)
Branch: installation
PR: #31
In the last section of setup page (Install conda) the link to installation episode is broken.
see the last line:
More details on this process are available in the Installation episode.
Currently the ESMValTool tutorial is called "ESMValTool Introduction". It says so at the top of the title page and at the top of each lesson. This also means that when users are starting section 01_introduction, the see at the top of the page:
ESMValTool Introduction
Introduction
I think this is not the best name for this tutorial since there are also advanced lessons at the end. Also this issue might be related to #22 since we might want these names to match.
Currently the pages are made with an eScience Academy theme, but this should be made so it looks more like ESMValTool. Also the repository is full of references to the Software Carpentries. We should probably give them credit for the original template somewhere, but in most places these mentions should be removed.
I started working on episode 03 in the current structure that is about configuration. Also, I made a draft pull request for that.
Checklist:
There is a typo in the second objective in episode 02-installation: Demonstate that the installation was successful
The word Demonstate should be fixed.
I believe the full path to the diagnostic must be given if we run the tool with the conda installation so that should be amended in the text and/or the recipe as needed.
it needs:
Short description of the material
This is the title page.
Branch and pull request
We'll be working in a google doc first. Please contact @ledm if you want access to the google doc.
The PR is in #38.
Title page:
As a contributor the Codacy warnings are hard to resolve. I need to wait for Codacy analysis to finish to see if I fixed the warnings and sometimes I flipflop between issues. It would be useful to me if there where instructions in the CONTRIBUTING guide how to run the Codacy analysis locally.
This problem was raised by @nielsdrost
This issue is to discuss changes needed to the file Contributing.md which has instructions for adding new material to the tutorial. This issue discusses some specific details with regards to cloning the repo and installing necessary packagaes and such.
I want to bring this forward and have a bit of discussion around it today when we'll all be in at the sprint (at 1pm CET). A few points I'd like to raise:
Having said that, I believe we need to dedicate a lot of effort to finding out what's the best way to construct a first-pass, quick user guide that has a few key attributes:
I am fairly sure if we don't get this sort of thing done first, then point the users to this tutorial and the documentation, they will not even bother looking at the tutorial, skim the documentation and next thing we know we get a dozen or so gitHub issues with questions like 'where do I get data from'. Anyways, up for discussiong this more later on 🍺
It might be a good idea to add a couple of sample config-user.yml files to extras or some such directory with each file suited to a specific server such as JASMIN or DLR's. We should probably discuss this during the Lesson Polishing stage.
The initial branch "gh-pages" is changed to "master". There are two points that need attention:
1- lesson template
According to Software Carpentry guide, using the gh-pages branch of the lesson template will ensure that the most “stable” version of the template repository is used.
2- tutorial webpage on github pages
According to the manual on creating gh-pages, the site must be on the gh-pages branch. GitHub will not care about the master branch, which is the default branch in git.
@bouweandela @ledm I am not sure whether these are already fixed.
"How to cite" has appeared in
the readme,
Index,
introduction, last line in the section what is esmvaltool.
episode7-conclusion
and about page.
It appears too many places. We should just add it in an extras page, and link to it in those other places.
I want to add a pull request template for this repository.
(Following #29, #34)
The left over issues of the recipe tutorial lesson "episode 4" are the following:
The https://esmvalgroup.github.io/tutorial/setup.html (after #46 is merged) shows that if I want to use my own machine then I should be able to download CMIP. However it does not mention how to do that. It seems synda is involved, but there is no description what it does or how to install and configure it.
It would be nice to install synda in this section. Something like instructions at https://docs.esmvaltool.org/en/latest/input.html#models
Working with preprocessors
Description : More complex example- optional. This example includes a more complex preprocessor chain, perhaps turning a 3D file into a smaller regional average, but outputting a time series plot again)
Internal name : preprocessor
Tentative structure and plan :
Questions:
Objectives:
Contents:
Discussion: Inspect the example recipe
Exercise: edit the file to add a depth average preprocessor and run it
Exercise: Make a minor edit to the recipe such as:
Callout: How to find what CMIP data is available:
- Link to the CMOR pages where users can find what variables exist
- Link to ESGF page to see what data has been submitted.
Key points:
Any additional things that cannot be done or is unsupported at this time
@bouweandela, is there any way that we can make a local build of the tutorial to test it before making a PR?
If there is, can we include it in CONTRIBUTING.md
This Issue summarises and build on the the Discussion at the ESMValTool core developers workshop on 11/03/2020.
To summarise the discussion:
We're not confident that EGU 2020 will go ahead. So instead of preparing for a meeting which may not happen, we will produce a tutorial that can be taught online, in person, as a webinar, or could even be done without the help of a an ESMValTool expert.
There is already a skeleton Tutorial in the e-science academy and we can use this as a basis for the ESMValTool online tutorial:
Once this repository has been set up, Lee, V and Ranjini (if she's free) will populate it with our tutorial resources.
The final tutorial webpage can be hosted on github pages, but we want to make sure we don't add another address to the long list of ESMValTool website. (ESMvaltool project page, ESMValTool github page, readthedocs, the GMD paper, etc...)
A potential issue with these kind of tutorials is that students may get used to the virtual box or jupyter environment and not be able to move to a "real" environment. We must include instructions on how to transition to a typical ESMValTool conda environment at the end of the beginner tutorial.
Personally, I think that this method, if successful, would be a significant improvement of the current ad hoc tutorial methods. Also, this could hopefully drive many more first time users than we could ever attract at 8:30am on Monday morning of the EGU during a viral outbreak!
This translate to the following actions:
Please add additional actions.
Lessons episodes 04, 03 and 06 should use the same editor vim or nano. And in episode 02, in the section install julia, where it says: Open the file in your favorite editor, should be changed to one of the text editors.
In the espisode 04-recipe, there is an item about model grid as:
- model grid (for CMIP6 data only, key: grid)
It is helpful to make a link to esmvaltool documentation on this matter, see https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html?highlight=irregular%20grid#recipe-section-datasets,
where it explains that
model grid (native grid grid: gn or regridded grid grid: gr, for CMIP6 data only).
Also, add a note that: "CMIP6 ocean data is on native tripolar grids."
Internal name: first_recipe
The outline document says:
Questions:
Objectives:
Contents:
Discussion:
Discussion: Inspect the output:
Exercise:
Exercise:
Callout: Common issues & tips
Key points:
To facilitate contribution, I want to add four issue-templates for:
The pull request is here #21
We'll need to work on the Readme file too.
However, this file isn't really that useful. It's just the cover page for the tutorial gitpage.
It should include:
The name "tutorial" is not quite descriptive, especially when this repo is forked. It will be good to consider renaming, some names are already suggested in #3 : ESMValTutorial, ESMValTool_Tutorial, ESMValTraining, ... .
I think that "ESMValTool_Tutorial" is fine.
We can use the same in esmvaltool repository .
The current citation information on the title page line 81 - 97 requires some reviewing and consideration:
This is a very short conclusion to the main tutorial, and explains what the subsequent mini-tutorials are.
After this mornings discussion, we're still rather unsure about what needs to go into the setup.md file. However, I will soon make a branch/PR with our current text, and this can be the point of reference for this discussion
In the software carpentry, there's a drop down menu labelled extras.
At the moment, this contains:
about.md discuss.md figures.md guide.md
I'm not sure that we need all of them, but an about.md page would be nice, and a page of instructor notes in guide would also be handy.
I don't think we'll need the discussion page, but the figures.md page may be useful for someone one day.
Describe the bug
ESMValGroup/ESMValCore#605 changes the command line interface for the v2 release. It would be good to align the tutorial with the new interface, i.e. use for example:
esmvaltool recipes list
esmvaltool recipes show recipe_albedolandcover.yml
esmvaltool recipes get recipe_albedolandcover.yml
esmvaltool config get_config_user
esmvaltool config get_config_developer
esmvaltool run examples/recipe_python.yml
esmvaltool run --config_file config_user.yml recipe_python.yml
esmvaltool run recipe_albedolandcover.yml --write_plots=False
Note that
get
commend to copy recipes from the package to your local working directoryLast week, we also discussed setting up virtual machines, using a ready-made conda on jasmin, and we have also discussed a jupyter notebook. These are supposedly easy way for people to get started on ESMValTool at face-to-face tutorials without having to install ESMValTool.
None of this appears in the tutorial at the moment, so it would be good to set something. I don't know anything about that though. Can someone volunteer to write something in 02-installation.md? @valeriupredoi 🍻 if you do.
The second question is: this is useful for a face to face set up, but less so for a DIY try out ESMValTool yourself at home setting. I guess we should include both options in the installation page, but be clear that one is for tutorials and one if for "real" use.
Once the governance document is agreed on, I think a User Engagement Team group should be made in the ESMValGroup organization and that team should be the owner of this repository.
There are several places where we want users of the lesson to be able to contact the tutorial development team. I suppose this should be the ESMValTool user engagement team. Does this team already have a specialized email address or one specified contact person of which we could add the email address?
Places to add the email address:
_config.yml:33
CONTRIBUTING.md:164
index.md:65 (this line now specifies the esmvaltool mailinglist, but this might not be the best place to point users of this tutorial)
@ledm, @BenMGeo, @rswamina, @hb326, @LisaBock, @Peter9192, @bouweandela, @nielsdrost, @valeriolucarini, @JaroCamphuijsen
Hi all,
As discussed in the esmvaltool monthly meeting, we will plan a one-day meeting to talk about topics related to the tutorial and work on some issues. The meeting covers:
If you want to discuss other topics that are not listed above, please mention them in this issue.
We kindly invite you to join the meeting. To decide the date, please participate in the doodle poll the link below by Next Monday 13th July. Thank you.
https://doodle.com/poll/fprm28ux4xah4eww
#55 implemented an alternative quality check to replace the codacy checks. Some of the checks that we have previously ignored because of all the confusing codacy fuzz now surface again, and need to be fixed:
Episode ./_episodes/conclusions.md has badly-formatted filename
Missing or non-consecutive episode numbers [1, 2, 3, 4, 5, 6, 7, 10]
./CONTRIBUTING.md: Line(s) too long: 3, 7, 12, 13, 30, 43, 44, 48, 49, 57, 61, 64, 77
./README.md: Internally-defined links may be missing definitions: "issues"=>"FIXME"
./README.md: Line(s) too long: 3, 5, 9, 12, 13, 14, 18, 20
./_config.yml: configuration carpentry value ea is not in ('swc', 'dc', 'lc', 'cp')
./_episodes/01-introduction.md: Line(s) too long: 26, 45, 47, 49, 51, 53, 55, 57, 59, 63, 67, 73, 78, 80, 82, 83, 90, 94, 98, 104, 108, 114
./_episodes/02-installation.md: Line(s) too long: 28, 32, 64, 97, 99, 116, 172, 181, 182, 183, 196, 212
./_episodes/03-configuration.md: Line(s) too long: 26, 109, 130
./_episodes/03-configuration.md:80: Unknown or missing code block type language-YAML
./_episodes/03-configuration.md:98: Unknown or missing code block type language-YAML
./_episodes/03-configuration.md:119: Unknown or missing code block type language-YAML
./_episodes/03-configuration.md:140: Unknown or missing code block type language-YAML
./_episodes/03-configuration.md:164: Unknown or missing code block type language-YAML
./_episodes/03-configuration.md:192: Unknown or missing code block type language-YAML
./_episodes/03-configuration.md:202: Unknown or missing code block type language-YAML
./_episodes/05-preprocessor.md: Line(s) too long: 22, 26, 30, 39, 41, 45, 51, 53, 55, 58, 62, 81, 123, 225, 277, 338, 342, 344, 346
./_episodes/05-preprocessor.md:32: Unknown or missing code block type language-yaml
./_episodes/05-preprocessor.md:47: Unknown or missing code block type None
./_episodes/05-preprocessor.md:66: Unknown or missing code block type language-yaml
./_episodes/05-preprocessor.md:84: Unknown or missing code block type language-yaml
./_episodes/05-preprocessor.md:127: Unknown or missing code block type language-yaml
./_episodes/05-preprocessor.md:172: Unknown or missing code block type language-yaml
./_episodes/05-preprocessor.md:228: Unknown or missing code block type language-yaml
./_episodes/05-preprocessor.md:280: Unknown or missing code block type language-yaml
./_episodes/06-debugging.md: Line(s) too long: 17, 63, 91, 103, 149, 174, 210, 211, 219, 247, 280, 281, 286
./_episodes/06-debugging.md:126: Unknown or missing code block type language-YAML
./_episodes/06-debugging.md:178: Unknown or missing code block type language-YAML
./_episodes/06-debugging.md:192: Unknown or missing code block type language-YAML
./_episodes/06-debugging.md:256: Unknown or missing code block type language-YAML
./_episodes/06-debugging.md:269: Unknown or missing code block type language-YAML
./_episodes/06-debugging.md:290: Unknown or missing code block type language-YAML
./_episodes/10-development-setup.md: Line(s) too long: 16, 20, 21, 52, 58, 82, 93, 116, 117
./_episodes/conclusions.md: Line(s) too long: 20, 32, 97
make: *** [Makefile:135: lesson-check-all] Error 1
@rswamina, @hb326, @bouweandela, @valeriupredoi, @nielsdrost, @zklaus, @mattiarighi, @axel-lauer, @ledm, @Peter9192
Hi all,
We want to plan a Tutorial sprint of 3 days (virtual). The sprint covers lesson developments (adding the existing material to the repo), refactoring the episodes and reviewing the pull requests. Specifically, we will focus on the working items discussed in the shared google document.
We kindly invite you to join the sprint. To decide the date, please participate in the doodle poll the link below by Next Monday 8th June. Thank you.
https://doodle.com/poll/ykurzky6scdna59m
If I missed someone from the team, please mention him/her here.
In the section the-configuration-file, where it is mentioned that:
It will save the file to: {HOME}/.esmvaltool/config-user.yml.
Make extra note that the folder .esmvaltool is hidden and can be found via running command ls -la
in the HOME dir.
The initial tutorial structure is:
Are we happy with that structure?
If the tutorial is going to use github pages or a virtual box, then perhaps we could move the installation section later? It would be nice to get into doing science as early as possible.
In _episodes folder:
To do:
In episode 03-configuration, section Rootpath to input data, there is a callout Setting the correct rootpath:
In the section What next in episode 07, there are several links to other episodes. They should be fixed.
I should start saying, great job you all! At first glance esmvaltool can be quite intimidating (I had no clue where to start) but the tutorial got me going and able to make myself familiar enough to try it out. It meets most of issue #52 except the data package (more comments on that). I have not gone extensively throughout the whole tutorial though and need more time get better insights but hope to contribute with a few perceptions below. :)
Potential troublesome points
Data Access - I believe the access to the CMIP data is a big part of ESMValTool application itself and is not something trivial when dealing with synda/esgf-pyclient as discussed on #54, but I agree the tutorial is the place for making operational and not really focusing on acquiring data. I would then highly recommend shipping a CMIP sample data used in the tutorial. Unfortunately, I couldn't manage to get access to the data note but I had few CMIP data and also downloaded random alike datasets along the way to be able to reproduce some examples. That was the major issue of not going through the entire tutorial.
gn/gr - when trying to adapt with the datasets I had on hand I realise most of the structure of the examples are based either on gr data or are remapped on preprocessing. CMIP6 ocean data is coming heavily on native tripolar grids (which I think was not the case for CMIP5). As changing preprocessor orders or even the dataset/variable (eg when using area/zonal statistics) will cause it to break it might be a good idea to put an explicit note on that maybe? It can be sorted out looking at the log and going through the Docs, finding that iris function can't deal with multidimensional coordinates/irregular grid but a note will be helpful :) ) #82
Roothpath structure - I spent some time trying to sort out how exactly directory structure for local files should be like (not on nodes). I found useful the 'Docs > Getting Started > Finding data' note on "having all data in the same top-level directory". That makes it clear, I think it was just the broken link's fault on having me spending time to get there, fixing it should let it more straight-forward. #83
Typos & Broken links
I'm not quite familiar with github protocols and I'm aware too much writing is not one of them (trouble on synthesizing) but hope it is of any help. Huge thanks for the 138ppl involved on ESMValTool and for the tutorial which was a crucial kick-off for me!
In the software carpentry, there's a drop down menu labelled extras.
At the moment, this contains:
about.md discuss.md figures.md guide.md
I'm not sure that we need all of them, but an about.md page would be nice, and a page of instructor notes in guide would also be handy.
I don't think we'll need the discussion page, but the figures.md page may be useful for someone one day.
contributing.md
should explain why our lessons are designed the way they are and how others can contribute to the tutorial.
The current contributing.md file needs several improvements in terms of the structure, content, links and guides. I improved it because this promotes user engagements.
The pull request for this issue can be found here #23 , which is ready for a review.
I'd like to ask one of the members of the User engagement team (@Peter9192, @rswamina, @hb326, @bouweandela, @ledm ) to review this pull request. If you are willing to do it, please assign yourself as the reviewer in that pull request. Thank you in advance for your helpful comments.
If I missed someone from the team, please mention him/her here.
This is the installation script I needed to set up a tutorial conda:
conda create --name tutorial
conda activate tutorial
conda install gxx_linux-64
conda install -c conda-forge ruby
conda install -c conda-forge rb-nokogiri
make serve
Presumably, it's very easy to convert the conda commands into an environment.yml
file.
We need to update time estimates for "teaching" and "exercises" in each episode.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.