astropy / astropy-tutorials

Tutorials for the Astropy Project

Astropy Tutorials

This repository contains tutorial IPython notebooks for the Astropy project. These are typically longer-form, more narrative presentations of functionality in the Astropy core package and any affiliated packages. The tutorials are therefore different from the Astropy core package documentation, which presents a more structured and exhaustive view of the Astropy core package.

View the tutorials rendered as HTML pages

To see the tutorials rendered as static web pages, see the Learn Astropy website.

To execute the tutorials interactively, you can either clone this repository to your local machine or use Binder to run the tutorials remotely, as described below.

Run the tutorials locally

To run the tutorials locally, you should start by cloning this repository with git or downloading an archive of this repository from GitHub. You will need to have Jupyter notebook and IPython installed and will need to install the tutorial dependencies specified in requirements.txt:

python -m pip install -r requirements.txt

To check that your environment is set up to run the tutorials, you can use the Makefile provided in this repository with the custom envcheck command:

make envcheck

If this line fails, your environment is missing packages, and you should run the pip install line shown above.

The notebook files themselves live in the tutorials directory of this repository, organized by the names of the tutorials. These can be opened with Jupyter notebook as with any other notebook files.

Run the tutorials on Binder

You can also get started with a remote environment to run the tutorial notebooks in your browser using Binder

Contributing tutorial material

We are always interested in incorporating new tutorials into Learn Astropy and the Astropy Tutorials series. We welcome tutorials covering astro-relevant topics and they do not necessarily need to use the Astropy package in order to be hosted or indexed here. If you have astronomy tutorials that you would like to contribute to this repository, or if you have a separate tutorial series that you would like indexed by the Learn Astropy website, please see the Contributing Guide on the Learn Astropy website for information on how to get started.

astropy-tutorials's Issues

Set defaults for tracebacks?

I wonder if we should alter the default tracebacks in notebooks to be somewhat less verbose? I'm mostly thinking of http://www.astropy.org/astropy-tutorials/Quantities.html here, which has a number of places where the tracebacks are intended, but are lengthy and distracting - it would be better to just show the error and not the traceback at all, in fact.

This could be changed easily enough just for that one tutorial, but it might also make sense to do this as a general thing? What do you think, @adrn

Issues in catalog plotting tutorial

I can help fix these, but don't have time, so just to make sure we don't lose track:

• latitude grid lines are missing in final plots
• longitude should increase to the left!

Set up automatic travis-based build of the tutorials page

Right now someone has to manually run the deploy script to generate the gh-pages branch. We should be able to use travis to install astropy from conda, use that to build the tutorials, and then post them to GitHub pages. An example of a ruby project that does something like that is here... I'm pretty sure we could do something fairly similar.

@adrn or @astrofrog, was there a specific reason we didn't do this previously, or was it just that we didn't know how?

Write a template tutorial

Once we decide on exactly how to structure the repo, we need to write a template tutorial with instructions to encourage other users to contribute to this repository.

Add "edit this page on github" to tutorial pages

See title.

Automatically add authors/date to notebook headers

@adrn, what do you think about updating the build script to automatically add the names of the authors and the date metadata to all the notebooks? That way we can make sure they're all shown consistently...

Add Disqus comments at the bottom of each tutorial

It would be reasonably easy to update the setup.py to inject a Disqus comment thread at the bottom of each notebook. What do you think?

Deprecate repository?

Now that the tutorials are in the main docs, should we deprecate this repository?

We should make sure we keep track of the existing issues and pull requests though, so we should not remove the directory yet, only mark it as being deprecated?

cc @eteq @kelle @adrn

Add a script for checking that requirements are met for running tutorial

For the SciPy tutorials they asked as to prepare a script to verify that the required packages were available to run the tutorial. The result was:
https://github.com/astropy/astropy4scipy2014/blob/master/check_env.py

This was thrown together very quickly, but something along these lines might be a good idea for the Astropy tutorials as well. It might make sense to have a check_env script for each tutorial since the requirements for different tutorials might be different.

Note that we uncovered an apparent bug in distutils.version.LooseVersion in that it fails to support comparison of versions that contain strings (like dev) in Python 3. Hence the current check_env.py script just does a string comparison of versions, but this would need to be fixed.

Change plotting font?

I was wondering whether it would make sense to switch the font to a 'modern' font? While the current 'retro' look works in some cases, I'm a little worried that it gives a PGPLOT look to some of the plots which might make people think that the plotting package is old. Matplotlib can make really nice modern-looking plots, so it'd be a shame to miss out on that. Using Arial (with bold axis labels) works nicely for example.

Set up continuous integration for testing?

Inspired by a discussion on the Facebook Python in Astronomy group, I was wondering if we can set up Travis to run pytest-ipynb (https://pypi.python.org/pypi/pytest-ipynb) on the notebooks to make sure they don't break with astropy updates?

Astropy tutorials on Docker / tempnb?

With Docker and tempnb it should be possible to make the Astropy tutorials available online ... any user can run the notebooks from any computer without needing to install things locally.

Check out https://tempnb.org for a demo and https://lambdaops.com/ipythonjupyter-tmpnb-debuts/ for some more info on how it works.

Here's what needs to happen:

• Make a free account for the Astropy organization here: https://registry.hub.docker.com/search?q=astropy (@astrofrog or @eteq: could you do this and give me access?)
• Make one or a few Docker containers (e.g. one with Astropy only, then one with the prominent affiliated packages installed, and one with the tutorial included) and put them on the docker hub (I'm interested in setting this up and maintaining it.)
• Find someone willing to deploy this or give some money so that we can deploy this in some cloud (maybe we could ask Numfocus or Rackspace).

This could be nice to have in the Astropy 1.0 release announcement.

On tutorial front-end, figure out organization

On the tutorial front-end site, we should offer a few different sort options for the tutorials -- e.g., by type of tutorial (data reduction? data analysis? etc.), data used (SDSS, 2MASS, etc.), astropy subpackages used, and "difficulty".

Use nicer mpl default plots

(as requested by @adrn)

Use a nicer default set of matplotlib parameters, e.g. http://tonysyu.github.io/mpltools/auto_examples/style/plot_ggplot.html

Fix up redshift plot tutorial

As discussed in #85, there were some issued uncovered with the cosmology plotting tutorial after it was merged. @nhmc, are you still planning to do update this? It looks to me look the outstanding issues are:

• Address the problem of the two different cosmology axes not matching, perhaps by demonstrating the current version as a "gotcha".
• change the name to redshift_plot or something similar (unless others are against that - this was my suggestion in #85)
• While testing this locally (via python prepare_deploy.py run convert after setting "published" to true on this tutorial), the first image comes out as a broken link. I think this is because nbconvert doesn't copy over images that aren't embedded in the notebook. @adrn, do you know how to make this happen? Alternatively, a cell saying IPython.display('image.png') will embed the image in the notebook, and that will copy over correctly.
• Change the "published" marker from false to true (I set it to false until we can address this).

UVES tutorial indentation errors

There are some indentation errors at the end of the UVES tutorial.

Make a reproject tutorial

@astrofrog – How about a tutorial IPython notebook on reproject to make it easier for people to find it and try it out?

I could take the main examples from http://reproject.readthedocs.org/en/latest/ and do this today.
Keep it short and link to the reproject, wcsaxes and healpix docs at the end.

High level table operations Tutorial

High level table operations - database-y things.
Table with list of multiple observations for different objects. filter by groups. select based on phot, or calculated operations. TOM A. + someone at CfA
@taldcroft

Figure out why conda isn't generating the tutorials right

#102 implemented machinery to use conda to auto-deploy the tutorials when something gets merged into master. Turns out it was a good idea to start with a test branch (https://github.com/astropy/astropy-tutorials/tree/test-gh-pages) before doing it "live": for some reason travis is having issues with any astropy >1.0.4 (or maybe it's numpy?).

The trouble started in 154432e - before that the auto-deploy worked fine, but it was giving some spurious warnings that were fixed in 1.0.5 - so 154432e updated astropy to the latest stable. However, that required updating numpy to 1.10 (because conda doesn't build newer astropys against numpy 1.9). And with that change it's all broken.

You can see the result by looking at https://github.com/astropy/astropy-tutorials/tree/test-gh-pages (which for now I've put up at http://eteq.github.io/astropy-tutorials/ for quicker inspection). You can see that almost all the tutorials are failing with something like RuntimeError: module compiled against API version a but this version of numpy is 9.

So if I'm reading that right, the baseline problem is that for some reason Travis is importing numpy 1.9 even though conda happily installed and seemingly activated 1.10.

Note that this is also happening in the regular travis builds not just the deployment, although it's not obvious because they don't actually fail. But if you look in e.g. https://travis-ci.org/astropy/astropy-tutorials/builds/94774614 you'll see that even before deployment, the tests run like this:

WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
---------------------------------------------------------------------------
None                                      Traceback (most recent call last)
None: None
[IPKernelApp] WARNING | Eventloop or matplotlib integration failed. Is matplotlib installed?

I strongly suspect this is the same problem.

@mwcraig @bsipocz @astrofrog @adrn, anyone have any ideas?

tutorial to fit a Sersic profile to a galaxy image

this idea came from DotAstro in Oxford. i dont know if it should be part of a larger tutorial or if it's enough to stand on its own.

Big Data Handling Tutorial

Big data handling - I have a data file (FITS image, big ascii table) that i want to manipulate that is a few gigabytes...reading it in, not screwing up memory, doing operations, writing it back out again.

pros and cons vs IRAF vs CASA
read in FITS in parts. manipulate, write back out in parts?

@keflavich & @demitri

Travis-based auto-deploy machinery

I thought there was already an issue for this but I can't fine it (maybe @adrn can confirm?). It would be great if we can make travis automatically deploy the "published" tutorials.

I think this may be pretty easy to do using https://github.com/ewanmellor/gh-publisher - (which I stumbled upon thanks to http://jakevdp.github.io/multiband_LS/)

All Sky Projection Snippet

This is a great snippet...plot a bunch of RA/Dec points in a 2D projection. It's already included in one of the Tutorials, but needs to be updated based on some keyword change in astropy.coords.

clairification on FITS tables tutorial

Could the rationale behind the step to create a boolean mask (line 10) and a definition of "zeroth order events" please be added? it could also use a bit more info about what tg_m is. It becomes obivous later that tg_r and tg_d are Ra and Dec but tg_m remains mysterious. These steps and variables have implications for the last exercise and I don't get it.
@eblur

How to publish these tutorials?

By "publish" I just mean "make visible on the web".

@eteq and I discussed this a bit when he was at Columbia last month, but I think there are still some open questions. I think we concluded that we could point to them from tutorials.astropy.org, but then the remaining questions:

• The tutorials are written in ReST but since they live here (in a separate repository), what's the best way to incorporate them into the main site?
• Do we create a separate Sphinx project for this repo?
• Can we set it up so the tutorial code can be run with doc tests or even main astropy tests?

@eteq @astrofrog @kelle @jfoster17

Adding RA/Dec to plot using WCS

Would it be possible to update the tutorial on viewing and manipulating FITS images to show how to display the RA/Dec coordinate grid on an image, by retrieving the WCS coordinates from the header?

VO Cone Search Tutorial

perform a code search around a short list of targets.
@pllim

Quanities Tutorial

Compute a mass of a galaxy given blah blah. Erik + Ana
@eteq

Add some README instructions for deploying and/or testing build

It would be useful if the README included instructions for how to deploy the tutorials, or at least some info on how to "test" deploy them (e.g., do python setup.py run and python setup.py build and then look in html). A few surprises for me: that runipy is a dependency, and that pandoc has to be installed for nbconvert to work. So just noting that would save the trouble of looking at the cryptic python errors to find what needs to be present...

I was going to try to just write this up myself, but I think it might be better if @adrn do it, as he better knows just what all needs to be in here.

Fit models to 2D scatter plot Tutorial

it a straight line/polynomial/power law/gaussian chi^2 min to a 2D scatter plot.

• fit a Sersic profile to a galaxy image

EG: Markwardt fitting table, matplotlib, modeling - NADIA. + staff/student
@nden

Cosmology Distance Snippet

Convert redshift to Mpc and the like. @parejkoj

Write astroplan tutorial

We should show off the new features in a demonstration of how to prepare for an observing run. Rough sketch:

• Read in target list from text file with astropy.io.ascii
• Query catalogs to find nearby standard stars
• Plot visibility curves (airmass curves) for each object

If we want to go crazy, we could even optimize over slew time and "time-til-transit" to auto-generate a queue...

FITS header manipulation Snippet

Read a FITS file, edit header and write it back out. Already done but needs to be turned into a ipython notebook and committed.
@jfoster17

Binder link broken

If I click on the launch binder in the README, I get:

Cannot GET /repo/astropy/astropy-tutorials/tutorials

Bug with RA increasing to the right

This plot in Quantities is wrong:

The RA should increase to the left! Same goes with the plot-catalog tutorial:

Add a prominent mention of the snippets and how they relate to the tutorials

When/if the snippets gallery in the astropy docs is merged, the astropy-tutorials front page should be updated to mention the snippets and describe what the distinction is between snippets and tutorials @adrn @kelle

Improve FITS header tutorial

Over in this comment I griped about the current state of the FITS-Header tutorial. However, I was pretty vague as to what I don't like about it. And actually, upon a second look, I don't have as much problem as I thought I did. As a very introductory tutorial I think it's fine to promote the "convenience functions", since this will be a lot easier for many newcomers to wrap their heads around.

I think I would just like to make a few slight tweaks, and also to at least mention the more "preferred" object-oriented interface, with appropriate links to the more advanced docs?

Add content to VO conesearch tutorial

Right now, the tutorial reads a bit like documentation and is primarily just focused on conesearch (and some Quantity). I don't think it would be much work to add a few things to highlight synergies with other astropy subpackages and make it feel more like a workflow.

A few ideas:

• Add a section where you write out a catalog to disk using astropy.table
• Query a catalog and turn it into a Latex table
• Use aplpy and overplot circles marking known sources with some constraint over an image
• Use an SDSS query to get positions for sources, then query the SDSS SAS (http://dr12.sdss3.org/) to get back a few spectra to plot

Tutorials should provide link to data

It would be nice if tutorials could be reproduced by users, which means that they need to be able to access the data. For example, the following tutorial does not:

http://www.astropy.org/astropy-tutorials/FITS-header.html

Small suggestions for notebook style

Just a couple of low-priority suggestions for the style of the notebooks:

• Change the width of the notebook to 900px (instead of 960px)
• Change font size to 16px (including the code)
• Make links blue (currently black with blue underline)
• Make more of the method names links to the API docs
• Remove last empty cell

Coordinate Conversion Tutorial

Read in an IRSA table, convert to galactic, output as LaTex table.
@mdboom

Make cosmology tutorial image download a bit simpler after IPython fix

ipython/ipython#9022 fixes an issue that required a slightly awkward workaround in #97 (see #97 (comment) ). Once the next ipython is released, we may want to consider just using that to simplify the cosmology tutorial.

But it would then require a new IPython to work, so we might want to wait for a while until that IPython becomes more standard...

How do we deal tutorials that are already hosted elsewhere?

So far, this repository is fairly empty, but there are already several astropy tutorials "in the wild".

@astrofrog wrote on if it:
http://astropy4mpik.readthedocs.org/en/latest/index.html

I made a slightly modified version:
http://python4astronomers.github.io/astropy/astropy.html

A second tutorial, that focuses on other other aspects of astropy is here
http://python4astronomers.github.io/astropy-UVES/UVES.html

(or, if you want to see the source code here:
https://github.com/python4astronomers/python4astronomers )
In that case I tried to make it a consistent story "How did I get from the data to the publication using astropy?"

How do we handle those tutorials? I see two options: Either I open a pull request to this repository and we keep everything in here (and I set links from python4astronomers) or we keep them separated and set links from here to python4astronomers.

In both cases they can be found, used and forked.

@adrn @jfoster17 What do you think?

WCS Snippet

Load a fits and get WCS out. pixel to RA/Dec for thousands of points. and the other way around.
Nadia Dencheva [email protected]. @nden

Coord matching Tutorial

cross matching two tables. - ERIK T. + Ana
@eteq

Improve fonts in CSS

At the moment, the font sizes look a little inconsistent in the tutorials:

I think it would be good to have someone go though and try and polish up the CSS a little to make font sizes more consistent, and improve spacing, etc.

Maybe you could ask on the astropy-dev list if any CSS experts would be interested in helping?

Make a ccdproc tutorial

a draft exists, based on Phil Massey's classic work, but needs follow through. started by @kelle and @crawfordsm. Maybe @mwcraig can take the lead.

add a photutils tutorial

started by @JenniferKarr and @bsipocz

Time Snippet

Covert EST to UT to JD.
maybe @demitri ?

tutorial versioning/testing

It would be useful to have a list of astropy versions for which each tutorial is known to work. I have tried to modify one of the tutorials to work on the current stable release thinking that it was written for a previous release, before I noticed that it runs fine with the latest development version. Maybe not very important now that things are still changing but at some point it would be good to show in the tutorial index which astropy versions have been tested and are supposed to run without problems.