planetarypy / planetaryimage

Python PDS and Isis Cube file parser.

planetaryimage - Open PDS and Isis CubeFiles in Python

NOTE This is Alpha quality software that is being actively developed, use at your own risk.

Planetary image parser

• Free software: BSD license
• Documentation: https://planetaryimage.readthedocs.org.

Features

• Reads in PDS Images as NumPy arrays.
  • Supports GZIP and BZ2 compressed PDS Images.
  • Supports writing out PDS3 images.
• Reads in Isis Cube Files as NumPy arrays.

Check out a few simple examples of opening and viewing PDS and Isis CubeFiles in an IPython notebook.

Quickstart

The example below will walk you through setting up a Python virtual environment and installing the necessary software as well as a few handy extras. It then downloads a sample Pancam PDS image, opens and displays that image in your web browser in an IPython Notebook. The example assumes you have Python, virtualenv, and pip installed on your system. If you don't, don't know what this means or aren't thrilled by the opportunity to learn what this means, this software may be a little too immature for you to use at this point.

Create and activate a virtual environment:

virtualenv venv
source venv/bin/activate

Upgrade pip, then pip install the package and IPython notebook and matplotlib to help display the image:

pip install -U pip
pip install planetaryimage matplotlib ipython[notebook]

This quick example will show how to open and display a Pancam PDS image using this module. First, grab a sample image:

wget http://pds-imaging.jpl.nasa.gov/data/mer/opportunity/mer1po_0xxx/data/sol2840/edr/1p380322615effbr43p2443l1m1.img

Now run python in an IPython Notebook (a browser window should pop up after entering the following command):

$ ipython notebook

Create a new notebook in your web browser and then paste the following code into a cell and execute it by pressing Shift+ENTER. This will load and display the image:

%matplotlib inline
import matplotlib.pyplot as plt
from planetaryimage import PDS3Image
image = PDS3Image.open('1p380322615effbr43p2443l1m1.img')
plt.imshow(image.image, cmap='gray')

See Usage for full documentation on how to use planetaryiamge.

planetaryimage's Issues

Make sure that mapping table which maps DTYPE to SAMPLE_TYPE is correct

As per Sample_Types mapping table, we might have 1 dtype for multiple sample_types.

Make sure that, dtype mapping table has main sample_type which should not be an alias of other sample_type.

Make `planetaryimage` property to return Data property first if Data exists

We currently return property values from the label information. This will result into an error when user changes data properties. (e.g. Cropping an image).

Make property to return its values with priority listed below:
(1) Data Property (If data exists)
(2) Label Property (Required during decoding)

Make proper Python Module

Please accept my merge request and then make a properly formatted Python module out of what we have so I can build something and push it to pypi as soon as reasonable.

PDS3 Image decodes Lunar Reconnaissance Orbiter Camera Narrow Angle Camera EDR images as signed 8-bit images instead of unsigned 8-bit images

Problem: The LROC NAC EDR images contain 8-bit unsigned raw instrument DN counts, although the PDS3Image class decodes them to signed 8-bit integers. This leads to incorrect DN values in the imported numpy image array.

Reason: This is because the EDR binary files have "SAMPLE_TYPE = LSB_INTEGER" in their PDS3 header, which is interpreted by PDS3Image class as a signed integer.

Solution: I am unsure if this is an issue with the PDS3Image class mapping from SAMPLE_TYPE values to numpy dtypes, or if the EDR file header SAMPLE_TYPE values are incorrect. Potentially the SAMPLE_TYPE mapping should be updated, or a specific exception for NAC EDR images could be made.

Add write support for PDS3Image

There may be parts we will choose to extract up into the PlanetaryImage class, but you can start with PDS3Image.

The pixel_type code needs extending to handle more types

We only handle integer types. We should handle all valid types.

For example 32 bit float PC_REAL to support this:

RyanBalfanz/PyPDS#20

Also, the fact that we are using both a chain of if statements AND dictionary lookup seems a little redundant to me. I think the existing implementation can be changed to use nested dictionaries exclusively. Or at least nearly so.

Write some docs

We only have stubbed out docs, we should write something useful in docs/ and in README.

Error on opening files from Cassini

I am trying to execute
imdata = PDS3Image.open('./Data/02NASA_Cassini/N1454725799_1.IMG')
in my Jupyter notebook, and I see the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-ed1161877cab> in <module>()
----> 1 imdata = PDS3Image.open('./Data/02NASA_Cassini/N1454725799_1.IMG')

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in open(cls, filename)
     90         else:
     91             with open(filename, 'rb') as fp:
---> 92                 return cls(fp, filename)
     93 
     94     def __init__(self, stream_string_or_array, filename=None, compression=None):

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in __init__(self, stream_string_or_array, filename, compression)
    119 
    120             #: A numpy array representing the image
--> 121             self.data = self._load_data(stream_string_or_array)
    122 
    123     def __repr__(self):

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in _load_data(self, stream)
    198 
    199     def _load_data(self, stream):
--> 200         if self.data_filename is not None:
    201             return self._load_detached_data()
    202 

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in data_filename(self)
    172     def data_filename(self):
    173         """Return detached filename else None."""
--> 174         return self._data_filename
    175 
    176     @property

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/pds3image.py in _data_filename(self)
    299     @property
    300     def _data_filename(self):
--> 301         return self._image_pointer.filename
    302 
    303     @property

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/pds3image.py in _image_pointer(self)
    312     @property
    313     def _image_pointer(self):
--> 314         return Pointer.parse(self.label['^IMAGE'], self.record_bytes)
    315 
    316     @property

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/pvl/_collections.py in __getitem__(self, key)
    121         if isinstance(key, INDEX_TYPES):
    122             return self.__items[key]
--> 123         return dict_getitem(self, key)[0]
    124 
    125     def __delitem__(self, key):

KeyError: '^IMAGE'

My default Python evironemnt is

Notebook Executed:	 2017-09-12 21:39:34.026349
================================================================================
Python Version:
--------------------------------------------------------------------------------
3.5.3 |Anaconda custom (64-bit)| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
================================================================================
PlanetaryImaging Version:
--------------------------------------------------------------------------------
0.4.1 (also tried with 0.5.0 with the same error message)
================================================================================
PVL Version:
--------------------------------------------------------------------------------
0.2.0
================================================================================

I will try some debugging over night, but would really appreciate any help.

PlanetaryPy Moving Forward

This issue is a collection of my thoughts after connecting with the OpenPlanetary group last week. The point of writing it is so everyone else knows where I am headed with this work. How everything works out really depends on what other people may do. Please feel free to share your thoughts in the comments.

PlanetaryImage

At this point, the PlanetaryImage package supports reading the PDS mission/instrument/data product types I need supported for my day job so I will only be adding support for additional products in my spare time. The code is in an acceptable state, but there is some refactoring and cleanup that I would like to do and I would like to extend test coverage as well. There is also at least one ticket, #19, that I think might require a change to the current API, though not necessarily.

I do want to complete the task of identifying PlanetaryImage's support across as many mission/instrument/data products as possible over the next few weeks. This task includes implementing the planetary_test_data package and then writing a generic data driven test suite that validates all of the collected data products, #10. There are a few tickets already filed (#22, #18) that will add support for other data product types. As I said earlier, I will try to complete these in my spare time, as I have been doing, if no one beats me to them.

If there are people interested in supporting Isis Cubefiles, please advocate for improvements, send sample data and pull requests. I don't ever use Isis so I feel like that part of PlanetaryImage will need an advocate, because I know I won't be doing it justice. Along those lines, I have tagged issues that are low priority, as far as my needs are concerned, with the help wanted tag.

After I get through the test data collection and assessment I will start working on serializing PDS files. The first task will be to write an encoder for PDS3 in the pvl package. The current encoder, encodes to Isis cubefile style labels. Then I will implement the PDS3 full product writer in PlanetaryImage.

I think the last thing I want to say is that since we selected the BSD license we cannot require that people cite the project or any publication, but we can ask them to do so, elsewhere (in the docs, in the readme, etc). I think its a great idea to ask that people do so, but any language requesting it should be clear that it is not a requirement.

PlanetaryPy

There is the stub of a website for PlanetaryPy, (http://planetarypy.github.io/)[http://planetarypy.github.io/]. If anyone wants to do anything with that they can. I will eventually put up a simple static site (using jekyll) if no one else gets to it. The site would be brief and point to the projects.

gdal_pds

I took a quick look at my earlier package, gdal_pds, and it is very basic. Someone is welcome to dust it off, replace my broken label parser with pvl, reconstruct the package with our template cookie cutter and start playing with it. There's not much to it, so starting fresh would be reasonable. I do think that providing a consistent API between the GDAL based reader and PlanetaryImage is a good idea. This will make integrating them in a single package down the road easier.

pdsview

The existing pdsview package is very rough. I will leave it there for now since its functional enough to serve as a very basic viewer for PDS images. If someone wants to undertake a more functional viewer, please do! There is a possibility that I fork the existing project and giving the fork a mission specific name and start doing some Mastcam-Z specific implementation on that fork.

None.png is broken

Looks like a case issue

https://planetaryimage.readthedocs.org/en/latest/supported_planetary_image_types.html

Error on opening files from New Horizons MVIC data

I have been trying to read images from New Horizons MVIC data on a Linux OS (Ubuntu 20.04.2 LTS). I tried both adding .img and .lbl file but neither worked. Below the error:

from planetaryimage import PDS3Image
image = PDS3Image.open('....\mvic_0299178092_0x536_sci_pl.img')

or

image = PDS3Image.open('....\mvic_0299178092_0x536_sci_pl.lbl')

----------------------------for img file--------------------------------
KeyError Traceback (most recent call last)
in
1 from planetaryimage import PDS3Image
----> 2 image = PDS3Image.open('mvic_0299178092_0x536_sci_pl.img')

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in open(cls, filename)
35 else:
36 with open(filename, 'rb') as fp:
---> 37 return cls(fp, filename)
38
39 def init(self, stream_string_or_array, filename=None, compression=None):

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in init(self, stream_string_or_array, filename, compression)
75
76 #: A numpy array representing the image
---> 77 self.data = self._load_data(stream_string_or_array)
78
79 def repr(self):

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in _load_data(self, stream)
156
157 def _load_data(self, stream):
--> 158 if self.data_filename is not None:
159 return self._load_detached_data()
160

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in data_filename(self)
130 def data_filename(self):
131 """Return detached filename else None."""
--> 132 return self._data_filename
133
134 @Property

~/anaconda3/lib/python3.7/site-packages/planetaryimage/pds3image.py in _data_filename(self)
300 @Property
301 def _data_filename(self):
--> 302 return self._image_pointer.filename
303
304 @Property

~/anaconda3/lib/python3.7/site-packages/planetaryimage/pds3image.py in _image_pointer(self)
313 @Property
314 def _image_pointer(self):
--> 315 return Pointer.parse(self.label['^IMAGE'], self.record_bytes)
316
317 @Property

~/anaconda3/lib/python3.7/site-packages/pvl/collections.py in getitem(self, key)
173 if isinstance(key, (int, slice)):
174 return self.__items[key]
--> 175 return dict_getitem(self, key)[0]
176
177 def delitem(self, key):

KeyError: '^IMAGE'

----------------------------for lbl file------------------------------------
FileNotFoundError Traceback (most recent call last)
in
1 from planetaryimage import PDS3Image
----> 2 image = PDS3Image.open('mvic_0299178092_0x536_sci_pl.lbl')

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in open(cls, filename)
35 else:
36 with open(filename, 'rb') as fp:
---> 37 return cls(fp, filename)
38
39 def init(self, stream_string_or_array, filename=None, compression=None):

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in init(self, stream_string_or_array, filename, compression)
75
76 #: A numpy array representing the image
---> 77 self.data = self._load_data(stream_string_or_array)
78
79 def repr(self):

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in _load_data(self, stream)
157 def _load_data(self, stream):
158 if self.data_filename is not None:
--> 159 return self._load_detached_data()
160
161 stream.seek(self.start_byte)

~/anaconda3/lib/python3.7/site-packages/planetaryimage/image.py in _load_detached_data(self)
172 filename = os.path.abspath(os.path.join(dirpath, self.data_filename))
173
--> 174 with open(filename, 'rb') as stream:
175 return self._decode(stream)

FileNotFoundError: [Errno 2] No such file or directory: '/home/emran/SPAC/Pluto/New Horizons/copied file/Pluto Encounter Surface Composition/data/color/MVIC_0299178092_0X536_SCI_PL.IMG'

Change .travisci.yml to run the tox tests instead

It's always annoyed me that we use tox to run tests against multiple pythons and then in travis we redo this work and test multiple environments differently. I think I have finally found an example that uses travis in conjunction with tox. Check this out:

language: python
python:
 - 2.7
sudo: false
env:
 - TOX_ENV=py27
 - TOX_ENV=docs
 - TOX_ENV=lint
install:
 - pip install tox
 - pip install coveralls
script:
 - tox -e $TOX_ENV
after_success:
 - coveralls

Maybe it will work for us.

Example from RTFD: https://github.com/rtfd/readthedocs.org/blob/8ca4056da4bce471fa87e6e8a10f305c55f2fa53/.travis.yml

Setup coveralls.io

We need coveralls to help remind us to keep our test coverage up.

Write sample new file support test

Every file/mission type we claim to support must have a minimal test implemented. Please implement one of these tests for a PDS3 pancam product. It should be roughly

# -*- coding: utf-8 -*-
import pytest
import os
import numpy
from numpy.testing import assert_almost_equal
from planetaryimage.pds3image import PDS3Image

DATA_DIR = os.path.join(os.path.dirname(__file__), 'data/')

def test_pds3_pancam():
    filename = os.path.join(DATA_DIR, '1P....IMG')
    image = PDS3Image.open(filename)

    assert image.format == 'BAND_SEQUENTIAL'
    assert image.pixel_type == numpy.dtype('int16')
    assert image.dtype == numpy.dtype('>i2')

    expected = numpy.loadtxt(
        os.path.join(DATA_DIR, 'pds3_pancam.txt')).reshape(1, 10, 10)
    assert_almost_equal(image.data[20:30,20:30], expected)

Add a new image sublcass VicarImage

Some older planetary image data may be in the Vicar format described here:

• http://www-mipl.jpl.nasa.gov/external/vicar.html
• http://www-mipl.jpl.nasa.gov/external/VICAR_file_fmt.pdf
• http://en.wikipedia.org/wiki/VICAR_file_format

Handle LINE_SUFFIX and PREFIX data defined in external tables.

Our test Cassini image (W1782844276_1.IMG, W1782844276_1.LBL) has a LINE_PREFIX_TABLE which looks like this:

OBJECT = LINE_PREFIX_TABLE
      INTERCHANGE_FORMAT = BINARY
      ROWS = 1024
      COLUMNS = 10
      ROW_BYTES = 24
      ROW_SUFFIX_BYTES = 1024
      ^LINE_PREFIX_STRUCTURE = "../../label/prefix3.fmt"
END_OBJECT = LINE_PREFIX_TABLE

Without accounting for the line prefix data an image looks like this:

The PDS Spec has the following to say in section A.20:

In more complex images, each individual line may have some attached data
which are not part of the image itself (engineering data, checksums, time tags,
etc.). In this case the additional, non- image parameters are accounted for as
either LINE_PREFIX_BYTES or LINE_SUFFIX_BYTES, depending on whether
they occur before or after the image samples in the line. These keywords indicate
the total number of bytes used for the additional data, so that software processing
the image can clip these bytes before attempting to display or manipulate the
image. The structure of the prefix or suffix bytes is most often defined by a
TABLE object (in the same label), which will itself have ROW_SUFFIX_BYTES
or ROW_PREFIX_BYTES, to allow table-processing software to skip over the
image data. Figure A.1 illustrates the layout of prefix and suffix bytes around an
image.

As far as this Cassini data is concerned, there is not a simple LINE_PREFIX_BYTES label. There is an entire TABLE object that describes the suffix data. This implies that we need to be able to parse TABLE objects to handle these more complicated images.

Improve Error when opening Detached Label

There should be a better error when attempting to open a detached label image with the image file rather than the label file. This is the most common error I come across:

    121         if isinstance(key, INDEX_TYPES):
    122             return self.__items[key]
--> 123         return dict_getitem(self, key)[0]
    124 
    125     def __delitem__(self, key):

KeyError: '^IMAGE'

Which doesn't really explain to me that there is a detached label I should open. I don't have any immediate ideas on how to make this more explicit. In a perfect world, we would search for the label file ourselves but since there is a lack of consistency in the extensions, I don't see how my perfect world can exist.

Licence

Looks like you currently have MIT licence for this repo. It would be advantageous to have one licence for all the planetarypy projects. We should update this to BSD or change pvl to MIT. I'm fine with either.

We should also get a cookiecutter package together for planetarypy. I can put that together if you'd like.

Comments on PlanetaryImage

Here is a hastily implemented version of the PDS3 image parser derived from your cubefile code. There is a parent class PlanetaryImage and a subclass Pds3Image. Your CubeFile would inherit from PlanetaryImage. There are a few methods that still need to be copied into PlanetaryImage. Do you think there is sufficient overlap to justify this?

There are some minor issues with the implementation that I am still looking into.

Tox and travis tests - make test

So, in a feature branch, I've just made some changes to the test target in the Makefile and it has occurred to me that the tox.ini and the travis.yml issue explicit py.test tests commands rather than running make test. Is there some reason we shouldn't use make test in those places?

CC: @wtolson

Handle pointer offsets that are expressed in bytes

The test file ESP_030168_1755_BG12_0.IMG defines the beginning of the product data using an explicit offset in bytes. The parse_pointer method might handle this properly if it was called correctly by start_byte in these cases. Sadly, it is not. Fix it!

^IMAGE                         = 101337 <BYTES>

Allow the creation of a PDS3Image object from a numpy array

Allow save to create a PDS3Image object from a numpy array, which will serialized it out to a PDS3 file.

• Make a label for an array which contains useful information about an image. Such as,
  • Lines
  • Line Samples
  • Bands
  • Sample_Bits
  • Sample_Type
  • ^Image
• Make save function to distinguish between a numpy array and PDS Image with label

Refactor _save

The _save needs to be refactored which means to separate the steps into smaller functions that can be tested and make the steps more "pythonic" (i.e. context managers, xrange, etc.). Think about usability and how that can be improved as well

Can't save correctly with new data type

Im running into a problem with saving an image after I convert the data to a different dtype. I need to convert because I have to make changes to the data and I was having problems making the needed changes with uint8. I also had to make changes to the label and need those changes seen in the new file. Here is how to recreate the problem:

>>> from planetaryimage import PDS3Image
>>> import numpy as np
>>> old_im = PDS3Image.open('tests/mission_data/0615ML0026220010301833C00_DXXX.IMG')
>>> old_im.data
array([[[ 8,  5,  8, ...,  5,  7,  4],
        [ 2,  2,  5, ...,  4,  4,  4],
        [72, 55, 84, ..., 30, 41, 26],
        ...,
        [54, 34, 51, ..., 82, 62, 42],
        [71, 51, 71, ..., 79, 75, 59],
        [52, 34, 50, ..., 44, 64, 45]]], dtype=uint8)
>>> old_im.data = old_im.data.astype(np.int64)
>>> new_im = 'new_im.IMG'
>>> old_im.save(new_im)
>>> PDS3Image.open(new_im).data
array([[[ 576460752303423488,  360287970189639680,  576460752303423488,
         ...,  360287970189639680,  504403158265495552,  288230376151711744],
        [ 144115188075855872,  144115188075855872,  360287970189639680,
         ...,  288230376151711744,  288230376151711744,  288230376151711744],
        [5188146770730811392, 3963167672086036480, 6052837899185946624,
         ..., 2161727821137838080, 2954361355555045376, 1873497444986126336],
        ...,
        [3891110078048108544, 2449958197289549824, 3674937295934324736,
         ..., 5908722711110090752, 4467570830351532032, 3026418949592973312],
        [5116089176692883456, 3674937295934324736, 5116089176692883456,
         ..., 5692549928996306944, 5404319552844595200, 4251398048237748224],
        [3746994889972252672, 2449958197289549824, 3602879701896396800,
         ..., 3170534137668829184, 4611686018427387904, 3242591731706757120]]])

I think this is part of a larger bug because I don't get the same array when I just save it as is:

>>> from planetaryimage import PDS3Image
>>> old_im = PDS3Image.open('tests/mission_data/0615ML0026220010301833C00_DXXX.IMG')
>>> old_im.data
array([[[ 8,  5,  8, ...,  5,  7,  4],
        [ 2,  2,  5, ...,  4,  4,  4],
        [72, 55, 84, ..., 30, 41, 26],
        ...,
        [54, 34, 51, ..., 82, 62, 42],
        [71, 51, 71, ..., 79, 75, 59],
        [52, 34, 50, ..., 44, 64, 45]]], dtype=uint8)
>>> new_im = 'new_im.IMG'
>>> old_im.save(new_im)
>>> PDS3Image.open(new_im).data
array([[[ 32,  32,  32, ...,   7,   4,   4],
        [  5,   2,   5, ...,   2,   1,   4],
        [  5,   2,   2, ...,  44,  57,  40],
        ...,
        [118,  95, 107, ...,  56,  81,  54],
        [ 80,  55,  83, ...,  87, 105,  84],
        [104,  89,  95, ...,  71,  91,  45]]], dtype=uint8)

Looking at the test for save, it doesn't seem like the data is tested to be correct, only the surrounding parameters. I can write a test to expose this bug but I wanted to check if I was using save correctly before I did so.

Adding support for Mastcam PDS .dat images

Many Mastcam PDS images are distributed as compressed .dat files that need to be decompressed into .IMG files.

The source for a utility dat2img is available in the PDS to do this conversion:

http://pds-imaging.jpl.nasa.gov/data/msl/MSLMST_0007/SOFTWARE/

It may be possible to build this using distutils:

https://docs.python.org/2/extending/building.html

Sample data files:

http://pds-imaging.jpl.nasa.gov/data/msl/MSLMST_0007/DATA/EDR/SURFACE/0640/0640ML0027190060302751I01_XXXX.LBL
http://pds-imaging.jpl.nasa.gov/data/msl/MSLMST_0007/DATA/EDR/SURFACE/0640/0640ML0027190060302751I01_XXXX.IMG

test_mission_data fails in Python 3

Fix this issue: https://travis-ci.org/planetarypy/planetaryimage/jobs/70781416

While doing so, extend the label tests to be more generic and test a few more label values for the core data products.

Collect sample data.

We need to collect a decent set of sample data to ensure that the library is robust. I started this in my gdal_pds repo, it grabs some MER data, but its all very similar:

https://github.com/godber/gdal_pds/blob/master/get_test_data.py
https://github.com/godber/gdal_pds/blob/master/data.json

I'd really like to see data from every PDS Image Node mission represented.

Add support for saving the PDS file when `dtype` is changed.

While calibrating the image, we decompand it first. During this operation, input data is of 8 bits which is then changed to 12 bits. So, our storage dtype changes to uint16 from uint8.

Now, Line 139 saves data with the dtype of the input data, there we need to check dtype for the data which is to be written.

Image should accept a URL and open it

Anyone have a good idea on what syntax we should add to allow users to open a file from a URL? Should we offer a separate class method for it? Inspect stream and see if its a URL then treat it as such? Do we then download the image into a cache (probably)?

Extend and update docs

There needs to be docs explaining the different attributes/properties and methods of the image objects. Right now there is only a small walkthrough

Handle simple LINE_PREFIX_BYTES and LINE_SUFFIX_BYTES case

ESP_030168_1755_BG12_0.IMG has simple integer values for line suffix/prefix bytes:

/* Image data description. */

OBJECT = IMAGE
    LINES             = 30000
    LINE_SAMPLES      = 512
    SAMPLE_BITS       = 8
    SAMPLE_BIT_MASK   = 2#11111111#
    SAMPLE_TYPE       = MSB_UNSIGNED_INTEGER
    MISSING_CONSTANT  = 16#FF#
    LINE_PREFIX_BYTES = 18
    LINE_SUFFIX_BYTES = 16
    DESCRIPTION       = "Observation image data."
END_OBJECT = IMAGE

This needs to be handled somehow, and it's really a bigger job than just throwing the prefix/suffix away so we can display an image. These prefix/suffix data need to be preserved. For instance, in the example above, the suffix data is dark reference pixels that would be necessary for calibrating the image and is described as follows:

/* Image data line suffix description. */

OBJECT = LINE_SUFFIX_TABLE
    INTERCHANGE_FORMAT = BINARY
    ROWS               = 30000
    COLUMNS            = 1
    ROW_BYTES          = 16
    ROW_PREFIX_BYTES   = 530
    OBJECT = COLUMN
        NAME        = "Dark Reference Pixels"
        DATA_TYPE   = MSB_UNSIGNED_INTEGER
        START_BYTE  = 1
        BYTES       = 16
        ITEMS       = 16
        ITEM_BYTES  = 1
        DESCRIPTION = "Dark reference pixel values produced by masked
                      detectors."
    END_OBJECT = COLUMN
END_OBJECT = LINE_SUFFIX_TABLE

So what needs to be worked out prior to implementing this ticket is the interface we expect to use to access image data while maintaining access to this additional data.

Handle PDS3 Data Objects in a more general manner

Right now we handle a single data object ^IMAGE. However, a PDS label can define multiple data objects. We should continue to expose these as attributes but also keep track of them as data objects so that a list of data objects can be returned. Note that the Data Object Pointers are paired with Data Object definitions

See the PDS Ref sections:

• 5.3.3 - Data Object Pointers
• 5.3.6 - Data Object Definitions

Add Support to save an Image with Band Interleaved by Pixel

make test and make test-all fail

On Master if you run make test twice in a row the second one will fail due to the left over temp.IMG file from the previous run. This will also be a problem when you run make test-all.

Add Support to save an Image with Band Interleaved by Line

Support gz compressed images

This is not part of the PDS spec, but sometimes PDS images are gzip compressed to save space. Planetary image should be able to open these. Assume the filenames end with .gz.

Sample band sequential isis cube file

The Isis Tile data format is a separate data storage format right? Can someone track down a sample file? @sbraden

How do we handle mission specific extensions to planetaryimage?

There are probably a number of mission specific extensions that can be implemented for planetaryimage. Things like filename parsing (for faster metadata extraction than actually opening a file) and image orientation information (e.g. Mastcam images appear inverted by default and need to be flipped when viewing them). It seems there might be a whole variety of these sorts of things, some of which might be needed to properly access the image data. For instance the Mastcam PDS products are JPEG compressed and claim they need to be converted into a .IMG using DAT2IMG. I suspect that Mastcam PDS archived products can simply have the JPEG data decoded by planetaryimage skipping any redundant product generation.

There may be need to support internal/proprietary extensions. Subclassing would work but if we implemented a plugin architecture of some sort that allowed extending the behavior of planetaryimage without changing the syntax by simply importing and registering a handler of some sort. Or maybe this is a bad idea for some reason.

PDS3Image.open() fails when we try to open Compressed PDS file (.gz -or- .bz2) with Sample_Bits = 64

Error:

ValueError: total size of new array must be unchanged

Parse PDS table descriptions

PDS tables are used not only for data storage but also to describe LINE_SUFFIX and LINE_PREFIX in some cases. We will need a means to interpret this data.

See also:
#18

GDAL Support?

There was discussion at the Data Users Workshop Hackday of building parallel PDS image viewers: one in pure Python and one that uses GDAL + PVL. (See: https://github.com/planetarypy/planetaryimage/wiki/Requirements) It looks like maybe the first is underway but not the second. Is anybody working on the GDAL angle, and do you still think it's necessary?

Update pvl to 0.3.0 and use strict parameter

The open method should include a kwarg strict=True to pass into pvl when opening an image. Either that or just always use strict=False when opening the label but I think we should give the user the option just like we give the user the option in pvl. Thoughts?

Support JPEG2000 based products

Some products (notably HiRISE) are JPEG2000 encoded. Special support will be needed for these.

Perhaps the OpenJPEG wrapper will help:
http://glymur.readthedocs.org/en/latest/introduction.html

Make `Band_Sequential` USGS ISIS Cubefile and add test for it

Generally, we deal with Tile cube file. We recently changed decoder for Band_Sequential input type. To test that, make USGS ISIS Cubefile and add test for it.

Does not support detached labels

I tried opening the following Cassini image that has a detached label but it exploded:

http://pds-imaging.jpl.nasa.gov/data/cassini/cassini_orbiter/coiss_2090/data/1782761458_1782849980/W1782844276_1.LBL
http://pds-imaging.jpl.nasa.gov/data/cassini/cassini_orbiter/coiss_2090/data/1782761458_1782849980/W1782844276_1.IMG

from planetaryimage import PDS3Image
image = PDS3Image.open('W1782844276_1.LBL')

Here is the error I got

/home/pbvarga1/Workspace/planetaryimage/planetaryimage/pds3image.py in start_byte(self)
     69                     % pointer.units)
     70         else:
---> 71             raise ValueError('Unsupported image pointer type')
     72 
     73     @property

ValueError: Unsupported image pointer type

There may be other reasons this image doesn't open.

Replace get_nested_dict with dict lookups in PDS3Image

I think directly doing dictionary lookups in PDS3Image would be better than using get_nested_dict. The get_nested_dict was "necessary" to inform the parent class, PlanetaryImage where it could find certain things. I think its just a needless distraction/abstraction at this level.

Also, "necessary" is in quotes above because I suspect there's a better way to accomplish what get_nested_dict is doing. For instance properties that return the appropriate values.

Add support for SAMPLE_INTERLEAVED PDS Data

Right now PDS3Image only supports BAND_INTERLEAVED data. I just came across an example of SAMPLE_INTERLEAVED data.

http://pds-imaging.jpl.nasa.gov/data/lcross/LCRO_0001/DATA/20091009113022_IMPACT/MIR1/CAL/LCROSS_MIR1_CAL_20091009113134589.LBL
http://pds-imaging.jpl.nasa.gov/data/lcross/LCRO_0001/DATA/20091009113022_IMPACT/MIR1/CAL/LCROSS_MIR1_CAL_20091009113134589.IMG

Special Pixels

Should the these high special pixels be positive?