sentinelsat / sentinelsat

Search and download Copernicus Sentinel satellite images

sentinelsat

⚠️ Sentinelsat is currently not functional for downloading from the CDSE ⚠️

The Copernicus Open Access Hub (aka SciHub), that Sentinelsat was primarily built for, is permanently closed.

The official successor data distribution system, the Copernicus Data Space Ecosystem (CDSE), uses different API logic that Sentinelsat does not (yet?) support. Contributions are welcome to change that, please check issues and discussions for ongoing efforts or alternatives.

Sentinelsat can still be used against other data hubs powered by the DHuS software.

Legacy project documentation

Please note that examples referencing the old Copernicus Open Access Hub are not working anymore.

Sentinelsat makes searching, downloading and retrieving the metadata of Sentinel satellite images from the Copernicus Open Access Hub easy.

It offers an easy-to-use command line interface

sentinelsat -u <user> -p <password> --location Berlin --sentinel 2 --cloud 30 --start NOW-1MONTH

and a powerful Python API.

from sentinelsat import SentinelAPI, read_geojson, geojson_to_wkt

api = SentinelAPI('user', 'password')
footprint = geojson_to_wkt(read_geojson('search_polygon.geojson'))
products = api.query(footprint,
                     producttype='SLC',
                     orbitdirection='ASCENDING',
                     limit=10)
api.download_all(products)

Documentation is published at http://sentinelsat.readthedocs.io/.

Installation

Install sentinelsat through pip:

pip install sentinelsat

Usage

Sentinelsat provides a Python API and a command line interface to search, download and retrieve the metadata for Sentinel products.

Python Library

from sentinelsat import SentinelAPI, read_geojson, geojson_to_wkt
from datetime import date

# connect to the API
api = SentinelAPI('user', 'password', 'dhus-hub-url')

# download single scene by known product id
api.download(<product_id>)

# search by polygon, time, and Hub query keywords
footprint = geojson_to_wkt(read_geojson('map.geojson'))
products = api.query(footprint,
                     date = ('20151219', date(2015, 12, 29)),
                     platformname = 'Sentinel-2',
                     cloudcoverpercentage = (0, 30))

# download all results from the search
api.download_all(products)

# GeoJSON FeatureCollection containing footprints and metadata of the scenes
api.to_geojson(products)

# GeoPandas GeoDataFrame with the metadata of the scenes and the footprints as geometries
api.to_geodataframe(products)

# Get basic information about the product: its title, file size, MD5 sum, date, footprint and
# its download url
api.get_product_odata(<product_id>)

# Get the product's full metadata available on the server
api.get_product_odata(<product_id>, full=True)

Valid search query keywords can be found at the Copernicus Open Access Hub documentation.

Command Line Interface

A basic search query consists of a search area geometry as well as the username and password to access the Copernicus Open Access Hub.

sentinelsat -u <user> -p <password> -g <geojson>

Search areas are provided as GeoJSON files, which can be created with QGIS or geojson.io. If you do not specify a start and end date only products published in the last 24 hours will be queried.

Example

Search and download all Sentinel-1 scenes of type SLC, in descending orbit, for the year 2015.

sentinelsat -u <user> -p <password> -g <search_polygon.geojson> -s 20150101 -e 20151231 -d \
--producttype SLC -q "orbitdirection=Descending" \
--url "https://apihub.copernicus.eu/apihub"

Username, password and DHuS URL can also be set via environment variables for convenience.

# same result as query above
export DHUS_USER="<user>"
export DHUS_PASSWORD="<password>"
export DHUS_URL="https://apihub.copernicus.eu/apihub"

sentinelsat -g <search_polygon.geojson> -s 20150101 -e 20151231 -d \
--producttype SLC -q "orbitdirection=Descending"

Options

-u	--user	TEXT	Username [required] (or environment variable DHUS_USER)
-p	--password	TEXT	Password [required] (or environment variable DHUS_PASSWORD)
	--url	TEXT	Define another API URL. Default URL is 'https://apihub.copernicus.eu/apihub/'.
-s	--start	TEXT	Start date of the query in the format YYYYMMDD or an expression like NOW-1DAY.
-e	--end	TEXT	End date of the query.
-g	--geometry	PATH	Search area geometry as GeoJSON file.
	--uuid	TEXT	Select a specific product UUID. Can be set more than once.
	--name	TEXT	Select specific product(s) by filename. Supports wildcards. Can be set more than once.
	--sentinel	INT	Limit search to a Sentinel satellite (constellation).
	--instrument	TEXT	Limit search to a specific instrument on a Sentinel satellite.
	--producttype	TEXT	Limit search to a Sentinel product type.
-c	--cloud	INT	Maximum cloud cover in percent. (requires --sentinel to be 2 or 3)
-o	--order-by	TEXT	Comma-separated list of keywords to order the result by. Prefix '-' for descending order.
-l	--limit	INT	Maximum number of results to return. Defaults to no limit.
-d	--download		Download all results of the query.
	--fail-fast		Skip all other other downloads if one fails.
	--path	PATH	Set the path where the files will be saved.
-q	--query	TEXT	Extra search keywords you want to use in the query. Example: '-q producttype=GRD -q polarisationmode=HH'. Repeated keywords are interpreted as an "or" expression.
-f	--footprints	FILENAME	Create a GeoJSON file at the provided path with footprints and metadata of the returned products. Set to '-' for stdout.
	--include-pattern	TEXT	Glob pattern to filter files (within each product) to be downloaded.
	--exclude-pattern	TEXT	Glob pattern to filter files (within each product) to be excluded from the downloaded.
	--timeout	FLOAT	How long to wait for a DataHub response (in seconds, default 60 sec).
	--gnss		Query orbit products form the GNSS end-point ("https://scihub.copernicus.eu/gnss").
	--fmt	TEXT	Specify a custom format to print results. The format string shall be compatible with the Python "Format Specification Mini-Language".
	--info		Display DHuS server information.
	--version		Show version number and exit.
	--debug		Print debug log messages.
-h	--help		Show help message and exit.

Tests

To run the tests on sentinelsat:

git clone https://github.com/sentinelsat/sentinelsat.git
cd sentinelsat
pip install -e .[dev]
pytest -v

By default, prerecorded responses to Copernicus Open Access Hub queries are used to not be affected by its downtime. To allow the tests to run actual queries against the Copernicus Open Access Hub set the environment variables

export DHUS_USER=<username>
export DHUS_PASSWORD=<password>

and add --disable-vcr to pytest arguments. To update the recordings use --vcr-record with once, new_episodes or all. See vcrpy docs for details.

Documentation

To build the documentation:

git clone https://github.com/sentinelsat/sentinelsat.git
cd sentinelsat
pip install -e .[dev]
cd docs
make html

The full documentation is also published at http://sentinelsat.readthedocs.io/.

Changelog

See CHANGELOG. You can also use GitHub's compare view to see the changes in the main branch since last release.

Contributors

We invite anyone to participate by contributing code, reporting bugs, fixing bugs, writing documentation and tutorials and discussing the future of this project. Please check CONTRIBUTE.rst.

For a list of maintainers and contributors please see AUTHORS.rst and the contributor graph.

License

GPLv3+

sentinelsat's Issues

version 0.6 fails on pip install

When trying to install sentinelsat through pip it seems to be missing the requirements.txt.

vm:~$ pip install sentinelsat --no-cache-dir
Collecting sentinelsat
  Downloading sentinelsat-0.6.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/tmp/pip-build-a84kSh/sentinelsat/setup.py", line 30, in <module>
        install_requires=open('requirements.txt').read().splitlines(),
    IOError: [Errno 2] No such file or directory: 'requirements.txt'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-a84kSh/sentinelsat

Installing from github works fine:

vm:~$ git clone https://github.com/ibamacsr/sentinelsat.git
vm:~$ cd sentinelsat/
vm:~$ python setup.py install
...
Finished processing dependencies for sentinelsat==0.6

This was reproduced multiple times on Win_x64 with Python 2.7 & 3.4 as well as Ubuntu 14.04 with the system Python 2.7.

TypeError raised by GEOJSON with XYZ coordinates

I get the following error:

sc = api.query(area=get_coordinates(r"F:\S2\AOIs\test.geojson"), 
               initial_date='20150101', end_date='20151231', 
               platformname='Sentinel-2', cloudcoverpercentage='[0 TO 30]')

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-68033ba1fa36> in <module>()
----> 1 sc = api.query(area=get_coordinates(r"F:\S2\AOIs\test.geojson"), 
      2                initial_date='20150101', end_date='20151231',
      3                platformname='Sentinel-2', cloudcoverpercentage='[0 TO 30]')

F:\Programme\Anaconda2\lib\site-packages\sentinelsat\sentinel.pyc in get_coordinates(geojson_file, feature_number)
    495     coordinates = geojson_obj['features'][feature_number]['geometry']['coordinates'][0]
    496     # precision of 7 decimals equals 1mm at the equator
--> 497     coordinates = ['%.7f %.7f' % tuple(coord) for coord in coordinates]
    498     return ','.join(coordinates)
    499 

TypeError: not all arguments converted during string formatting

With the GEOJSON

{
"type": "FeatureCollection",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
                                                                                                                                  
"features": [
{ "type": "Feature", "properties": { "Name": "32UPU_to_33UUP" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 11.072495581640744, 48.249169452900695, 0.0 ], [ 13.060647163314195, 48.253249090436974, 0.0 ], [ 13.063710570990118, 48.232847647952589, 0.0 ], [ 12.423458366722397, 48.192020350122611, 0.0 ], [ 11.069432173964822, 48.226725628443738, 0.0 ], [ 11.072495581640744, 48.249169452900695, 0.0 ] ] ] } }
]
}

The GEOJSON was created by QGIS 2.14.3..

I think it can be easily solved by replacing

coordinates = ['%.7f %.7f' % tuple(coord) for coord in coordinates]

with (not tested)

coordinates = ['%.7f %.7f' % (coord[0], coord[1]) for coord in coordinates]

Add option for filtering single-tile products

It might be worthwile to add an option to select only single-tile products - or exclude them - in queries.

This is related to our query tests currently failing as mentioned in #81.

Revert some changes to SentinelAPI from last PR

@valgur, in the last PR, there were a few changes to SentinelAPI you wanted to revert. I think you made good points, so here they are for further discussion:

2. I would rename the max_query to page_size (since we are doing pagination) and change the max_query to actually limit the number of items returned. Again, to reduce stress on the SciHub server if we actually do just need a limited number of query results.
3. I preferred the query_raw() over load_query() which sounds too similar and redundant to query() to me. Could be named something else as well, but either way, I think a function that exposes the text-search format of the SciHub server should be a first-class citizen in the API, since it is much more powerful, if necessary, than the main query() function. For example, it allows for any logical statements that include OR and parentheses to be used. I also occasionally find footprint:Contains() to be useful instead of footprint:Intersects(). I have been intending to add the raw text query support to the CLI as well but have not found time for that yet.

"Full authentication is required to access this resource"

Hello,

if I try to code a query (python API), I got the error message:

SentinelAPIError: # HTTP Status 401 - Full authentication is required to access this resource

---

type Status report
message Full authentication is required to access this resource
description This request requires HTTP authentication.

---

Apache Tomcat/7.0.47

However, I´m sure that my username and password are correct.

If I try to search a sentinel scene with the command line, I got the error message:

Traceback (most recent call last):[...]
sentinelsat.sentinel.SentinelAPIError: (HTTP status: 500, code: None)

HTTP Status 500 - Server returned HTTP response code: 400 for URL: [...]

---

type Exception report
message _Server returned HTTP response code: 400 for URL: [...]
description The server encountered an internal error that prevented it from fulfilling this request.
exception
java.io.IOException: Server returned HTTP response code: 400 for URL: [...]
note The full stack trace of the root cause is available in the Apache Tomcat/7.0.47 logs.

Here is the script for the Python API:

from sentinelsat.sentinel import SentinelAPI, get_coordinates
from datetime import date
import os

api = SentinelAPI('xxx', 'yyy', 'https://scihub.copernicus.eu/dhus/')

WORKDIR    = "C:\...\test"
geofile =  WORKDIR + os.sep + os.listdir(WORKDIR)[0]

# search by polygon, time, and SciHub query keywords

api.query(get_coordinates(geofile), \
          "20151219", date(2015, 12, 29), \
          keywords={"platformname": "Sentinel-1"})

Here is the script for the command line:

sentinel search -f -s 20140101 -e 20151004 -q "producttype=GRD" -u "https://scihub.copernicus.eu/dhus/" xxx yyy C:\\...\\map.geojson

What am I doing wrong?

HTTP authentication required

Hello,

i updated sentinelsat and tried to run the example sentinel search --sentinel2 <user> <password> <poly.geojson>

I get an error:

* * *

**type** Status report

**message** _Full authentication is required to access this resource_

**description** _This request requires HTTP authentication._

* * *

### Apache Tomcat/7.0.47

certifi is installed and updated

edit: same via the python api

S2: download tiles

Sentinel-2 data is tiled according to the Military Grid Reference System (MGRS).

At the moment sentinelsat only allows the download of complete S2 products which contain a number of tiles.

It would be nice to have the option to search and download data for a specific tile (or a number of tilkes) only.

@olivierhagolle already implemented something similar in his sentinel_download.py.

test_download times out in v0.7.4

In Travis test_download tries for 10 minutes and then times out, resulting in a failed build and test_download_all being ignored.

https://travis-ci.org/Fernerkundung/sentinelsat/jobs/192275684

Since both use VCR cassettes I suspect this is an issue with them.

Support for large queries

sentinelsat currently can not perform search if query length is more than some limit. For example querying this geojson:
ETH_adm0.zip

results in:

u@m:~$ ~/.local/bin/sentinel search user password ETH_adm0.geojson 
Error: API returned unexpected response 414 .
Traceback (most recent call last):
  File "/home/u/.local/bin/sentinel", line 9, in <module>
    load_entry_point('sentinelsat', 'console_scripts', 'sentinel')()
  File "/home/u/.local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/u/.local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/u/.local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/u/.local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/u/.local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/u/sentinelsat/sentinelsat/scripts/cli.py", line 99, in search
    for product in api.get_products():
  File "/home/u/sentinelsat/sentinelsat/sentinel.py", line 130, in get_products
    raise ValueError('API response not valid. JSON decoding failed.')
ValueError: API response not valid. JSON decoding failed.
u@m:~$ 

(414 is Request URI too long)

add support for netrc file

Hi,

do you think netrc file support could be useful?
(I think yes so we don't need to pass user and specially password in the command line)

I started to implement it in netrc branc 52ee682

-s option reads ingested not acquired date

Using the -s option including a date limits the download to the time period where the image was uploaded to the hub, not when it was acquired.

sentinel search -d -s 20151219 -c 30 --md5 username password roi.geojson

Search and download all Sentinel-2 products with less than 30% cloud cover acquired since 2015-12-19.

E.g: using -s 20160525 will download data aquired from 2015 which was just added (ingested) to the Sentinel-2 hub this month.

Download improvement ideas

I have been thinking for a while about possibly replacing homura with something custom-made based on requests. Here's my reasoning:

• A major drawback of homura is that we have no direct control over downloading, a significant part of sentinelsat.
• Homura does not check for HTTP error statuses, which can cause it to append junk to the downloaded files. This is a frequent situation on SciHub.
• This will open up the opportunity to add parallel download support. My idea here would be to split the download for each file into n parts, one for each download thread, each part being saved as "<product>.zip.part<i>", which will be merged together once the download is complete and checked. This naming scheme will make it easier to identify incomplete downloads in the download directory.
• tqdm can be used to show download progress, speed and ETA. An additional progressbar can also be added to display overall download progress for multiple files. tqdm is already used during checksum calculation in sentinelsat.
• Gets rid of the resuming of > 2 GB files bug in pycurl.
• If logging is implemented, then more fine-grained information and debug output can be logged.
• No need to worry about certifi being installed or not.
• By using requests the downloading functionality becomes mockable with vcrpy as well. We will be also able to simulate edge cases, like download errors, better for testing.

What are your thoughts on these proposed changes?

HTTP Status 500 - connect timed out

Hi,

I'm new user of sentinelsat and I often get "HTTP Status 500 - connect timed out" response when I try to search. Is the problem on my request? (maybe the geojson is too big I'm trying to get scenes for Italy) or is the problem server side?

Thanks
Luca

sentinel search --sentinel2 -s 20160701 -e 20160801 -c 5 USER PASSWD italia_bbox.json 

Traceback (most recent call last):
  File "/usr/local/bin/sentinel", line 11, in <module>
    load_entry_point('sentinelsat==0.7.1', 'console_scripts', 'sentinel')()
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sentinelsat-0.7.1-py2.7.egg/sentinelsat/scripts/cli.py", line 86, in search
    api.query(get_coordinates(geojson), start, end, **search_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sentinelsat-0.7.1-py2.7.egg/sentinelsat/sentinel.py", line 143, in query
    self.load_query(query)
  File "/usr/local/lib/python2.7/dist-packages/sentinelsat-0.7.1-py2.7.egg/sentinelsat/sentinel.py", line 158, in load_query
    _check_scihub_response(content)
  File "/usr/local/lib/python2.7/dist-packages/sentinelsat-0.7.1-py2.7.egg/sentinelsat/sentinel.py", line 98, in _check_scihub_response
    raise api_error
sentinelsat.sentinel.SentinelAPIError: (HTTP status: 500, code: None) 
# HTTP Status 500 - connect timed out

* * *

**type** Exception report

**message** _connect timed out_

**description** _The server encountered an internal error that prevented it from fulfilling this request._

**exception**
    
    
    java.net.SocketTimeoutException: connect timed out
    	java.net.PlainSocketImpl.socketConnect(Native Method)
    	java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    	java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    	java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    	java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    	java.net.Socket.connect(Socket.java:579)
    	sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    	sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
    	sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
    	sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
    	sun.net.www.http.HttpClient.New(HttpClient.java:308)
    	sun.net.www.http.HttpClient.New(HttpClient.java:326)
    	sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
    	sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
    	sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
    	sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
    	fr.gael.dhus.server.http.webapp.search.controller.SearchController.search(SearchController.java:164)
    	fr.gael.dhus.server.http.webapp.search.controller.SearchController$$FastClassBySpringCGLIB$$e2477ad5.invoke(<generated>)
    	org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    	org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    	org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)
    	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    	org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    	fr.gael.dhus.server.http.webapp.search.controller.SearchController$$EnhancerBySpringCGLIB$$b1be1407.search(<generated>)
    	sun.reflect.GeneratedMethodAccessor371.invoke(Unknown Source)
    	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	java.lang.reflect.Method.invoke(Method.java:606)
    	org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    	org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    	org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    	org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
    	org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
    	org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    	org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
    	org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
    	org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    	org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
    	javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    	org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    	javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    	fr.gael.dhus.spring.CacheControlFilter.doFilter(CacheControlFilter.java:47)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    	org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    	org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	fr.gael.dhus.spring.security.filter.PreAuthFilter.doFilter(PreAuthFilter.java:79)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
    	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    	org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    	org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    	org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
    	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
    

**note** _The full stack trace of the root cause is available in the Apache Tomcat/7.0.47 logs._

* * *

### Apache Tomcat/7.0.47

Make sentinelsat citeable

I'm currently in the process of writing a research paper where I want to quote my use of sentinelsat. The easiest way to do this would be to register releases to DOIs with the help of Zenodo. This has become quite popular with scientific software repositories as it bolsters their visibility and makes citing as easy as possible.

There's a tutorial on the integration of Zenodo and github which basically sets-up a webhook which archives each release, adds a DOI to the release and allows us to include a badge with the DOI of the current release.

@willemarcel I could set this up myself if you are willing to grant me contributor status (I think I've allready shown that I am commited to this project and would like to keep improving it) or you could set-up the webhook yourself.

SENTINELSAT.SENTINEL.SENTINELAPIERROR: (HTTP STATUS: 500, CODE: NONE)

Hi everyone,
I am a new sentinelsat user. I have tried the command line below and faced an error.
Please instruct me to solve this problem.
Many thanks,
Trung

---

E:\CONG VIEC\MATLAB_SATELLITE\ImageDownload>sentinel search -f -s 20140101 -e 20151004 -q "producttype=GRD" -u "https://scihub.copernicus.eu/dhus/" user password BinhDinh.geojson
Traceback (most recent call last):
File "c:\anaconda2\lib\runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "c:\anaconda2\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Anaconda2\Scripts\sentinel.exe_main.py", line 9, in
File "c:\anaconda2\lib\site-packages\click\core.py", line 716, in call
return self.main(*args, **kwargs)
File "c:\anaconda2\lib\site-packages\click\core.py", line 696, in main
rv = self.invoke(ctx)
File "c:\anaconda2\lib\site-packages\click\core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\anaconda2\lib\site-packages\click\core.py", line 889, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\anaconda2\lib\site-packages\click\core.py", line 534, in invoke
return callback(*args, **kwargs)
File "c:\anaconda2\lib\site-packages\sentinelsat\scripts\cli.py", line 83, in search
api.query(get_coordinates(geojson), start, end, **search_kwargs)
File "c:\anaconda2\lib\site-packages\sentinelsat\sentinel.py", line 146, in query
self.load_query(query)
File "c:\anaconda2\lib\site-packages\sentinelsat\sentinel.py", line 161, in load_query
_check_scihub_response(content)
File "c:\anaconda2\lib\site-packages\sentinelsat\sentinel.py", line 98, in _check_scihub_response
raise api_error
sentinelsat.sentinel.SentinelAPIError: (HTTP status: 500, code: None)

Verify the integrity of the downloaded images

Hi,

I really like this utility for its Python and command line interfaces, which I find both to be very clean and well-thought-out. I would love to use it to manage my SciHub downloads but it is currently missing a critical function I need: ensuring that the downloaded images are complete and non-corrupted. Would you consider adding an integrity check after finishing a image download and possibly re-download the image until the check passes successfully (perhaps a limited number of times to prevent an infinite loop)?

Broken downloads are not at all rare in my experience, especially when downloading SLCs. For example, a snippet from a recent downloading session with my personal script:

Downloading z:\Sentinel-1a\Latvia\tormimurd\data\S1A_IW_GRDH_1SDV_20141009T160333_20141009T160356_002755_003185_F8C5.zip. Id = c998ec17-c7c6-429f-a2f6-38ff2f26f8f2, Checksum = D317054C7A5E71ADC9EC3A09B85EC15C, Length = 1,576,536,265
0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Downloaded 1576536265/1576536265 bytes, checksum is D317054C7A5E71ADC9EC3A09B85EC15C.
Downloading z:\Sentinel-1a\Latvia\tormimurd\data\S1A_IW_GRDH_1SDV_20141021T160333_20141021T160356_002930_003530_A0B5.zip. Id = 7dd51ac2-3f26-4b4c-9162-8275ece60149, Checksum = 40824789726D8248F6B55EAFF16BF970, Length = 1,575,458,978
0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Downloaded 1575458978/1575458978 bytes, checksum is 40824789726D8248F6B55EAFF16BF970.
Downloading z:\Sentinel-1a\Latvia\tormimurd\data\S1A_IW_GRDH_1SDV_20141023T043423_20141023T043448_002952_0035AA_6B85.zip. Id = 427d77d8-9f81-467a-b2a8-e96eae7c36d2, Checksum = C9F1FC5D8B90A8F1126997FA85626CAD, Length = 1,763,514,174
0%....10%....20%....
Downloaded 517355836/1763514174 bytes, checksum is E8A7A9F251199A06DA6FD600A8201C2E.
0%
Downloaded 2718612/1763514174 bytes, checksum is 10E3B673F03AEB28A7945CA57F33ED32.
0%
Downloaded 956/1763514174 bytes, checksum is 001D191BC2EDF21412EB82C3A003FE81.
0%
Downloaded 956/1763514174 bytes, checksum is 001D191BC2EDF21412EB82C3A003FE81.
0%
Downloaded 956/1763514174 bytes, checksum is 001D191BC2EDF21412EB82C3A003FE81.
0%..

SciHub product query responses include an MD5 checksum along with its other information, which can be used to test the file's integrity. Alternatively, since the products are always ZIP-files, https://docs.python.org/3/library/zipfile.html#zipfile.ZipFile.testzip should work as well.

If it's of any use, I have uploaded the current script I use for downloading from SciHub with a checksum test as a Gist here: https://gist.github.com/valgur/3e19ede9dd51e8bc5626
The key parts there are

• downloading with a check: https://gist.github.com/valgur/3e19ede9dd51e8bc5626#file-scihub_downloader-py-L222
• XPath for the checksum: https://gist.github.com/valgur/3e19ede9dd51e8bc5626#file-scihub_downloader-py-L109

I can also volunteer my own time for adding this functionality as well some time in December, if necessary.

Best regards,
Martin

"schannel: failed to retrieve ALPN result" when running on Windows 7

I`m running into issues on Windows 7, did anyone else encounter this problem ?

  File "C:\Users\myuser\AppData\Local\Continuum\Anaconda3\lib\site-packages\homura.py", line 167, in curl
    c.perform()
pycurl.error: (35, 'schannel: failed to retrieve ALPN result')

Might be an issue related to curl:
curl/curl#840

Looks like this is fixed in the latest version of curl 7.50:
curl/curl@34855fe

Refreshing download request?

At the moment I requested an order for 35 sentinel images amounting to approx. 200GiB. It started well, but I think it got interrupted by either the server being down or an internet interruption. The problem is that this made the sentinel download request halt/stop over-night and it wouldn't continue (see image attached, this is where it got to).

Is there anyway to refresh the download request to end the connection and reattempt to resume where it left off? I know the script has a "max_attempts" setting however not sure that it covers this type of issue?

I did a speed test on the net and the internet was definitely OK, then I reset the command request and now it is downloading as expected.

Not quite sure if/how this can be resolved...should I perhaps attempt running the python API independently rather than running the request through the CLI?

Thanks guys, and keep up the great work!!!

input geojson has to be of type Polygon - fails on type MultiPolygon

The input geojson has to contain a geometry of type Polygon. Input with a geometry of type MultiPolygon causes an error.

Ideally I'd expect the search function to loop through all polygons inside the multipolygon geometry, concatenate the results, remove duplicates and return the results for all polygongs at once.

Sentinel-3 support

This issue is supposed to be a starting point of discussion for Sentinel-3 support. As of 2016-10-20 Sentinel-3 images are published on Scihub.

• include --sentinel3 flag in CLI?
• expose selection of all Sentinel-3 instruments in CLI/API or focus on OLCI?
• does Sentinel-3 come with new search keywords? (cloud cover, etc.)
• Do search&download work or need to be refactored?
• Include tests once Sentinel-3 products leave pre-ops phase and are published on apihub

Issues with the certificate when downloading from the Sentinel Hub

Hi,

First of all, thanks for this helpful module.

I experienced an error with the certificate when trying to download a specific product from the Sentinel Hub.

Here is the command I entered in the shell followed by the error I got:

$ ~/Scripts/sentinelsat $ sentinel download <user> <password> 5c9746a6-4cac-4344-a86c-8873c487448a
Downloading 5c9746a6-4cac-4344-a86c-8873c487448a to ./S1A_IW_GRDH_1SDV_20150819T171924_20150819T171949_007335_00A129_A1AC.zip
Traceback (most recent call last):
File "/usr/local/bin/sentinel", line 9, in <module>
load_entry_point('sentinelsat==0.3', 'console_scripts', 'sentinel')()
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 700, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 680, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1027, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 873, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 508, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/sentinelsat/scripts/cli.py", line 58, in download
api.download(productid, path)
File "/usr/local/lib/python2.7/dist-packages/sentinelsat/sentinel.py", line 110, in download
download(product['url'], path=path, session=self.session)
File "/usr/local/lib/python2.7/dist-packages/homura.py", line 255, in download
hm.start()
File "/usr/local/lib/python2.7/dist-packages/homura.py", line 179, in start
raise e
pycurl.error: (60, 'server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none')

The only way to get rid of this error was to open the homura module > homura.py file > Homura::curl() method and make curl not verify the certificate of Sentinel Hub server (which is not very secure as the download operation will prone be to a man in the middle attack):

# h4k1m: don't check server certificate
c.setopt(pycurl.SSL_VERIFYPEER, 0)                                                                      
c.setopt(pycurl.SSL_VERIFYHOST, 0) 

Did you encounter this error before? and what's a more efficient way to resolve it?
Thanks.

Have query return generator to reduce server stress

@valgur, in the context of the last PR you suggested to have load_query return a generator over the pages of long (> 100) queries and that the download function advances only after downloading each page. This might reduce server stress, because the queries would not be sent right after each other.

Changing the load_query() to return a generator instead of a full list might be a good idea. As I see it, the motivation on the SciHub side for limiting the maximum query sizes was to limit spikes caused by large queries. Using a generator and doing another sub-query only when the current one's results have all been consumed would spread out the queries made, e.g. when iterating over the query results in download_all().
There is one minor problem with that approach, though. The SciHub's OpenSearch API provides the number of rows a full query result would contain but Python generators don't support len(), so exposing that information would need to be done via some other means.

I am not sure how big the benefit on the server side is, but I think this would be good practice. It works only, though, if the user does not need to list the details of all products returned by the query, before he downloads them. So in the CLI search function, we still need to load all the pages immediately.

api.download(<product_id>)

Hi,
first of all thanks for the useful tool. It already worked for me, but with the changes to the ESA data hub, I couldn't get it to run although installing the new version 0.5.1. Connecting seems to work using
api = SentinelAPI('user', 'password', 'https://scihub.copernicus.eu/dhus')
but the line
api.download('b7489729-a383-42f3-a06d-acb3e6106dc6')
gives me the error
Invalid API response. Trying again in 3 minutes.
I retrieved the number with the commandline (sentinel search ...) and also with the online search tool, so it should be right.

Thanks for checking.

Broken functionality due to SciHub upgrade

Since 08.10.2015 ESA is preparing the SciHub for Sentinel-2 distribution. During this they upgraded the datahub software and changed the format of the query returns.

This broke for instance the get_footprints method. I am working on a fix for that and just wait for the dhus to come back online so I can test it before creating a PR.

Anyone else found broken functionality that requires refactoring?

Regarding Sentinel-2:
should sentinelsat be able to download Sentinel-2 as well once it is available? I don't think it would require a lot of refactoring. How would this best be included in the CLI - a separate option, like --satellite or --sensor or document how to use the query with -q for that?

Errors: SSL Certificate or Json decoding failed

Hello,

I am trying to implement your tool on a new computer.
running on scihub.coperinicus/s2/ i get SSL certificate problem: self signed certificate in certificate chain

trying with scihub.coperinicus/dhus/: API response not valid. JSON decoding failed.

I tried logged in and with the guest//guest account. The JSON is fine and works on another computer with an older version of the sentinelsat.

Any ideas how I can get the new version to run?

timezone in datetime - python 2.X compability

In sentinel.py, suggest to replace

from datetime import datetime, date, timedelta, timezone

with

from datetime import datetime, date, timedelta
try:
  from datetime import timezone
except ImportError:
  from datetime import tzinfo, timedelta
  class timezone(tzinfo):
    def __init__(self, utcoffset, name=None):
      self._utcoffset = utcoffset
      self._name = name
    def utcoffset(self, dt):
      return self._utcoffset
    def tzname(self, dt):
      return self._name
    def dst(self, dt):
      return timedelta(0)

Download doesn't work anymore

Hello,

I've used your code and it worked very well until recently.
Now the products are found from my query but the downloaded file is just a faulty 276 byte .zip file. The download finsihes without error.

After updating all python packages (via pip) i get the following errors when a product is found and attempted to download (no error when no product):

Traceback (most recent call last):

  File "c:\anaconda2\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)

  File "c:\anaconda2\lib\runpy.py", line 72, in _run_code
    exec code in run_globals

  File "C:\Anaconda2\Scripts\sentinel.exe\__main__.py", line 9, in <module>
  File "c:\anaconda2\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)

  File "c:\anaconda2\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)

  File "c:\anaconda2\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "c:\anaconda2\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)

  File "c:\anaconda2\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)

  File "c:\anaconda2\lib\site-packages\sentinelsat\scripts\cli.py", line 97, in search
    api.download_all(path)

  File "c:\anaconda2\lib\site-packages\sentinelsat\sentinel.py", line 270, in download_all
    self.download(product['id'], path, checksum, **kwargs)

  File "c:\anaconda2\lib\site-packages\sentinelsat\sentinel.py", line 251, in download
    download(product['url'], path=path, session=self.session, **kwargs)

  File "c:\anaconda2\lib\site-packages\homura-0.1.3-py2.7.egg\homura.py", line 274, in download
    hm.start()

  File "c:\anaconda2\lib\site-packages\homura-0.1.3-py2.7.egg\homura.py", line 196, in start
    raise e

pycurl.error: (60, 'SSL certificate problem: self signed certificate in certificate chain')

Retry download when SciHub is unreachable

Lately I've had massive issues downloading data via the SciHub due to ESAs Server responding with 502/503 errors. The problem is that this also aborts the complete download session.

Should we implement a logic to retry every 5 to 10 minutes if the SciHub is unreachable? ESA suggests in their API usage policy to "retry only after a reasonable delay e.g. 5 minutes".

OpenSearch limits number of rows

Bad news from ESA:

many scripts using the OpenSearch API have been attempting to retrieve extremely large numbers of results from queries via use of the OpenSearch ‘rows' parameter. A new behaviour has been implemented in the data hub software to protect for this and allow a maximum number of rows to be returned in a single query, set at 100 for the moment.

At the moment we query up to 1500 rows. Meaning we will get HTTP Error 500 all the time.

To absolve this a new logic is required that iteratively queries the Scihub for results 1-100, 101-200, 201-300, etc. until no more results are returned.

Fortunately this can be done with the OpenSearch step parameter:

Results sets over the maximum can be obtained through paging of from different start values via use of the OpenSearch 'start' parameter.

Keep a changelog

With API breaking changes (as introduced in v0.8) I think it is high time we create a changelog. I propose one in the style of keepachangelog.com.

I started working on one here: 942d920. I'm currently trying to propagate it based on historic PRs and commit messages - so the first iteration will need some review.

New PRs should add their changes into an 'Unrealeased' section, which allows maintainers to just change the version number w/o having to writing the changelog - anyone creating a PR probably knows best what they changed/broke/fixed.

Proposed layout is:

[version] - YYYY-mm-dd

Added

• for new features.

Changed

• for changes in existing functionality.

Deprecated

• for once-stable features removed in upcoming releases.

Removed

• for deprecated features removed in this release.

Fixed

• for any bug fixes.

Security

• to invite users to upgrade in case of vulnerabilities.

Infintie loop when product is missing/deleted - bug in max_attempts

I found some unexpected behaviour using the sentinel download cli on a rolling dhus archive.

When using the -d option the download process hangs in an infinite loop, if a product that was found with the query is beeing rolled out/deleted/missing when the download for it starts.

i.e. I had a product with uuid: 6f1e3902-4bc8-4f14-bbbf-6252035b6960 which appeared as a result when the downloads started. However when the download for it was supposed to start it was allready rolled out / deleted.

The result was an infinite loop with the error message: (HTTP status: 500, code: None) No Products found with key '6f1e3902-4bc8-4f14-bbbf-6252035b6960' Trying again in 1 minute.

How come max_attempts=10 doesn't prevent this? Not finding the product should count as an attempt from how I understood the code.

Win7 x64, Python 3.5, Sentinelsat 0.8

Use logging instead of print

I would like to use sentinelsat in a operational script where I need to log to files. I could redirect output from the CLI, but it would be much more convenient if I could use Python's logging system.

Could we replace all the print statements by logging?

Single challenge I am seeing: How does homura do the status IO and how can that be logged?

I would be ready to take on this task, if you support this.

Add a gitter room to this repository.

I think this project could use a room to talk about PRs, provide troubleshooting help, etc.

I went ahead and created one: https://gitter.im/sentinelsat

The only thing missing is to add a badge to the Readme.

Sentinel-2 support

As of a few hours ago the pre-operational Sentinel-2 hub went live. Luckily the biggest change I can spot so far is the new url (which is already covered with the -u parameter) and some differences in returned metadata.

I think access to Sentinel-2 should be included in 'sentinelsat', especially as it doesn't require a lot of refactoring.

What I tested so far:

• you have to use 'https://scihub.esa.int/s2/' instead of 'https://scihub.copernicus.eu/s2' for the time being or you get a certificate error
• 'sentinel search' and 'sentinel download' behave as expected and I am currently downloading S-2 scenes with it

adding Travis CI

@willemarcel How is your stance on adding a Travis CI hook to this project?

I think as it continues to grow automated testing of new functions/refactoring for Python 2/3 should make the testing/merging of PRs a lot easier.

I just tested in on my fork and everything seems to work as easy as advertised (regarding this was my first time setting up a CI). https://travis-ci.org/Fernerkundung/sentinelsat

unpacking of keywords not working as documented

The behaviour of sentinelsat deviates from the documentation.

This example doesn't work anymore, as the keywords are not unpacked:

api.query(get_coordinates('map.geojson'), \
          "20151219", date(2015, 12, 29), \
          keywords={"platformname": "Sentinel-2", \
                    "cloudcoverpercentage": "[0 TO 30]"})

This works:

api.query(get_coordinates('map.geojson'), \
          "20151219", date(2015, 12, 29), \
          platformname = "Sentinel-2", \
         cloudcoverpercentage = "[0 TO 30]"})

We should change the docs to reflect this. (Unless someone depends on providing the keywords as dictionary).

Documentation

With new features constantly added to Scihub and therefore also to sentinelsat should the documentation be moved from the Readme.rst to something more structured?

There seem to be two popular options for this:

1. Use Sphinx or manually create the docs and use Read The Docs.
2. Create Documentation for each module with examples (e.g. like rasterio).

So far a number of the functions are not documented so users of the API can't use it to it's full potential. This way we could also clean up the Readme.rst to include a quick introduction and a separate documentation for more in-detail examples.

@willemarcel your thoughts on this?

Download doesn't work anymore

Hello,
I used sentinelsat until 4 days ago when this error showed up:

Traceback (most recent call last):
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\runpy.py", line 17
4, in _run_module_as_main
"main", fname, loader, pkg_name)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\runpy.py", line 72
, in run_code
exec code in run_globals
File "C:\Users\crossi\AppData\Local\Continuum\Anaconda2\Scripts\sentinel.exe_
main.py", line 9, in
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\clic
k\core.py", line 716, in call
return self.main(_args, *_kwargs)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\clic
k\core.py", line 696, in main
rv = self.invoke(ctx)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\clic
k\core.py", line 1060, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\clic
k\core.py", line 889, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\clic
k\core.py", line 534, in invoke
return callback(_args, *_kwargs)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\sent
inelsat\scripts\cli.py", line 83, in search
api.query(get_coordinates(geojson), start, end, *_search_kwargs)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\sent
inelsat\sentinel.py", line 130, in query
self.query_raw(query)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\sent
inelsat\sentinel.py", line 138, in query_raw
_check_scihub_response(self.content)
File "c:\users\crossi\appdata\local\continuum\anaconda2\lib\site-packages\sent
inelsat\sentinel.py", line 89, in _check_scihub_response
raise SentinelAPIError(response.status_code, code, msg, response.content)
sentinelsat.sentinel.SentinelAPIError: (HTTP status: 404, code: None) # HTTP Sta
tus 404 -

---

type Status report

message __

description The requested resource is not available.

---

Apache Tomcat/7.0.47

Any suggestions?

Reversed coordinate order in get_product_info() ?

The piece of code in SentinelAPI.get_product_info() which formats the footprint string has a misleading comment. Specifically it says:

        # parse the GML footprint to same format as returned
        # by .get_coordinates()

which is not true as GML -- parsed by the function -- stores coordinates in (lat, lon) order, while the GeoJSON -- used by get_coordinates() -- stores them in (lon, lat) order.
Thus the coordinates are actually returned in reverse coordinate order than the get_coordinates() format.

I'm not sure how much code is out there which relies on functionality of SentinelAPI.get_product_info(), but fixing it may break such code. If that is a concern, I'd just modify the comment.

Question about md5 checksum

Hi all,
I have a problem with checksum control with sentinel sat.
Download is ok, but at the end of the process, there is an error with checksum control. I can't write the corrupt_scenes.txt:

Traceback (most recent call last):
File "/home/geouser/anaconda2/bin/sentinel", line 11, in
sys.exit(cli())
File "/home/geouser/anaconda2/lib/python2.7/site-packages/click/core.py", line 716, in c all
return self.main(*args, **kwargs)
File "/home/geouser/anaconda2/lib/python2.7/site-packages/click/core.py", line 696, in mai n
rv = self.invoke(ctx)
File "/home/geouser/anaconda2/lib/python2.7/site-packages/click/core.py", line 1060, in in voke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/geouser/anaconda2/lib/python2.7/site-packages/click/core.py", line 889, in inv oke
return ctx.invoke(self.callback, **ctx.params)
File "/home/geouser/anaconda2/lib/python2.7/site-packages/click/core.py", line 534, in inv oke
return callback(*args, **kwargs)
File "/home/geouser/anaconda2/lib/python2.7/site-packages/sentinelsat/scripts/cli.py", lin e 95, in search
with open(os.path.join(path, "corrupt_scenes.txt"), "w") as outfile:
IOError: [Errno 20] Not a directory: u'/home/geouser/SENTINELSAT/DWNLD/S2A_OPER_PRD_MSIL1C_P DMC_20160928T010718_R044_V20160927T233112_20160927T233110.zip/corrupt_scenes.txt'
geouser@ubuntu:~/anaconda2/bin$

Anyone have an idea ?
( i am sorry if it's not the correct canal for this question)

change title of repository

Currently the title of the repo is "Utility to access the API of Sentinel-1 Scientific Data Hub". As sentinelsat allows access to all the Sentinel scenes available in the Hub I propose to change it to "Utility to access the API of Copernicus Sentinels Scientific Data Hub".

Error: Maximum number of 2 concurrent flows achieved

Hello,

when I try to download a series of images for a region the resulting download is a .zip file with 1kb. Opening with a text editor reveals:
<?xml version='1.0' encoding='UTF-8'?><error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><code/><message xml:lang="en">An exception occured while creating a stream : Maximum number of 2 concurrent flows achieved by the user "[user]"</message></error>

It's the only download running from this user.
Any ideas?

sentinelsat crashing when using command line and -d option

When i use
sentinel search --sentinel2 -s 20170115 user pwd aoi.json -d
i get the output

Will download 8 products Downloading [...]
and a corresponding ZIP file is created
then python crashes without an error (python.exe stopped responding)

I reinstalled python and checked if homura, PycURL and certifi are updated.

Windows 10
Python 2.7.12

scihub domain changes

This morning the Scientific Data Hub changes domain and becomes https://scihub.copernicus.eu instead of https://scihub.esa.int .
https://scihub.copernicus.eu/news/News00047
All the request will be rewriting, have you some problems for download the data?

Downloading 1 KB files.

Sometimes I got this problem. When I downloading products for date range I see that size of some files is 1KB (mean zero bytes). Like on this screenshot.

xfail SciHub tests on Travis

Should we add a logic to xfail the SciHub tests if the user/password environment variables are not set?

This would also allow Travis tests to pass on PRs.

Password typo generates an error

Hi All,

I was trying to use sentinelsat via command line using this example found on a website:

sentinel search -f -s 20140101 -e 20151004 -q "producttype=GRD" username password vietnam_coast.geojson

I have used the same geojson to accomplish the same request. Unfortunately, my password end with '....&a' and once I have done the command I get:

'a' is not recognized as an internal or external command.

If I just let usernamen and password I get, of course, another error but different:

simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

In this case seems the error is linked to the json file but I have actually copy and paste this file:

kr-stn/kr-stn.github.io@c59a202

I have also tried to use the python code:

from sentinelsat.sentinel import SentinelAPI

api = SentinelAPI('username','......&a')
api.query('-4.53 29.85, 26.75 29.85, 26.75 46.80,-4.53 46.80,-4.53 29.85')

but I get 502 error.

Furhtermore, it is implemented a way to obtain the footprint of the geojson also in Python script and not just by command line?

Thanks!

Change in SciHub API location

Looks like ESA is trying to limit the amount of data downloaded through the API.

• "API Hub is dedicated to users of the scripting interface. The API Hub Access is currently available only for users registered before the 20th of November 12:00 UTC, the user credentials as of the 20th November are valid to access this site."
• new URLs for search and download: https://scihub.esa.int/apihub/ , https://scihub.esa.int/apihub/odata/v1

Reduce statefulness of `SentinelAPI`

This is also a discussion from PR #60.

Currently, api.query stores the results in the api.products attribute, which api.download_all subsequently accesses. I agree with @valgur in his comment that this feels like a bad design choice, because the state of the SentinelAPI instance becomes very important for the whole flow query > download. What if we want to perform another query with the same API instance, while a download is in progress? Would this alter the list that the download_all routine is at the same time iterating over?

A solution would be to have query return the results and the user pass them to download_all. Another benefit of this solution would be that it becomes cleaner to filter the query results before downloading. For example, if I want to download my products grouped by date, I currently have to modify the api.products list before calling download_all but keep the original list.

A change is easy to implement but changes the way users have to deal with the API. Any opinions on whether we should pull this through?

Recurring error trying to use sentinel sat

Hi all,

This is my api query in python:
api.query(get_coordinates('E://xx_RS_Research//SENTINEL_Downloads//map.geojson'), \ "20151219", date(2015, 12, 29), \ keywords={"platformname": "Sentinel-2", \ "cloudcoverpercentage": "[0 TO 30]"})

This is my command line:
C:\Users\jens.hiestermann>sentinel search -s 20160901 --sentinel2 --cloud 30 -u "https://scihub.copernicus.eu/dhus" gti1 ######### E:\xx_RS_Research\SENTINEL_Downloads\map.geojson

I get the same error and not sure how to resolve it:
Traceback (most recent call last): File "c:\python27_64bit\lib\runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "c:\python27_64bit\lib\runpy.py", line 72, in _run_code exec code in run_globals File "C:\Python27_64bit\Scripts\sentinel.exe\__main__.py", line 9, in <module> File "c:\python27_64bit\lib\site-packages\click\core.py", line 716, in __call__ return self.main(*args, **kwargs) File "c:\python27_64bit\lib\site-packages\click\core.py", line 696, in main rv = self.invoke(ctx) File "c:\python27_64bit\lib\site-packages\click\core.py", line 1060, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "c:\python27_64bit\lib\site-packages\click\core.py", line 889, in invoke return ctx.invoke(self.callback, **ctx.params) File "c:\python27_64bit\lib\site-packages\click\core.py", line 534, in invoke return callback(*args, **kwargs) File "c:\python27_64bit\lib\site-packages\sentinelsat\scripts\cli.py", line 83, in search api.query(get_coordinates(geojson), start, end, **search_kwargs) File "c:\python27_64bit\lib\site-packages\sentinelsat\sentinel.py", line 130, in query self.query_raw(query) File "c:\python27_64bit\lib\site-packages\sentinelsat\sentinel.py", line 138, in query_raw _check_scihub_response(self.content) File "c:\python27_64bit\lib\site-packages\sentinelsat\sentinel.py", line 89, in _check_scihub_response raise SentinelAPIError(response.status_code, code, msg, response.content) sentinelsat.sentinel.SentinelAPIError: (HTTP status: 500, code: None) # HTTP Status 500 - Request processing failed; nested exception is java.lang.IllegalArgumentException: parameterrows` exceeds the maximum value
(100)

---

type Exception report

message Request processing failed; nested exception is java.lang.IllegalArgumentException: parameter rows exceeds the maximum value (100)

description The server encountered an internal error that prevented it from fulfilling this request.

exception

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalArgumentException: parameter `rows` exceeds the maximum value (100)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    fr.gael.dhus.spring.CacheControlFilter.doFilter(CacheControlFilter.java:47)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    fr.gael.dhus.spring.security.filter.PreAuthFilter.doFilter(PreAuthFilter.java:79)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
    org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)

root cause

java.lang.IllegalArgumentException: parameter `rows` exceeds the maximum value (100)
    fr.gael.dhus.server.http.webapp.search.controller.SearchController.search(SearchController.java:118)
    fr.gael.dhus.server.http.webapp.search.controller.SearchController$$FastClassBySpringCGLIB$$e2477ad5.invoke(<generated>)
    org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)
    org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    fr.gael.dhus.server.http.webapp.search.controller.SearchController$$EnhancerBySpringCGLIB$$7497c8f.search(<generated>)
    sun.reflect.GeneratedMethodAccessor320.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:606)
    org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
    org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
    org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
    org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
    org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    fr.gael.dhus.spring.CacheControlFilter.doFilter(CacheControlFilter.java:47)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    fr.gael.dhus.spring.security.filter.PreAuthFilter.doFilter(PreAuthFilter.java:79)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
    org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.47 logs.

---

Apache Tomcat/7.0.47`

Please assist because I am very keen to use this awesome tool.