microsoft / planetarycomputer Goto Github PK

Issues, discussions, and information about the Microsoft Planetary Computer

Home Page: https://planetarycomputer.microsoft.com/

License: MIT License

planetarycomputer's Introduction

Microsoft Planetary Computer - Issues and Discussions

This repository hosts issues and discussions that users can use to provide feedback and get help in using the Planetary Computer.

If you are experiencing issues with using the Planetary Computer, please file an issue on this repository. If you're looking for community feedback on an idea or suggestion, consider starting a discussion instead.

Links

Microsoft Planetary Computer

https://planetarycomputer.microsoft.com/

The main portal for the Planetary Computer. There you can learn more about the platform, browse our datasets, and read the documentation.

Open source repositories

The Planetary Computer has open sourced a number of its components, which you will find below. These projects rely on many others in the open source ecosystem as well, and you can find out more by exploring the various codebases.

Planetary Computer APIs - Backend
Planetary Computer Data Catalog - Frontend and documentation
Planetary Computer Examples - Example Jupyter notebooks specific to datasets
Planetary Computer Python package - Python library for interacting with the Microsoft Planetary Computer
Planetary Computer Tasks (aka pctasks) - Data Ingestion

planetarycomputer's People

Contributors

Stargazers

Watchers

Forkers

filefromakash odermody wangzhen-ryan andtise janeferdinant stevefirsake 777arc fkatada microsoftscat hgai000 ren-maomao l16by

planetarycomputer's Issues

POST search with complex MultiPolygon returns 500

Querying with the attached body to POST /search returns a status code 500 and the error message:

<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN' 'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'><html xmlns='http://www.w3.org/1999/xhtml'><head><meta content='text/html; charset=utf-8' http-equiv='content-type'/><style type='text/css'>body {font-family:Arial; margin-left:40px; }img  { border:0 none; }#content { margin-left: auto; margin-right: auto }#message h2 { font-size: 20px; font-weight: normal; color: #000000; margin: 34px 0px 0px 0px }#message p  { font-size: 13px; color: #000000; margin: 7px 0px 0px0px}#errorref { font-size: 11px; color: #737373; margin-top: 41px }</style><title>Service unavailable</title></head><body><div id='content'><div id='message'><h2>Our services aren't available right now</h2><p>We're working to restore all services as soon as possible. Please check back soon.</p></div><div id='errorref'><span>0iCZ9YgAAAABBl0Qd6v88ToXfn/noYxcZTU5aMjIxMDYwNjEzMDM5ADkyN2FiZmE2LTE5ZjYtNGFmMS1hMDlkLWM5NTlkOWExZTY0NA==</span></div></div></body></html>

q.json.txt

The shape is a fairly complex multipolygon in the New Ireland / New Britain area of Papua New Guinea.

Update R environment

Dear @TomAugspurger, would it be possible to update the R environment for the latest version, as follows:

(a) R: at least version 4.1., if possible 4.2.2
(b) sits: version 1.2
(c) Other packages
dplyr (>= 1.0.0),
gdalcubes (>= 0.6.0),
luz (>= 0.3.0),
parallel (>= 4.0.5),
purrr (>= 0.3.0),
rstac (>= 0.9.1-5),
sf (>= 1.0),
slider (>= 0.2.0),
stars (>= 0.6),
terra (>= 1.6),
testthat (>= 3.1.3),
tibble (>= 3.1),
tidyr (>= 1.2.0),
tmap (>= 3.3),
torch (>= 0.9.0),
torchopt(>= 0.1.2)

Many thanks!!

Missing Granules in the MOD11A2 Product

Below, is the data coverage over Canada and clearly missing a granule after querying PC STAC (left) with the same bounding box as used on the EarthData (right). In some date ranges, the number of returned granules is around 8 or 9 - whereas using EarthData gives a consistent 14 granules at any temporally ranged period for this AOI.

bounding_box = [-102.041016,41.836828,-50.493200,62.794935]
polygon = shapely.geometry.box(*bounding_box, ccw=True) 
polygon = gpd.GeoDataFrame(index=[0], crs='EPSG:4326', geometry=[polygon])
footprint = polygon.to_crs('epsg:4326').geometry[0].envelope
date_range = '2001-01-01/2001-12-30'

items = api.search( 
    collections = ['modis-11A2-061'], 
    intersects = footprint, 
    query={"eo:cloud_cover": {"lt": MAX_CLOUD}}, 
    datetime = date_range, 
).get_all_items()

Sentinel-2 BOA_ADD_OFFSET harmonisation

Hi all,

I wonder whether you might consider harmonisation of the Sentinel-2 data concerning the BOA_ADD_OFFSET, which was introduced in Jan 2022?

The Google Earth Engine do this (https://developers.google.com/earth-engine/datasets/catalog/sentinel-2) as does sentinel-hub (https://forum.sentinel-hub.com/t/sentinel-2-processing-baseline-changes-and-harmonizevalues/4635)

The BOA_ADD_OFFSET value is provided within the product-metadata XML file, but it is not very easy to read this offset when using several images. Even assuming the offset remains at -1000 and isn't read from the header file, it is still something that all users need to remember to apply, and this isn't documented on the planetary computer Sentinel-2 page.

Pete

Data download & Dask both failing

Hello,

I am trying to subset CMIP6 data using the MPC and save it to my local machine.

I have been using the same code for months now to subset and download data from the MPC, but now I can't seem to save and download anything. There really isn't anything crazy going on with the one function that I have written and nothing has changed on my end from when it was working before.

My guess is there is an issue with persisting the data from blob storage to local storage within my hub environment. The line for persisting throws no errors, but then trying to actually load() and save to netCDF is where everything fails and runs until it times out.

Also, my Dask dashboard is lifeless now when I try to see what is going on, showing no cores even to begin with when I have set up a minimum 4 cores to be scaled adaptively to 24 as needed. Dask will show tasks being added to the progress panel, but no progress will actually develop. I have tested this by also trying to just follow along with some of the provided MPC tutorials and I still run into the same issue. Dask just adds jobs to the que within the progress bar but then there are no workers to actually distribute the work.

Any help is greatly appreciated!

Thanks

Availability of opencv in HUB

I found that opencv package (cv2) is not available by default at the HUB. It is very important package, that has image handling, preprocessing, machine learning etc. capabilities.

Unable to attach or mount volumes, Spawn failed: did not start in 900 seconds

The first log line (0/116 nodes are available) makes me think it's a scaling limit issue. But the status page (https://planetarycomputer-status.microsoft.com/) looks fine...

Event log
Server requested
2023-01-26T16:32:17.088705Z [Warning] 0/116 nodes are available: 51 node(s) had taint {kubernetes.azure.com/scalesetpriority: spot}, that the pod didn't tolerate, 63 Insufficient cpu, 63 Insufficient memory.
2023-01-26T16:32:17.114684Z [Normal] Successfully assigned prod/jupyter-scottyh-40uw-2eedu to aks-user-37927680-vmss00019p
2023-01-26T16:36:38Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-scottyh-40uw-2eedu], unattached volumes=[user-etc-singleuser dshm volume-scottyh-40uw-2eedu]: timed out waiting for the condition
2023-01-26T16:41:12Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-scottyh-40uw-2eedu], unattached volumes=[volume-scottyh-40uw-2eedu user-etc-singleuser dshm]: timed out waiting for the condition
2023-01-26T16:42:25Z [Warning] AttachVolume.Attach failed for volume "pvc-7c839cb1-304b-4c05-8a08-6da914f50791" : timed out waiting for external-attacher of disk.csi.azure.com CSI driver to attach volume /subscriptions/9da7523a-cb61-4c3e-b1d4-afa5fc6d2da9/resourceGroups/MC_pcc-prod-2-rg_pcc-prod-2-cluster_westeurope/providers/Microsoft.Compute/disks/restore-b66b4a4b-b3a3-4f17-ac08-7e70b5c7c670
2023-01-26T16:43:27Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-scottyh-40uw-2eedu], unattached volumes=[dshm volume-scottyh-40uw-2eedu user-etc-singleuser]: timed out waiting for the condition
Spawn failed: pod prod/jupyter-scottyh-40uw-2eedu did not start in 900 seconds!

Seems separate from #117

How to make a big-ish median mosaic

I am working on creating a mosaic of sentinel data for a sizable area of interest -- around 600km by 400km. The goal is to get a median of the non-cloudy pixels over a 2 month period. My experience is that, with the code I've developed, it's not doable reliably even on a 400-core cluster.

To dig in to this I have a reproducible example of a smaller area -- 1/16 of the above example (roughly 150km x 100km), running on a 100 core cluster. This works about 1 out of 3 runs, otherwise dying with various errors. Below is the code, and then the output of 3 runs. The first died with a missing dependent, the second worked, and the third died with set size changed. (Note as regards my previously filed issue #11 -- this is with a static cluster size where I wait for all workers before proceeding.)

Just to clarify why the code is organized the way it is -- I am applying the median over time of each band, after NaNing out pixels that are identified in the Sentinel-2 SCL band as being clouds, I.e. I'm using pixel-by-pixel cloud removal instead of using a stac query based on overall percentage cloudiness, in order to maximize the information I can get from the tiles. This is probably contributing to the complexity. But it also seems like a very reasonable thing to do.

My questions are

Is the code inefficiently organized? Is there an obviously better way to do it? Can the where/isin/median process be merged into the stack call? (even though the 'isin' uses the SCL band to mask pixels in other bands)
Why are these various missing dependent / set size changed type errors happening? Is it that I'm trying to do too much with too little worker memory? Is there some kind of rule of thumb that would let me know how many workers I need for a particular computation? The full 670GB cube should be fitting roughly in the memory of the workers -- that is the rule of thumb I used to get to 100 cores.

Side note as regards scheduler: For larger areas (e.g. the full 600km x 400km) I can't even reliable get the computation going, even with 400 cores. I think it's overwhelming the scheduler. Should I try to (can I) increase memory to the scheduler somehow?

Or perhaps the real problem is the computation is just out of scope unless it's done in a more algorithmicly aligned manner.

Very interested in your thoughts.

import numpy as np
import dask
import matplotlib.pyplot as plt
from pystac_client import Client as pystac_client
#from pystac.extensions.eo import EOExtension as eo
import planetary_computer as pc
import stackstac
from dask.distributed import LocalCluster, Client, wait

import time
start_time = time.time()
def log(*args):
    el = time.time() - start_time
    current_time = time.strftime("%H:%M:%S", time.localtime())
    print(current_time, el, flush=True, *args)

log("STARTING")

import dask_gateway

nworkers = 100
cluster = dask_gateway.GatewayCluster()
client = cluster.get_client()
cluster.scale(nworkers)
log("DASHBOARD:", cluster.dashboard_link)
client.wait_for_workers(nworkers)
log("WORKERS READY.")

aoi_bounds_zim = (27.11764655, -21.18948202, 32.8563901, -16.99215633) # big chunk of zimbabwe
(xmn,ymn,xmx,ymx) = aoi_bounds_zim
aoi_bounds_half = (xmn, ymn, (xmn+xmx)/2, (ymn+ymx)/2)  # cut it in half in each dimension (/4 in area)
aoi_bounds_quarter = (xmn, ymn, (xmn+xmx)/2, (ymn+ymx)/2)  # cut it in quarter in each dimension (/16 in area)
#aoi_bounds_tiny = (xmn, ymn, xmn+0.1, ymn+0.1)
aoi_bounds = aoi_bounds_quarter

date_range = "2020-01-01/2020-03-01"
all_bands = ['B02', 'B03', 'B04', 'B08', 'SCL']
rgb_bands = ['B04', 'B03', 'B02']

catalog = pystac_client.open("https://planetarycomputer.microsoft.com/api/stac/v1")

search = catalog.search(
    collections=["sentinel-2-l2a"],
    bbox=aoi_bounds,
    datetime=date_range,
    #query={"eo:cloud_cover": {"lt": 10}},
)

# Check how many items were returned
items = list(search.get_items())
print(f"Returned {len(items)} Items")
# Note that there are more than 1 tile here:
print("Sentinel 2 Unique Tiles:", np.unique([i.properties['s2:mgrs_tile'] for i in items]))
# Print unique CRSs:
print("EPSGS:", np.unique([i.properties['proj:epsg'] for i in items]))

# Stack it
epsg_int = 4326  # because we (may) have multiple UTM zones, stick with lat/long
chunksize = 2048

signed_items = [pc.sign(item).to_dict() for item in search.get_items()]
data = (
        stackstac.stack(
            signed_items,
            epsg=epsg_int,
            #resolution=10, 
            #bounds=bounds,
            bounds_latlon=aoi_bounds,
            assets=all_bands,
            chunksize=chunksize,)
    .where(lambda x: x > 0, other=np.nan)  # sentinel-2 uses 0 as nodata
    .assign_coords(
        time=lambda x: x.time.dt.round("D"))  # round time to daily for nicer plot labels
    .groupby('time')          # one tile per datetime
    .apply(stackstac.mosaic)  # mosaic together tiles for same datetime
    )
log("Full data size GB:", data.nbytes/1e9)
log("Dimensions:", data.dims)

cloud_scl_values = [8,9,10]  # sentinel 2 cloudy pixel SCL values
mask = ~data.sel(band='SCL').isin(cloud_scl_values)
img = (data.sel(band=rgb_bands)
        .where(mask)
        .median(dim='time'))
log("Image size GB:", img.nbytes/1e9)

log("CALLING PERSIST")
img_p = img.persist()
log("AFTER CALL TO PERSIST, WAITING")
wait(img_p)
log("PERSIST COMPLETE, PLOTTING")

SCALE = 4
img_c = (img_p
         .coarsen({'x': SCALE, 'y': SCALE}, boundary='pad')
         .mean(skipna=True, keep_attrs=True)
         .compute() )

fig,ax = plt.subplots(1,1)
img_c.plot.imshow(ax=ax, x='x', y='y', robust=True)
fig.savefig("foo.png")
plt.show()

log("AFTER PLOT")
      
cluster.close()

RUN 1 -- missing dependent

17:20:04 9.5367431640625e-07 starting
17:20:17 13.552658557891846 DASHBOARD: https://pccompute.westeurope.cloudapp.azure.com/compute/services/dask-gateway/clusters/prod.cb21467cb71b46dc9aa9634aa6e82591/status
17:25:24 319.86250376701355 workers ready.
Returned 204 Items
Sentinel 2 Unique Tiles: ['35KNS' '35KNT' '35KNU' '35KPS' '35KPT' '35KPU' '35KQS' '35KQT' '35KQU'
 '35KRS' '35KRT' '35KRU']
EPSGS: [32735]
/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/accumulate_metadata.py:151: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  props_arr = np.squeeze(np.array(props))
17:25:28 323.98693227767944 Full data size GB: 673.44177984
17:25:28 323.9869964122772 Dimensions: ('time', 'band', 'y', 'x')
17:25:29 324.8450291156769 Image size GB: 16.836044496
17:25:29 324.84509015083313 CALLING PERSIST
17:26:34 390.26140332221985 AFTER CALL TO PERSIST, WAITING
17:32:58 773.6941854953766 PERSIST COMPLETE, PLOTTING
Traceback (most recent call last):
  File "big_image.py", line 99, in <module>
    img_c = (img_p
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataarray.py", line 951, in compute
    return new.load(**kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataarray.py", line 925, in load
    ds = self._to_temp_dataset().load(**kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataset.py", line 862, in load
    evaluated_data = da.compute(*lazy_data.values(), **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py", line 568, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 2671, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 1948, in gather
    return self.sync(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 845, in sync
    return sync(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py", line 326, in sync
    raise exc.with_traceback(tb)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py", line 309, in f
    result[0] = yield future
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 1813, in _gather
    raise exception.with_traceback(traceback)
ValueError: Could not find dependent ('where-getitem-db29c7abdfe1267a3a46298a39ef2497', 2, 0, 0).  Check worker logs

real    13m6.952s
user    1m57.003s
sys     0m4.014s

RUN 2 -- successful

(notebook) jovyan@jupyter-mike-40beller-2etech:~/github/carbon/msft$ time python big_image.py
17:34:32 1.9073486328125e-06 starting
17:34:44 11.641934633255005 DASHBOARD: https://pccompute.westeurope.cloudapp.azure.com/compute/services/dask-gateway/clusters/prod.ec4908e8f66c4ef78afd69b17e6ace5b/status
17:34:56 23.47306203842163 workers ready.
Returned 204 Items
Sentinel 2 Unique Tiles: ['35KNS' '35KNT' '35KNU' '35KPS' '35KPT' '35KPU' '35KQS' '35KQT' '35KQU'
 '35KRS' '35KRT' '35KRU']
EPSGS: [32735]
/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/accumulate_metadata.py:151: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  props_arr = np.squeeze(np.array(props))
17:35:01 28.43052911758423 Full data size GB: 673.44177984
17:35:01 28.430601119995117 Dimensions: ('time', 'band', 'y', 'x')
17:35:01 29.03855848312378 Image size GB: 16.836044496
17:35:01 29.038624048233032 CALLING PERSIST
17:36:08 95.49182105064392 AFTER CALL TO PERSIST, WAITING
17:42:17 464.612357378006 PERSIST COMPLETE, PLOTTING
17:42:45 492.436571598053 AFTER PLOT

real    8m16.301s
user    2m2.096s
sys     0m8.124s

RUN 3 -- set changed size error

(notebook) jovyan@jupyter-mike-40beller-2etech:~/github/carbon/msft$ time python big_image.py 
17:45:01 2.1457672119140625e-06 STARTING
17:45:12 10.9552001953125 DASHBOARD: https://pccompute.westeurope.cloudapp.azure.com/compute/services/dask-gateway/clusters/prod.45fe85a379c447da88f06e9e6c407135/status
17:45:25 23.309569597244263 WORKERS READY.
Returned 204 Items
Sentinel 2 Unique Tiles: ['35KNS' '35KNT' '35KNU' '35KPS' '35KPT' '35KPU' '35KQS' '35KQT' '35KQU'
 '35KRS' '35KRT' '35KRU']
EPSGS: [32735]
/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/accumulate_metadata.py:151: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  props_arr = np.squeeze(np.array(props))
17:45:30 28.56306266784668 Full data size GB: 673.44177984
17:45:30 28.563136100769043 Dimensions: ('time', 'band', 'y', 'x')
17:45:31 29.170450448989868 Image size GB: 16.836044496
17:45:31 29.17051339149475 CALLING PERSIST
17:46:37 95.30292534828186 AFTER CALL TO PERSIST, WAITING
17:53:00 478.3570439815521 PERSIST COMPLETE, PLOTTING
Traceback (most recent call last):
  File "big_image.py", line 94, in <module>
    img_c = (img_p
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataarray.py", line 951, in compute
    return new.load(**kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataarray.py", line 925, in load
    ds = self._to_temp_dataset().load(**kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/core/dataset.py", line 862, in load
    evaluated_data = da.compute(*lazy_data.values(), **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py", line 568, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 2671, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 1948, in gather
    return self.sync(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 845, in sync
    return sync(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py", line 326, in sync
    raise exc.with_traceback(tb)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py", line 309, in f
    result[0] = yield future
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py", line 1813, in _gather
    raise exception.with_traceback(traceback)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/optimization.py", line 969, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/to_dask.py", line 172, in fetch_raster_window
    data = reader.read(current_window)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/rio_reader.py", line 423, in read
    reader = self.dataset
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/rio_reader.py", line 419, in dataset
    self._dataset = self._open()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/rio_reader.py", line 369, in _open
    log_event("open_dataset_initial", dict(url=self.url))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/stackstac/rio_reader.py", line 42, in log_event
    worker.log_event(topic, dict(msg, thread=_curthread()))
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/worker.py", line 820, in log_event
    self.batched_stream.send(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/batched.py", line 142, in send
    self.waker.set()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/tornado/locks.py", line 224, in set
    for fut in self._waiters:
RuntimeError: Set changed size during iteration

real    8m8.199s
user    1m53.300s
sys     0m3.607s

any way I can increase the disk storage on MPC temporarily

Hi:

I can see for the CPU instance I have on MPC, It comes with 43G memory but only 14G diskspace.

The memory is good but disk space is too limited. Any way I can increase it for regular job?

Many thanks,

Bad FNF title

https://planetarycomputer.microsoft.com/dataset/alos-fnf-mosaic has title of just "C"

cannot setup pycaret on pccompute

!pip install pycaret

got the following error message, any idea?

error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [388 lines of output]
      Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation == "CPython"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation != "CPython"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
      Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment
      Collecting setuptools
        Using cached setuptools-65.5.0-py3-none-any.whl (1.2 MB)
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting Cython>=0.28.5
        Using cached Cython-0.29.32-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB)
      Collecting numpy==1.17.3
        Using cached numpy-1.17.3.zip (6.4 MB)
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting scipy>=0.19.1
        Using cached scipy-1.9.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.8 MB)
        Using cached scipy-1.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (33.8 MB)
        Using cached scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.9 MB)
        Using cached scipy-1.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.9 MB)
        Using cached scipy-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.2 MB)
      Building wheels for collected packages: numpy
        Building wheel for numpy (setup.py): started
        Building wheel for numpy (setup.py): finished with status 'error'
        error: subprocess-exited-with-error
      
        × python setup.py bdist_wheel did not run successfully.
        │ exit code: 1
        ╰─> [152 lines of output]
            Running from numpy source directory.
            blas_opt_info:
            blas_mkl_info:
            customize UnixCCompiler
            C compiler: /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /srv/conda/envs/notebook/include -fPIC -O2 -isystem /srv/conda/envs/notebook/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC
      
            creating /tmp/tmpc104cakj/tmp
            creating /tmp/tmpc104cakj/tmp/tmpc104cakj
            compile options: '-MMD -MF /tmp/tmpc104cakj/file.c.d -c'
            x86_64-conda-linux-gnu-cc: /tmp/tmpc104cakj/file.c
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
            /bin/sh: 1: svnversion: not found
            non-existing path in 'numpy/distutils': 'site.cfg'
            lapack_opt_info:
            lapack_mkl_info:
            customize UnixCCompiler
            C compiler: /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /srv/conda/envs/notebook/include -fPIC -O2 -isystem /srv/conda/envs/notebook/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC
      
            creating /tmp/tmp3xjwteue/tmp
            creating /tmp/tmp3xjwteue/tmp/tmp3xjwteue
            compile options: '-MMD -MF /tmp/tmp3xjwteue/file.c.d -c'
            x86_64-conda-linux-gnu-cc: /tmp/tmp3xjwteue/file.c
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
            /srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py:262: UserWarning: Unknown distribution option: 'define_macros'
              warnings.warn(msg)
            running bdist_wheel
            running build
            running config_cc
            unifing config_cc, config, build_clib, build_ext, build commands --compiler options
            running config_fc
            unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
            running build_src
            build_src
            building py_modules sources
            creating build
            creating build/src.linux-x86_64-3.9
            creating build/src.linux-x86_64-3.9/numpy
            creating build/src.linux-x86_64-3.9/numpy/distutils
            building library "npymath" sources
            get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
            customize Gnu95FCompiler
            Could not locate executable gfortran
            Could not locate executable f95
            customize IntelFCompiler
            Could not locate executable ifort
            Could not locate executable ifc
            customize LaheyFCompiler
            Could not locate executable lf95
            customize PGroupFCompiler
            Could not locate executable pgfortran
            customize AbsoftFCompiler
            Could not locate executable f90
            Could not locate executable f77
            customize NAGFCompiler
            customize VastFCompiler
            customize CompaqFCompiler
            Could not locate executable fort
            customize IntelItaniumFCompiler
            Could not locate executable efort
            Could not locate executable efc
            customize IntelEM64TFCompiler
            customize GnuFCompiler
            Could not locate executable g77
            customize G95FCompiler
            Could not locate executable g95
            customize PathScaleFCompiler
            Could not locate executable pathf95
            customize NAGFORCompiler
            Could not locate executable nagfor
            don't know how to compile Fortran code on platform 'posix'
            C compiler: /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /srv/conda/envs/notebook/include -fPIC -O2 -isystem /srv/conda/envs/notebook/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC
      
            compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/srv/conda/envs/notebook/include/python3.9 -c'
            x86_64-conda-linux-gnu-cc: _configtest.c
            /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc _configtest.o -o _configtest
            /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
            /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
            collect2: error: ld returned 1 exit status
            failure.
            removing: _configtest.c _configtest.o _configtest.o.d
            Traceback (most recent call last):
              File "<string>", line 2, in <module>
              File "<pip-setuptools-caller>", line 34, in <module>
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/setup.py", line 443, in <module>
                setup_package()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/setup.py", line 435, in setup_package
                setup(**metadata)
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/core.py", line 171, in setup
                return old_setup(**new_attr)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
                return distutils.core.setup(**attrs)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
                return run_commands(dist)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
                dist.run_commands()
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
                self.run_command(cmd)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
                super().run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
                cmd_obj.run()
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
                self.run_command('build')
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
                self.distribution.run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
                super().run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build.py", line 47, in run
                old_build.run(self)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
                self.run_command(cmd_name)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
                self.distribution.run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
                super().run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 142, in run
                self.build_sources()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 153, in build_sources
                self.build_library_sources(*libname_info)
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 286, in build_library_sources
                sources = self.generate_sources(sources, (lib_name, build_info))
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 369, in generate_sources
                source = func(extension, build_dir)
              File "numpy/core/setup.py", line 669, in get_mathlib_info
                raise RuntimeError("Broken toolchain: cannot link a simple C program")
            RuntimeError: Broken toolchain: cannot link a simple C program
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for numpy
        Running setup.py clean for numpy
        error: subprocess-exited-with-error
      
        × python setup.py clean did not run successfully.
        │ exit code: 1
        ╰─> [10 lines of output]
            Running from numpy source directory.
      
            `setup.py clean` is not supported, use one of the following instead:
      
              - `git clean -xdf` (cleans all files)
              - `git clean -Xdf` (cleans all versioned files, doesn't touch
                                  files that aren't checked into the git repo)
      
            Add `--force` to your command to use it anyway if you must (unsupported).
      
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed cleaning build dir for numpy
      Failed to build numpy
      Installing collected packages: wheel, setuptools, numpy, Cython, scipy
        Running setup.py install for numpy: started
        Running setup.py install for numpy: finished with status 'error'
        error: subprocess-exited-with-error
      
        × Running setup.py install for numpy did not run successfully.
        │ exit code: 1
        ╰─> [163 lines of output]
            Running from numpy source directory.
      
            Note: if you need reliable uninstall behavior, then install
            with pip instead of using `setup.py install`:
      
              - `pip install .`       (from a git repo or downloaded source
                                       release)
              - `pip install numpy`   (last NumPy release on PyPi)
      
      
            blas_opt_info:
            blas_mkl_info:
            customize UnixCCompiler
            C compiler: /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /srv/conda/envs/notebook/include -fPIC -O2 -isystem /srv/conda/envs/notebook/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC
      
            creating /tmp/tmphqamwzwz/tmp
            creating /tmp/tmphqamwzwz/tmp/tmphqamwzwz
            compile options: '-MMD -MF /tmp/tmphqamwzwz/file.c.d -c'
            x86_64-conda-linux-gnu-cc: /tmp/tmphqamwzwz/file.c
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
            /bin/sh: 1: svnversion: not found
            non-existing path in 'numpy/distutils': 'site.cfg'
            lapack_opt_info:
            lapack_mkl_info:
            customize UnixCCompiler
            C compiler: /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /srv/conda/envs/notebook/include -fPIC -O2 -isystem /srv/conda/envs/notebook/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC
      
            creating /tmp/tmp6ju63116/tmp
            creating /tmp/tmp6ju63116/tmp/tmp6ju63116
            compile options: '-MMD -MF /tmp/tmp6ju63116/file.c.d -c'
            x86_64-conda-linux-gnu-cc: /tmp/tmp6ju63116/file.c
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
              FOUND:
                libraries = ['mkl_rt', 'pthread']
                library_dirs = ['/srv/conda/envs/notebook/lib']
                define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
                include_dirs = ['/usr/local/include', '/usr/include', '/srv/conda/envs/notebook/include']
      
            /srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py:262: UserWarning: Unknown distribution option: 'define_macros'
              warnings.warn(msg)
            running install
            /srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
              warnings.warn(
            running build
            running config_cc
            unifing config_cc, config, build_clib, build_ext, build commands --compiler options
            running config_fc
            unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
            running build_src
            build_src
            building py_modules sources
            building library "npymath" sources
            get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
            customize Gnu95FCompiler
            Could not locate executable gfortran
            Could not locate executable f95
            customize IntelFCompiler
            Could not locate executable ifort
            Could not locate executable ifc
            customize LaheyFCompiler
            Could not locate executable lf95
            customize PGroupFCompiler
            Could not locate executable pgfortran
            customize AbsoftFCompiler
            Could not locate executable f90
            Could not locate executable f77
            customize NAGFCompiler
            customize VastFCompiler
            customize CompaqFCompiler
            Could not locate executable fort
            customize IntelItaniumFCompiler
            Could not locate executable efort
            Could not locate executable efc
            customize IntelEM64TFCompiler
            customize GnuFCompiler
            Could not locate executable g77
            customize G95FCompiler
            Could not locate executable g95
            customize PathScaleFCompiler
            Could not locate executable pathf95
            customize NAGFORCompiler
            Could not locate executable nagfor
            don't know how to compile Fortran code on platform 'posix'
            C compiler: /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /srv/conda/envs/notebook/include -fPIC -O2 -isystem /srv/conda/envs/notebook/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC
      
            compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/srv/conda/envs/notebook/include/python3.9 -c'
            x86_64-conda-linux-gnu-cc: _configtest.c
            /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc _configtest.o -o _configtest
            /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
            /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/12.1.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
            collect2: error: ld returned 1 exit status
            failure.
            removing: _configtest.c _configtest.o _configtest.o.d
            Traceback (most recent call last):
              File "<string>", line 2, in <module>
              File "<pip-setuptools-caller>", line 34, in <module>
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/setup.py", line 443, in <module>
                setup_package()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/setup.py", line 435, in setup_package
                setup(**metadata)
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/core.py", line 171, in setup
                return old_setup(**new_attr)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
                return distutils.core.setup(**attrs)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
                return run_commands(dist)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
                dist.run_commands()
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
                self.run_command(cmd)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
                super().run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/install.py", line 62, in run
                r = self.setuptools_run()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/install.py", line 36, in setuptools_run
                return distutils_install.run(self)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 698, in run
                self.run_command('build')
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
                self.distribution.run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
                super().run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build.py", line 47, in run
                old_build.run(self)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run
                self.run_command(cmd_name)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
                self.distribution.run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
                super().run_command(command)
              File "/srv/conda/envs/notebook/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 142, in run
                self.build_sources()
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 153, in build_sources
                self.build_library_sources(*libname_info)
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 286, in build_library_sources
                sources = self.generate_sources(sources, (lib_name, build_info))
              File "/tmp/pip-install-db5bfz0b/numpy_461db2cffacb4d36bfe6ada6a07d4bb1/numpy/distutils/command/build_src.py", line 369, in generate_sources
                source = func(extension, build_dir)
              File "numpy/core/setup.py", line 669, in get_mathlib_info
                raise RuntimeError("Broken toolchain: cannot link a simple C program")
            RuntimeError: Broken toolchain: cannot link a simple C program
            [end of output]

pystac_client sortby='datetime' does not result in monotonic sort

Reproducible snippet below:

import os
import pystac_client
import planetary_computer
import geopandas as gpd
import pandas as pd

# print(os.environ['JUPYTER_IMAGE'])

stac_client = pystac_client.Client.open('https://planetarycomputer.microsoft.com/api/stac/v1',
                                        modifier = planetary_computer.sign_inplace, #new!
                                       )
search = stac_client.search(
    collections = ['sentinel-1-rtc'],
    bbox = [-121, 46, -120, 47],
    datetime = '2021',
    sortby = 'datetime',
)

items = search.get_all_items()
print(f'Returned {len(items)} items')

gf = gpd.GeoDataFrame.from_features( items.to_dict(), crs='EPSG:4326')
gf['stac_id'] = [item.id for item in items]
gf['datetime'] = pd.to_datetime(gf.datetime)

print(gf.datetime.is_monotonic) 

gf.loc[6:11].datetime

pcccr.azurecr.io/public/planetary-computer/python:2022.9.16.0
Returned 207 items
False
6    2021-01-22 14:05:02.336295+00:00
7    2021-02-08 14:13:11.920637+00:00
8    2021-03-04 14:13:11.402244+00:00
9    2021-01-12 01:53:16.141408+00:00
10   2021-01-15 14:13:12.709696+00:00
11   2021-01-17 02:01:13.283701+00:00

How to ingest my own data layers through MPC's, or I have to build my own stac service and make it accessible by MPC?

Authentication Error: When trying to write the data more than 5 yrs into netcdf file

Downscale version: Conventions: CF-1.7 CMIP-6.2
activity_id: ScenarioMIP
contact: [email protected]
data_specs_version: 01.00.30
Python version: Python 3.9
Operating System: Ubuntu

Description

ClientAuthenticationError: Operation returned an invalid status 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.'
ErrorCode:AuthenticationFailed

Ran into a deserialization error. Ignoring since this is failsafe deserialization
Traceback (most recent call last):
File "/home/user/my_project_dir/anaconda3/lib/python3.9/site-packages/msrest/serialization.py", line 1509, in failsafe_deserialize

What I Did:

I am calculating month Tamax mean and then writing it into netcdff file as I need the downscale data to be fed as input in a model
subset = all_datasets.sel(lon=slice(25, 45),lat=slice(-10, 10),)
mtasmax = subset.resample(time="M").mean()
delayed_obj = mtasmax.sel( time=slice("2025-01-01", "2030-12-31")).to_netcdf('rgrp_yr2_2025_2030.nc',compute=False)

I am also using the temporary url (I think?) because in the documentation it was written that for this dataset tokens not need also the url should be "A SAS token is needed whenever you want to access Planetary Computer data at an Azure Blob URL. For example, an Azure Blob URL looks like:

https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/141/045/LC08_L2SP_141045_20210329_20210402_02_T1/LC08_L2SP_141045_20210329_20210402_02_T1_SR_B1.TIF "

this is what I am using:
catalog = pystac_client.Client.open(
"https://planetarycomputer.microsoft.com/api/stac/v1/",
modifier=planetary_computer.sign_inplace,
)

if I run for 10 years or 5 years or if the notebook is running for quite some time it shows this Client authentication error

can some please in regards to this?

sortby cloud cover, item order incorrect

When trying to sort an item search by "eo:cloud_cover", I receive a list that is only partly ordered.

For example, the following query contains an element that does not match the expected result: search via GET

result:

{
    "type": "FeatureCollection",
    "features": [
        {
            "id": "S2B_MSIL2A_20220222T101019_R022_T33UVP_20220228T142848",
            "collection": "sentinel-2-l2a",
            "properties": {
                "eo:cloud_cover": 89.252603,
            },
        },
        {
            "id": "S2A_MSIL2A_20220309T100841_R022_T33UVP_20220309T204833",
            "collection": "sentinel-2-l2a",
            "properties": {
                "eo:cloud_cover": 9.008443,
            },
        },
        {
            "id": "S2A_MSIL2A_20220408T100601_R022_T33UVP_20220408T223422",
            "collection": "sentinel-2-l2a",
            "properties": {
                "eo:cloud_cover": 94.83971,
            },
        }
    ]
}

result as textfile

Error in installation of torch library in R Kernel on MSPC

Describe the bug
Thank you for a great service. I am reporting a bug related to an installation error for torch package on MSPC

To Reproduce
install.packages("torch")

If reporting a change from previous versions

Please read https://cran.r-project.org/web/packages/sits/news/news.html first.

Additional context

Warning message in install.packages("torch"):
“installation of package ‘torch’ had non-zero exit status”
Updating HTML index of packages in '.Library'

Making 'packages.html' ...
 done

Details

R version 4.1.3 (2022-03-10) Platform: x86_64-conda-linux-gnu (64-bit) Running under: Ubuntu 22.04 LTS

Matrix products: default
BLAS/LAPACK: /srv/conda/envs/notebook/lib/libopenblasp-r0.3.21.so

locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] fansi_1.0.3 crayon_1.5.1 digest_0.6.29 utf8_1.2.2
[5] IRdisplay_1.1 repr_1.1.4 lifecycle_1.0.1 jsonlite_1.8.0
[9] evaluate_0.16 pillar_1.8.1 rlang_1.0.5 cli_3.4.0
[13] uuid_1.1-0 vctrs_0.4.1 IRkernel_1.3 tools_4.1.3
[17] glue_1.6.2 fastmap_1.1.0 compiler_4.1.3 base64enc_0.1-3
[21] pbdZMQ_0.3-7 htmltools_0.5.3

Problem loading NAIP 2016 scenes through QGIS STAC API Browser

Hello.
I'm trying to look at 2016 NAIP scenes over Santa Barbara County in CA using the STAC API Browser Plugin on QGIS. All the scenes I have loaded so far look washed out. Is there a fix for this? I need to visually select some places within the images but the incorrect coloring doesn't allow it. Thank for you help!

Collections endpoint on prod points to staging

Hey there PlanetaryComputer team,

I just noticed that when you visit the production STAC collections endpoint the individual collection feature links href's point to the staging environment, eg

{

   id: "io-lulc",
   links: [{
       href: "https://planetarycomputer-staging.microsoft.com/api/stac/v1/collections/io-lulc/items"
       rel: "items"
      type: "application/geo+json"
   }]
}

I'm guessing something has gone a little astray :)

Cheers
Rowan

Really hard to recover from interrupted kernel

I typically can't get the cluster to stop/shutdown if I interrupt my kernel while a graph was running. Calling cluster.shutdown() or cluster.close() just hangs. I do see there is an issue for dask gateway (dask/dask-gateway#155) that seems to be related.

But when this happens to me on MPC, I have the further problem that when I restart my kernel, I often acquire a cluster which is "lame" and refuses to start executing any tasks. Perhaps this is because there are still old resources allocated to me?

Basically it is very hard to recover from interrupted jobs. Thoughts on how I can work around this?

National Wetlands Inventory example notebook not working

Excited to see the newly added National Wetlands Inventory. I just tested the example notebook and got the following error.
https://planetarycomputer.microsoft.com/dataset/fws-nwi

item = collection.get_item("DC")

HTTPError: 404 Client Error: Not Found for url: https://planetarycomputer.microsoft.com/api/sas/v1/token/landcoverdata/fws-nwi-onboarding

Spawn failed: Server Error

How can I solve this problem?

Thanks

Planetary Computer Hub Fails to Launch

Hi,

When trying to launch the jupyter hub service I get a failure, which looks like the volume with my data isn't mounting for some reason. I don't know if this is working for other people but this was working for me until sometime yesterday, as when I came to log in in the evening this failure started to occur.

This is the event log when it is trying to start up:

Server requested
2022-10-17T14:24:05.810775Z [Normal] Successfully assigned prod/jupyter-pfb-40aber-2eac-2euk to aks-user-37927680-vmss0000j7
2022-10-17T14:24:18Z [Normal] AttachVolume.Attach succeeded for volume "pvc-73654c85-e481-4874-9b36-19016c164a23"
2022-10-17T14:26:08Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-pfb-40aber-2eac-2euk], unattached volumes=[volume-pfb-40aber-2eac-2euk user-etc-singleuser dshm]: timed out waiting for the condition
2022-10-17T12:34:04Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-pfb-40aber-2eac-2euk], unattached volumes=[dshm volume-pfb-40aber-2eac-2euk user-etc-singleuser]: timed out waiting for the condition
2022-10-17T12:35:38Z [Normal] Container image "jupyterhub/k8s-network-tools:1.2.0" already present on machine
2022-10-17T12:35:38Z [Normal] Created container block-cloud-metadata
2022-10-17T12:35:38Z [Normal] Started container block-cloud-metadata
2022-10-17T12:36:00Z [Warning] Back-off restarting failed container
2022-10-17T14:24:05.810775Z [Normal] Successfully assigned prod/jupyter-pfb-40aber-2eac-2euk to aks-user-37927680-vmss0000j7
2022-10-17T14:24:18Z [Normal] AttachVolume.Attach succeeded for volume "pvc-73654c85-e481-4874-9b36-19016c164a23"
2022-10-17T14:26:08Z [Warning] Unable to attach or mount volumes: unmounted volumes=[volume-pfb-40aber-2eac-2euk], unattached volumes=[volume-pfb-40aber-2eac-2euk user-etc-singleuser dshm]: timed out waiting for the condition
2022-10-17T14:28:18Z [Warning] Back-off restarting failed container
Spawn failed: Server at http://10.244.84.8:8888/compute/user/[email protected]/ didn't respond in 30 seconds

Any suggestions?

Many thanks,

Pete

Tensorflow environment failing to start

Trying to launch the tensorflow environment is currently failing.

Looking at the pod logs:

  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 573, in initialize
    return super().initialize(argv)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/traitlets/config/application.py", line 88, in inner
    return method(app, *args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/notebook/notebookapp.py", line 2151, in initialize
    self.init_webapp()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 628, in init_webapp
    super().init_webapp()
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/notebook/notebookapp.py", line 1759, in init_webapp
    self.web_app = NotebookWebApplication(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/notebook/notebookapp.py", line 179, in __init__
    settings = self.init_settings(
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/notebook/notebookapp.py", line 302, in init_settings
    nbextensions_path=jupyter_app.nbextensions_path,
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py", line 412, in nbextensions_path
    path = super().nbextensions_path
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/notebook/notebookapp.py", line 1329, in nbextensions_path
    from IPython.paths import get_ipython_dir
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/IPython/__init__.py", line 56, in <module>
    from .terminal.embed import embed
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/IPython/terminal/embed.py", line 16, in <module>
    from IPython.terminal.interactiveshell import TerminalInteractiveShell
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 35, in <module>
    from .debugger import TerminalPdb, Pdb
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/IPython/terminal/debugger.py", line 9, in <module>
    from .ptutils import IPythonPTCompleter
  File "/srv/conda/envs/notebook/lib/python3.8/site-packages/IPython/terminal/ptutils.py", line 40
    string = string.replace('...','\N{HORIZONTAL ELLIPSIS}')
                                  ^
SyntaxError: (unicode error) \N escapes not supported (can't load unicodedata module)

That indicates something strange is going on with unicode in the image. I saw this briefly with the Python image, but a rebuild fixed it.

Request for any environment results in GPU server (and spawn failure)

I have been unable to get access to a GPU PyTorch server for the last week. I thought I would try my luck on the weekend assuming there would be fewer GPU users. I was again unable to launch a server so I thought I would try a normal python environment. The message is either stuck on GPU or it is trying to actually launch a GPU server. Either way, I am unable to access any server, and I think it is unlikely my home directory is full.

Here is a screenshot from my attempt to launch a CPU python environment.

Duplicated items from pystac_client search if n_items > paging limit

It seems that a planetary computer STAC API search returns duplicated items when paging is necessary. The default search setting is 100 items per page and it appears that subsequent pages include a repeat item:

import pystac_client
import planetary_computer
import geopandas as gpd


stac_client = pystac_client.Client.open('https://planetarycomputer.microsoft.com/api/stac/v1',
                                        modifier = planetary_computer.sign_inplace, #new!
                                       )
search = stac_client.search(
    collections = ['sentinel-1-rtc'],
    bbox = [-121.68811311,   46.83262832, -120.19715332,   47.84592105],
    datetime = '2019-01-01/2019-12-31',
    limit=100, #default=100, max=1000
)

items = search.get_all_items()
print(f'Returned {len(items)} items')
# Returned 341 items

gf = gpd.GeoDataFrame.from_features( items.to_dict(), crs='EPSG:4326')
gf['stac_ids'] = [item.id for item in items]
with gpd.pd.option_context('display.max_colwidth', None):
    display(gf[gf.stac_ids.duplicated()]['stac_ids'])

#200    S1B_IW_GRDH_1SDV_20190602T140414_20190602T140439_016518_01F17B_rtc
#300    S1B_IW_GRDH_1SDV_20190212T142035_20190212T142100_014914_01BD7D_rtc
#301    S1B_IW_GRDH_1SDV_20190211T014511_20190211T014536_014892_01BCC5_rtc

You can only increase the paging limit to 1000, so a workaround is to filter out the duplicates from the search results

Potential issues in STAC items for USGS 3DEP-seamless dataset

I'm noticing inconsistencies with the USGS 3DEP elevation data when trying to load tiles into DataArrays via stackstac. I posted an issue in the stackstac repo but I'm starting to think the issue might be in the item entries in the STAC itself.

I'm encountering an issue where stackstac throws an error (which appears to be because it can't find the resolution in the item). Specifying a resolution the the stackstac.stack call manually fixes that issue, but then I end up empty (0-length) indices for band and time, and a few other attributes are missing as well.

Any idea what might be going on? MWEs are below.

To create the wonky DataArray:

import planetary_computer as pc
import stackstac
from pystac_client import Client as stac_client

bbox = (-104.79075, 30.86168, -104.77074999999999, 30.881680000000003)

# Open the catalog with pystac_client  
catalog = stac_client.open("https://planetarycomputer.microsoft.com/api/stac/v1")

# Search the catalog
search = catalog.search(collections=["3dep-seamless"], bbox=bbox)
items = list(search.items())

# We only want the high res layers
items_high_res = [
    pc.sign(item).to_dict()
    for item in items
    if item.properties["gsd"] == 10
]

# Lazily initialize our data layers
elevation = stackstac.stack(
    items_high_res,
    bounds=bbox,
    chunksize=64,
    resolution=(9.2592593e-05, 9.2592593e-05) # This is the native resolution, hard coded 
                                # as a workaround for a bug
).squeeze()

elevation

Output:

>>> elevation
<xarray.DataArray 'stackstac-2f63e84b98241f78db41c110675eccdc' (time: 0,
                                                                band: 0,
                                                                y: 217, x: 217)>
dask.array<fetch_raster_window, shape=(0, 0, 217, 217), dtype=float64, chunksize=(0, 0, 64, 64), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 
    id       (time) float64 
  * band     (band) float64 
  * x        (x) float64 -104.8 -104.8 -104.8 -104.8 ... -104.8 -104.8 -104.8
  * y        (y) float64 30.88 30.88 30.88 30.88 ... 30.86 30.86 30.86 30.86
    epsg     int64 5498
Attributes:
    spec:        RasterSpec(epsg=5498, bounds=(-104.790833794413, 30.86166680...
    crs:         epsg:5498
    transform:   | 0.00, 0.00,-104.79|\n| 0.00,-0.00, 30.88|\n| 0.00, 0.00, 1...
    resolution:  9.2592593e-05

To create a normal DataArray with a different bbox

import planetary_computer as pc
import stackstac
from pystac_client import Client as stac_client

## This bbox line is the only difference from the above code block
bbox = (-97.73474, 33.785180000000004, -97.71473999999999, 33.80518)

# open the catalog with pystac_client  
catalog = stac_client.open("https://planetarycomputer.microsoft.com/api/stac/v1")

# Search the catalog
search = catalog.search(collections=["3dep-seamless"], bbox=bbox)
items = list(search.items())

# We only want the high res layers
items_high_res = [
    pc.sign(item).to_dict()
    for item in items
    if item.properties["gsd"] == 10
]

# Lazily initialize our data layers
elevation = stackstac.stack(
    items_high_res,
    bounds=bbox,
    chunksize=64,
    resolution=(9.2592593e-05, 9.2592593e-05) # This is the native resolution, hard coded 
                                # as a workaround for a bug
).squeeze()

elevation

output:

>>> elevation
<xarray.DataArray 'stackstac-bb3aa93860305fb871c9f6e04bdf9be2' (y: 217, x: 217)>
dask.array<getitem, shape=(217, 217), dtype=float64, chunksize=(64, 64), chunktype=numpy.ndarray>
Coordinates: (12/15)
    time             datetime64[ns] 2019-11-25
    id               <U10 'n34w098-13'
    band             <U4 'data'
  * x                (x) float64 -97.73 -97.73 -97.73 ... -97.72 -97.71 -97.71
  * y                (y) float64 33.81 33.81 33.81 33.8 ... 33.79 33.79 33.79
    proj:shape       object {10812}
    ...               ...
    gsd              int64 10
    start_datetime   <U20 '2018-03-09T00:00:00Z'
    threedep:region  <U7 'n30w100'
    title            <U40 'USGS 1/3 arc-second n34w098 1 x 1 degree'
    description      <U1849 'This tile of the 3D Elevation Program (3DEP) sea...
    epsg             int64 5498
Attributes:
    spec:        RasterSpec(epsg=5498, bounds=(-97.734815244848, 33.785092741...
    crs:         epsg:5498
    transform:   | 0.00, 0.00,-97.73|\n| 0.00,-0.00, 33.81|\n| 0.00, 0.00, 1.00|
    resolution:  9.2592593e-05

STAC Search API Timeouts

Over the last week, I've been seeing intermittent outages of the Planetary Computer STAC API Search Endpoint. Requests return the Error 500, with the following body:

{
  "detail": "canceling statement due to statement timeout"
}

This doesn't seem to depend on access method, or on the content of the request - e.g. even attempting to visit the search API with no parameters via my browser times out: https://planetarycomputer.microsoft.com/api/stac/v1/search?limit=250

The Status page claims that "STAC API: Search" is operational, and no incidents are noted: https://planetarycomputer-status.microsoft.com/

Request pysptools package to be installed in the default environment

The "Abundance Maps" tool in this package is very useful in image processing

Sentinel-1-RTC proj metadata issues

I noticed recently that proj:shape and proj:bbox are incorrect for some S1 RTC products. According to https://github.com/stac-extensions/projection#projshape shape should be in Y,X order, but s1-rtc has it flipped:

https://planetarycomputer.microsoft.com/api/stac/v1/collections/sentinel-1-rtc/items/S1A_IW_GRDH_1SDV_20220805T015427_20220805T015452_044411_054CC1_rtc

 'proj:transform': [10.0, 0.0, 519100.0, 0.0, -10.0, 5454150.0, 0.0, 0.0, 1.0],
 'proj:shape': [28793, 21758],
 'proj:bbox': [523650.0, 5237390.0, 804660.0, 5453770.0],

As a consequence it appears proj:bbox is also off (proj:transform is fine):

import pystac
import planetary_computer
import rasterio

stac_item = 'https://planetarycomputer.microsoft.com/api/stac/v1/collections/sentinel-1-rtc/items/S1A_IW_GRDH_1SDV_20220805T015427_20220805T015452_044411_054CC1_rtc'
item = planetary_computer.sign(pystac.read_file(stac_item))
href = item.assets["vv"].href

with rasterio.open(href) as src:
    print(list(src.transform))
    print(src.shape)
    print(list(src.bounds))
    
#[10.0, 0.0, 519100.0, 0.0, -10.0, 5454150.0, 0.0, 0.0, 1.0]
#(21758, 28793)
#[519100.0, 5236570.0, 807030.0, 5454150.0]

possibly related: gjoseph92/stackstac#196

Server stops quickly if not actively connected to Hub

When I was initially started using Planetary Computer Hub, I was able to leave a script running on my server but close my connection to the Hub on my browser. In the last couple weeks I have noticed when I do this, my server stops soon after. I have tested closing it and returning to it within a couple minutes and it is fine, but when I leave it for a couple hours, it always seems to stop. Is this intended? Should I not expect to be able to leave a long-running script on the server? Thanks!

Update stackstac to the latest version

Would it be possible to update stackstac to the latest version (0.4.1)?

Landsat 8 Collection 2 notebook example 404

I was checking the documentation and I realise that the Landsat8 Collection 2 link is not found.

Searching and the anti-meridian

I've noticed some quirky behaviour when using a bbox search that sometimes the results include items which cross the anti-meridian.

For example this I was searching for items in the middle of Australia using a query like
https://planetarycomputer.microsoft.com/api/stac/v1/search?collections=landsat-8-c2-l2&bbox=129.885371,-29.068322,137.230638,-23.538097&ids=LC08_L2SR_072076_20211222_02_T2
Note I've included the item id in the above query just so it's obvious :)

The results include this item.
https://planetarycomputer.microsoft.com/api/stac/v1/collections/landsat-8-c2-l2/items/LC08_L2SR_072076_20211222_02_T2
The item has a geometry which travels from 179.2700366 to -178.1091546 and so it wraps around the world.

Ah the joys of dealing with the anti-meridian :)

Cheers
Rowan

Date selection bug in Explore datepicker UI

When using the datepicker in the custom filter for data acquisition dates, there seems to be a bug in the datepicker's UI. When I select a date by clicking on a day, the day before is selected. For example, if I click on June 16, I get June 15 selected instead. In the image below, if I click on the first "29" that appears (effectively May 29), nothing happens, and going back a month reveals that May 28 was in fact selected.

This does not seem to be related to a specific dataset, as I have had it happen with multiple datasets.

URL to reproduce: https://planetarycomputer.microsoft.com/explore?c=-4.4614%2C46.6354&z=5.94&v=2&d=landsat-c2-l2&s=false%3A%3A100%3A%3Atrue&ae=0&sr=desc&m=cql%3A8a04719e4e045ecff772d47fcc6ad348&r=Natural+color

Sometimes cluster.scale(n) doesn't get n workers

Hi. Apologies in advance if this is not the right place for this issue.
If I acquire a cluster as follows:

cluster = dask_gateway.GatewayCluster()
client = cluster.get_client()
cluster.scale(16)

Typically, looking in the cluster dashboard, I see fewer than 16 workers. Today when I was working, I typically saw 6 or maybe 10. How does the dask_gateway decide how many resources I get? Is there a reason I'm not getting my 16 workers?

Note I am not talking about large n. Haven't even tried more than maybe n=32.

Should Sentinel-2 STAC Collection declare EO extension?

Should the Sentinel 2 collection declare the eo extension as well? Right now it puts eo:bands on item_assets but the top-level stac_extensions array doesn't declare the EO extension. https://planetarycomputer.microsoft.com/api/stac/v1/collections/sentinel-2-l2a

@TomAugspurger pointed out https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#stac_extensions, which says

This must not declare the extensions that are only implemented in child Collection objects or child Item objects.

But I'd counter by pointing out that the EO extension spec's json schema also applies to a collection's item_assets. Therefore, I'd think that since the Sentinel 2 Collection file defines eo:bands on item_assets, the extension should be declared.

NSRDB example notebook fails on Hub (requires auth)

When trying to run the National Solar Radiation Database example notebook on the PC hub, I get a couple errors:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In [22], line 7
      4 from adlfs import AzureBlobFileSystem
      6 # Not used directly, but used by xarray to read .h5 files
----> 7 import h5pyd, h5netcdf
      9 # Year to investigate and plot
     10 year = 2015

ModuleNotFoundError: No module named 'h5pyd'

Fine, let's see if we can just list the blobs at least.

fs = AzureBlobFileSystem(account_name=storage_account_name)
annual_files = fs.glob(folder + '/*.h5')
print('Found {} annual files:'.format(len(annual_files)))
for k in range(0,10):
    print(annual_files[k])
print('...')

ClientAuthenticationError: Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:88a674e3-101e-001a-2e82-39eeab000000
Time:2023-02-05T16:54:22.8859881Z
ErrorCode:NoAuthenticationInformation
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>NoAuthenticationInformation</Code><Message>Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:88a674e3-101e-001a-2e82-39eeab000000
Time:2023-02-05T16:54:22.8859881Z</Message></Error>

So some form of auth (which isn't automatically set up for me on the Hub) is required to access the bucket?

FWIW, trying this on a whim didn't work:

pc.sas.get_token("nrel", "nrel-nsrdb")

HTTPError: 404 Client Error: Not Found for url: https://planetarycomputer.microsoft.com/api/sas/v1/token/nrel/nrel-nsrdb

Also, following the link in the "Mounting the Container" section (which seems to include a pre-generated SAS token?) gives

<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:a6a276ee-701e-006e-1783-39da5b000000 Time:2023-02-05T17:03:43.1837858Z</Message>
<AuthenticationErrorDetail>Signature did not match. String to sign used was /blob/nrel/$root nrel-nsrdb-ro 2020-08-04 c </AuthenticationErrorDetail>
</Error>

Broken tutorial: Radiant MLHub Land Cover

Hi!

I wanted to share that while working with the Radiant MLHub Land Cover tutorial I've found an issue with the chosen dataset.

On cell 5 I get an APIError: {"detail":"Collection ref_landcovernet_v1_labels do not exist."} when querying for "ref_landcovernet_v1_labels". Upon inspecting the Radiant MLHub datasets I've found that there's no longer a general purpose LandCoverNet dataset available.

There's an easy fix by replacing the collection_id with ref_landcovernet_??_v1_labels, filling in ?? with either sa for South America, af for Africa, as for Asia, eu for Europe or au for Australia.

---------------------------------------------------------------------------
APIError                                  Traceback (most recent call last)
Cell In [5], line 4
      1 collection_id = "ref_landcovernet_v1_labels"
      2 #collection_id = "ref_landcovernet_sa_v1_labels"
----> 4 collection = client.get_collection(collection_id)
      5 collection_sci_ext = ScientificExtension.ext(collection)
      6 print(f"Description: {collection.description}")

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pystac_client/client.py:232, in Client.get_collection(self, collection_id)
    229 if self._supports_collections() and self._stac_io:
    230     url = f"{self.get_self_href()}/collections/{collection_id}"
    231     collection = CollectionClient.from_dict(
--> 232         self._stac_io.read_json(url),
    233         root=self,
    234         modifier=self.modifier,
    235     )
    236     call_modifier(self.modifier, collection)
    237     return collection

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pystac/stac_io.py:198, in StacIO.read_json(self, source, *args, **kwargs)
    181 def read_json(self, source: HREF, *args: Any, **kwargs: Any) -> Dict[str, Any]:
    182     """Read a dict from the given source.
    183
    184     See :func:`StacIO.read_text <pystac.StacIO.read_text>` for usage of
   (...)
    196         given source.
    197     """
--> 198     txt = self.read_text(source, *args, **kwargs)
    199     return self.json_loads(txt)

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pystac_client/stac_api_io.py:97, in StacApiIO.read_text(self, source, *args, **kwargs)
     95 href = str(source)
     96 if bool(urlparse(href).scheme):
---> 97     return self.request(href, *args, **kwargs)
     98 else:
     99     with open(href) as f:

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pystac_client/stac_api_io.py:144, in StacApiIO.request(self, href, method, headers, parameters)
    142     raise APIError(str(err))
    143 if resp.status_code != 200:
--> 144     raise APIError.from_response(resp)
    145 try:
    146     return resp.content.decode("utf-8")

APIError: {"detail":"Collection ref_landcovernet_v1_labels does not exist."}

But applying this workaround arises a new issue: on cell 6, LabelExtension validates internally the STAC schema received from the API. But the newly proposed collection_id's schema does not match the one on database.

---------------------------------------------------------------------------
ExtensionNotImplemented                   Traceback (most recent call last)
Cell In [18], line 5
      3 first_item = next(item_search.get_items())
      4 print(first_item)
----> 5 first_item_label_ext = LabelExtension.ext(first_item)
      7 label_classes = first_item_label_ext.label_classes
      8 for label_class in label_classes:

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pystac/extensions/label.py:701, in LabelExtension.ext(cls, obj, add_if_missing)
    695 """Extends the given STAC Object with properties from the :stac-ext:`Label
    696 Extension <label>`.
    697
    698 This extension can be applied to instances of :class:`~pystac.Item`.
    699 """
    700 if isinstance(obj, pystac.Item):
--> 701     cls.validate_has_extension(obj, add_if_missing)
    702     return cls(obj)
    703 else:

File /srv/conda/envs/notebook/lib/python3.10/site-packages/pystac/extensions/base.py:176, in ExtensionManagementMixin.validate_has_extension(cls, obj, add_if_missing)
    173     cls.add_to(obj)
    175 if cls.get_schema_uri() not in obj.stac_extensions:
--> 176     raise pystac.ExtensionNotImplemented(
    177         f"Could not find extension schema URI {cls.get_schema_uri()} in object."
    178     )

ExtensionNotImplemented: Could not find extension schema URI https://stac-extensions.github.io/label/v1.0.1/schema.json in object.

PostgresSyntaxError","description":"syntax error at or near \") when fetching items

When I do a stac search against the Planetary Computer stac api (https://planetarycomputer.microsoft.com/api/stac/v1) and get the ItemSearch, then trying to get the individual items from the paginated ItemSearch, oine of the pages throws an exception with the following error
"code":"PostgresSyntaxError","description":"syntax error at or near ")""

Wraping the loop like so does not catch the exception that seems to come from stac_utils.
try: for item in items: results.append(compose_results(item, geojson_aoi)) except:

Consequently, we loose all the items in this Item Search, is an improvement that could be done to gracefully handle these errors. I suspect the problem is while getting the paginated items one of the queries parses a bad character to Postgres on Planetary server side and hence the error. But I'm out of options on how to handle it and continue with the remaining good results.

Sample code to reproduce

from pystac_client import Client
def test_reproduce_stac_api_error():
    client = Client.open("https://planetarycomputer.microsoft.com/api/stac/v1")

    item_search = client.search(
        datetime= "2022-06-01T00:00:00Z/2022-11-30T00:00:00Z",
        collections= ["sentinel-2-l2a"],
        intersects= {"coordinates": [
            [[-123.02807090795983, 49.270111606171525], [-123.02279737380513, 49.2989211208955],
             [-123.11640256572116, 49.307087919613224], [-123.17572980003439, 49.30107040967934],
             [-123.22846511640321, 49.282583461923394], [-123.23769379729633, 49.24989190143958],
             [-123.24692247818946, 49.21330330987789], [-123.2699941798202, 49.1706541972527],
             [-123.23250051250207, 49.11939296946704], [-123.07587575971581, 49.087100572108426],
             [-122.93076784010388, 49.09482384190619], [-122.8384810311727, 49.1461629742241],
             [-122.81606852043195, 49.231897087071246], [-122.99940095844943, 49.29780237148185],
             [-123.00071934143389, 49.26856207396219], [-122.83789904281912, 49.22682163950586],
             [-122.87415457489953, 49.14970281858811], [-122.9756700647242, 49.122100433404995],
             [-123.12728410796859, 49.1328844445041], [-123.22616283182361, 49.16220508925912],
             [-123.20375032108322, 49.26899220395606], [-123.07916312902576, 49.28275438380882],
             [-123.02807090795983, 49.270111606171525]]], "type": "Polygon"},
        query= {"eo:cloud_cover": {"lt": 100.0}},
        sortby= [{"field": "properties.created", "direction": "desc"},
                   {"field": "properties.eo:cloud_cover", "direction": "asc"}]
    )
    for item in item_search.items():
        print(f"Item ID {item.id}")

Will throw an error os this sort

POST https://planetarycomputer.microsoft.com/api/stac/v1/search Headers: {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Length': '1368', 'Content-Type': 'application/json'} Payload: {"limit": 100, "datetime": "2022-06-01T00:00:00Z/2022-11-30T00:00:00Z", "collections": ["sentinel-2-l2a"], "intersects": {"coordinates": [[[-123.02807090795983, 49.270111606171525], [-123.02279737380513, 49.2989211208955], [-123.11640256572116, 49.307087919613224], [-123.17572980003439, 49.30107040967934], [-123.22846511640321, 49.282583461923394], [-123.23769379729633, 49.24989190143958], [-123.24692247818946, 49.21330330987789], [-123.2699941798202, 49.1706541972527], [-123.23250051250207, 49.11939296946704], [-123.07587575971581, 49.087100572108426], [-122.93076784010388, 49.09482384190619], [-122.8384810311727, 49.1461629742241], [-122.81606852043195, 49.231897087071246], [-122.99940095844943, 49.29780237148185], [-123.00071934143389, 49.26856207396219], [-122.83789904281912, 49.22682163950586], [-122.87415457489953, 49.14970281858811], [-122.9756700647242, 49.122100433404995], [-123.12728410796859, 49.1328844445041], [-123.22616283182361, 49.16220508925912], [-123.20375032108322, 49.26899220395606], [-123.07916312902576, 49.28275438380882], [-123.02807090795983, 49.270111606171525]]], "type": "Polygon"}, "query": {"eo:cloud_cover": {"lt": 100.0}}, "sortby": [{"field": "properties.created", "direction": "desc"}, {"field": "properties.eo:cloud_cover", "direction": "asc"}], "token": "next:S2A_MSIL2A_20221027T191511_R056_T10UEV_20221028T085251"}
<Response [500]>

Problem accessing TerraClimate: open_zarr() got unexpected keyword arguments engine

Hi all, since today we are getting the following error when accessing the TerraClimate dataset:

products/biodiversity/src/stac_data_loader.py:557: in load_data
    ds = xr.open_zarr(mapper, **asset.extra_fields["xarray:open_kwargs"])

[...]

        if kwargs:
>           raise TypeError(
                "open_zarr() got unexpected keyword arguments " + ",".join(kwargs.keys())
            )
E           TypeError: open_zarr() got unexpected keyword arguments engine

It looks like open_zarr() does not like the new argument engine that was added to xarray:open_kwargs?

I added the following line to fix the problem:

asset.extra_fields["xarray:open_kwargs"].pop("engine", None)

Connect to Google Drive

My Deep Learning datasets are stored in Google Drive, of size 1.2 TB. I tried to access the Jupyterlab-GoogleDrive extension, but that is outdated. Is there any possibility to mount google drive, just like Colab pro. If I try using python-apis like pydrive/google-api etc. , I would need to download my dataset, then upload my training performances back, which is not feasible due to storage limits in Planetary Hub.
I am trying to switch from Colab Pro, but these limitations are hindering me.

Computations fail with "could not find dependent"

Frequently, large dask computations on large clusters seem to fail with "could not find dependent" errors:

/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1811                             exc = CancelledError(key)
   1812                         else:
-> 1813                             raise exception.with_traceback(traceback)
   1814                         raise exc
   1815                     if errors == "skip":

ValueError: Could not find dependent ('broadcast_to-concatenate-where-getitem-4916603adba736a1cf7a63ef1786c8d8', 8, 1, 0, 0).  Check worker logs

In checking the logs, by looking in the master logs for the item ID, and then tracking it down to a specific worker, the worker log looks like this:

distributed.worker - INFO - Start worker at: tls://10.244.47.9:36493
distributed.worker - INFO - Listening to: tls://10.244.47.9:36493
distributed.worker - INFO - dashboard at: 10.244.47.9:8787
distributed.worker - INFO - Waiting to connect to: tls://dask-258014f969ff4b288a395f5e84f2354b.prod:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 8.00 GiB
distributed.worker - INFO - Local Directory: /home/jovyan/dask-worker-space/worker-5bjfqpay
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tls://dask-258014f969ff4b288a395f5e84f2354b.prod:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Can't find dependencies {<Task "('broadcast_to-concatenate-where-getitem-bceb982d462d09c679da971dd8c4d874', 8, 2, 0, 0)" fetch>} for key ('getitem-bceb982d462d09c679da971dd8c4d874', 8, 2, 0, 0)
distributed.worker - INFO - Dependent not found: ('broadcast_to-concatenate-where-getitem-bceb982d462d09c679da971dd8c4d874', 8, 2, 0, 0) 0 . Asking scheduler
distributed.worker - INFO - Can't find dependencies {<Task "('broadcast_to-concatenate-where-getitem-bceb982d462d09c679da971dd8c4d874', 8, 2, 0, 0)" fetch>} for key ('getitem-bceb982d462d09c679da971dd8c4d874', 8, 2, 0, 0)
distributed.worker - INFO - Dependent not found: ('broadcast_to-concatenate-where-getitem-bceb982d462d09c679da971dd8c4d874', 8, 2, 0, 0) 1 . Asking scheduler
... many repeats
distributed.worker - ERROR - Handle missing dep failed, retrying Traceback (most recent call last): File "/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/worker.py", line 2550, in handle_missing_dep for dep in deps: RuntimeError: Set changed size during iteration

The general situation is I've stacked a large, multi-tile stack (25 times, 5 bands, x~~60000 y~~40000) -- so 2.5TB. The cluster size tried is varying from 45 -> 300 cores.

Can you give me general guidance on this sort of error? I can't reproduce it very reliably, so it's hard to provide a "minimal example" that can reproduce the error. Perhaps someone might have some guidance about what is happening and what I could do to explore possible reasons / solutions?

Unable to filter GBIF data

There appears to be a relatively recent issue with GBIF parquet files hosted on the Planetary Computer.

If I attempt to filter the parquet files by species name or family using something like:

import dask.dataframe as dd
df = dd.read_parquet(
    signed_asset.href,
    columns=['decimallatitude','decimallongitude','species'],
    filters=[('species', 'in', ['Acer rubrum'])],
    storage_options=signed_asset.extra_fields["table:storage_options"],
    parquet_file_extension=None,
)
data=df.get_partition(0).compute()
data

I get an error

typeError: '<=' not supported between instances of 'float' and 'str'

This code used to work.

My understanding is that there are rows that contain values that are not strings in those columns ?

Broken tutorial: Land Cover Classification

Hi!

While working with the Land Cover Classification tutorial, I've found a recurring "EOFError('Ran out of input')" when running the Aligning images block 6. This happens while running in the GPU - PyTorch VM profile, with the run all command.

2022-12-01 18:32:48,396 - distributed.worker - WARNING - Compute Failed
Key:       load_model-7b38892c0930e4df99b829abc6110f2b
Function:  load_model
args:      ()
kwargs:    {}
Exception: "EOFError('Ran out of input')"

Prior to this block, there are no errors aside from the following info log showing up on an error box after block 1.

2022-12-01 18:50:27,341 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

Add 'servers' field to the API definition document

OpenAPI allows for the use of a servers field for providing information about connectivity to a target server.

https://spec.openapis.org/oas/v3.0.3#openapi-object

The field is very useful in a variety of use cases.

Please add a servers field in the OpenAPI document.

I presume it would be something like:

    "servers": [
        {
            "description": "Microsoft Planetary Computer STAC API",
            "url": "https://planetarycomputer.microsoft.com/api/stac/v1"
        }
    ]

Spawn failed: Timeout

It looks like I'm facing the same issue as another user here, I'll delete files if I can login which I'm unable to do.

Trouble installing geowombat package

Hello,

I am attempting to install the geowombat package (https://github.com/jgrss/geowombat) on the planetary computer hub. The issue has to do with C compilers not being able to find libraries.

Attempting to install geowombat with no changes I get the following error

/srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include build/temp.linux-x86_64-cpython-38/src/geowombat/moving/_moving.o -o build/lib.linux-x86_64-cpython-38/geowombat/moving/_moving.cpython-38-x86_64-linux-gnu.so -fopenmp /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc collect2: error: ld returned 1 exit status

If try find /srv -name "*lgcc*" nothing is returned (similarly for find /usr -name "*lgcc*".)

I have tried to install various packages from conda, mamba install -c conda-forge compilers and mamba install -c conda-forge gxx_linux-64, both of which result in similar missing library errors. For compilers I get

/srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include build/temp.linux-x86_64-cpython-38/src/geowombat/moving/_moving.o -o build/lib.linux-x86_64-cpython-38/geowombat/moving/_moving.cpython-38-x86_64-linux-gnu.so -fopenmp /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/10.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /usr/lib64/libpthread_nonshared.a collect2: error: ld returned 1 exit status

Thanks!

Not able to connect with VSCODE

HI,
I've been using Planetary HUB, but I'm struggling to connect to the HUB server through VSCODE, as instructed in: https://planetarycomputer.microsoft.com/docs/overview/ui-vscode/

I have started the server, created a specific Token and used the following URL on VSCODE:

After Reloading VSCode as demanded, when I try to run a simple (2+2) cell, it asks for a "kernel".

Everything seems right... I connect to another existing (remote) jupyter server to test my local configuration (VSCODE and extension) and it worked flawlessly.

Thanks

Bad Landsat File

I'm getting an error with the file at https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/055/076/LC08_L2SR_055076_20211231_20220106_02_T2/LC08_L2SR_055076_20211231_20220106_02_T2_SR_B4.TIF

from planetary_computer import sign
import rioxarray

url = 'https://landsateuwest.blob.core.windows.net/landsat-c2/level-2/standard/oli-tirs/2021/055/076/LC08_L2SR_055076_20211231_20220106_02_T2/LC08_L2SR_055076_20211231_20220106_02_T2_SR_B4.TIF'
rioxarray.open_rasterio(sign(url)).rio.to_raster('test.tif')

ERROR 1: TIFFFillTile:Read error at row 6144, col 3840, tile 295; got 81147 bytes, expected 101991
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: IReadBlock failed at X offset 25, Y offset 9: TIFFReadEncodedTile() failed.
Traceback (most recent call last):
  File "rasterio/_io.pyx", line 936, in rasterio._io.DatasetReaderBase._read
  File "rasterio/_io.pyx", line 174, in rasterio._io.io_multi_band
  File "rasterio/_err.pyx", line 191, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: IReadBlock failed at X offset 25, Y offset 9: TIFFReadEncodedTile() failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/rioxarray/raster_array.py", line 1137, in to_raster
    return RasterioWriter(raster_path=raster_path).to_raster(
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/rioxarray/raster_writer.py", line 271, in to_raster
    data = encode_cf_variable(out_data).values.astype(numpy_dtype)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/conventions.py", line 273, in encode_cf_variable
    var = coder.encode(var, name=name)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/coding/times.py", line 664, in encode
    dims, data, attrs, encoding = unpack_for_encoding(variable)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/coding/variables.py", line 104, in unpack_for_encoding
    return var.dims, var.data, var.attrs.copy(), var.encoding.copy()
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/dataarray.py", line 670, in data
    return self.variable.data
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/variable.py", line 354, in data
    return self.values
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/variable.py", line 527, in values
    return _as_array_or_item(self._data)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/variable.py", line 267, in _as_array_or_item
    data = np.asarray(data)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/indexing.py", line 653, in __array__
    self._ensure_cached()
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/indexing.py", line 650, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/indexing.py", line 623, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/indexing.py", line 524, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/rioxarray/_io.py", line 322, in __getitem__
    return indexing.explicit_indexing_adapter(
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/xarray/core/indexing.py", line 815, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/srv/conda/envs/notebook/lib/python3.10/site-packages/rioxarray/_io.py", line 306, in _getitem
    out = riods.read(band_key, window=window, masked=self.masked)
  File "rasterio/_io.pyx", line 612, in rasterio._io.DatasetReaderBase.read
  File "rasterio/_io.pyx", line 939, in rasterio._io.DatasetReaderBase._read
rasterio.errors.RasterioIOError: Read or write failed. IReadBlock failed at X offset 25, Y offset 9: TIFFReadEncodedTile() failed.

Error in Chloris Biomass Data Notebook

Just noting an error with the Chloris biomass data notebook: chloris-biomass-example.ipynb

The image shown at the bottom obviously has errors (biomass numbers are not what they should be)

I think the error is made in the second-last code block.

nodata should be filtered out / treated as nodata BEFORE the grid is downsampled. Failure to do this means that cells with real values surrounding nodata get contaminated with nodata values. Specific code block:

"""

%%time
factor = 10
coarse_biomass = (
ds.biomass.coarsen(dim={"x": factor, "y": factor}, boundary="trim").mean().compute()
)

Filter out nodata

coarse_biomass = coarse_biomass.where(coarse_biomass < ds.attrs["_FillValue"])
coarse_biomass.shape

"""

microsoft / planetarycomputer Goto Github PK

planetarycomputer's Introduction

Microsoft Planetary Computer - Issues and Discussions

Links

Microsoft Planetary Computer

Open source repositories

planetarycomputer's People

Contributors

Stargazers

Watchers

Forkers

planetarycomputer's Issues

RUN 1 -- missing dependent

RUN 2 -- successful

RUN 3 -- set changed size error

To create the wonky DataArray:

To create a normal DataArray with a different bbox

Filter out nodata

Recommend Projects

Recommend Topics

Recommend Org

Jobs