A python package to read and calibrate NOAA and Metop AVHRR GAC and LAC data
Home Page: https://pygac.readthedocs.org/
License: GNU General Public License v3.0
Pygac is a Python package to read, calibrate and navigate data from the AVHRR instrument onboard NOAA and MetOp satellites in GAC and LAC format.
The documentation is available at https://pygac.readthedocs.io/.
Dear pygac team,
My colleagues Marie Doutriaux Boucher and Oliver Sus at EUMETSAT encountered a potential mistake in the conversion from brightness temperature to radiance in pygac.
The critical part is the linear temperature transformation in the exponent. The temperature in the equation is an effective brightness temperature. In order to use the measured brightness temperature, it is possible to re-define the constants of the linear equation.
It appears like the coefficient A has changed sign in pygac, but has not been divided by B. Since B is close to one, the results will not change dramatically. However, the differences accumulate when transforming back and forth.
Abhay Devasthale is already double checking the coefficients and comparing to patmos-x.
It would be nice to replace correct_tsm_issue.mean_filter with a pure python implementation.
A possible solution is:
import numpy as np
import bottleneck as bn
def rolling_window(a, shape): # rolling window for 2D array
s = (a.shape[0] - shape[0] + 1,) + (a.shape[1] - shape[1] + 1,) + shape
strides = a.strides + a.strides
return np.lib.stride_tricks.as_strided(a, shape=s, strides=strides)
def std_filter(data, box_size):
shape = (box_size, box_size)
border = box_size // 2
# need to surround the data with NaNs to calculate values at the boundary
padded_data = np.pad(
data, (border, border),
mode='constant',
constant_values=np.nan
)
windows = rolling_window(padded_data, shape)
n3, n2, n1, n0 = windows.shape
windows = windows.reshape((n3, n2, n1*n0))
result = bn.nanstd(windows, axis=2)
return resultUsing bottleneck instead of the equivalent numpy function reduces the processing time to become competitive with Cython or numba.
Another possibility would be an implementation with numba:
import numpy as np
import numba as nb
@nb.njit
def std_filter_2(data, box_size):
N = box_size // 2
nrows, ncols = data.shape
std = np.empty_like(data)
for j in range(nrows):
for i in range(ncols):
total = 0.0
count = 0
jj_min = max(0, j-N)
jj_max = min(nrows, j+N+1)
ii_min = max(0, i-N)
ii_max = min(ncols, i+N+1)
for jj in range(jj_min, jj_max):
for ii in range(ii_min, ii_max):
value = data[jj, ii]
if np.isnan(value):
continue
total += value
count += 1
if count == 0:
std[j, i] = np.nan
continue
mean = total / count
diff2 = 0.0
for jj in range(jj_min, jj_max):
for ii in range(ii_min, ii_max):
value = data[jj, ii]
if np.isnan(value):
continue
diff2 += (value - mean)**2
std[j, i] = np.sqrt(diff2 / count)
return stdEdit: Slightly cleaner numba implementation.
I could not process the following files:
NSS.GHRR.NC.D82038.S2015.E2207.B0323839.GC
NSS.GHRR.NC.D82039.S1655.E1825.B0325051.WI
NSS.GHRR.NC.D82041.S0251.E0445.B0327071.WI
NSS.GHRR.NC.D82041.S0812.E1006.B0327374.GC
NSS.GHRR.NC.D82042.S0428.E0622.B0328586.WI
NSS.GHRR.NC.D82044.S0211.E0405.B0331213.WI
NSS.GHRR.NC.D82045.S0014.E0203.B0332526.GC
NSS.GHRR.NC.D82046.S0524.E0718.B0334243.WI
NSS.GHRR.NC.D82046.S1844.E2024.B0335051.WI
NSS.GHRR.NC.D82047.S0134.E0328.B0335455.WI
NSS.GHRR.NC.D82048.S0122.E0316.B0336869.WI
NSS.GHRR.NC.D82048.S1148.E1334.B0337475.GC
NSS.GHRR.NC.D82049.S0637.E0823.B0338586.WI
NSS.GHRR.NC.D82049.S1137.E1322.B0338889.GC
NSS.GHRR.NC.D82050.S0435.E0629.B0339899.WI
NSS.GHRR.NC.D82050.S1627.E1757.B0340506.WI
NSS.GHRR.NC.D82051.S1253.E1440.B0341718.GC
NSS.GHRR.NC.D82052.S1100.E1246.B0343031.GC
NSS.GHRR.NC.D83183.S0343.E0537.B1042930.WI
NSS.GHRR.NC.D83183.S1721.E1850.B1043738.WI
NSS.GHRR.NC.D83184.S1834.E2019.B1045253.WI
NSS.GHRR.NC.D83185.S0128.E0322.B1045657.WI
NSS.GHRR.NC.D83185.S1656.E1827.B1046566.WI
NSS.GHRR.NC.D83186.S2138.E2326.B1048283.GC
NSS.GHRR.NC.D83187.S0812.E0959.B1048889.WI
NSS.GHRR.NC.D83187.S2302.E2314.B1049697.GC
NSS.GHRR.NC.D83189.S0748.E0934.B1051617.WI
NSS.GHRR.NC.D83190.S1605.E1741.B1053536.GC
NSS.GHRR.NC.D83191.S1723.E1853.B1055051.WI
NSS.GHRR.NC.D83192.S1350.E1536.B1056263.GC
NSS.GHRR.NC.D83193.S1320.E1343.B1057676.GC
NSS.GHRR.NC.D83193.S1519.E1704.B1057778.GC
NSS.GHRR.NJ.D97187.S0423.E0441.B1296262.WI
due to an IndexError in PODReader._adjust_clock_drift giving a traceback like:
Traceback (most recent call last):
File "../venv/bin/pygac-run", line 164, in <module>
pygac.process_file(filename, start_line, end_line)
File ".../venv/lib/python3.7/site-packages/pygac/runner.py", line 113, in process_file
sunsatangles_dir=sunsatangles_dir
File ".../venv/lib/python3.7/site-packages/pygac/reader.py", line 324, in save
self.get_lonlat()
File ".../venv/lib/python3.7/site-packages/pygac/reader.py", line 621, in get_lonlat
self._adjust_clock_drift()
File ".../venv/lib/python3.7/site-packages/pygac/pod_reader.py", line 460, in _adjust_clock_drift
complete_lons[self.scans["scan_line_number"] - min_idx] = self.lons
IndexError: index 1448 is out of bounds for axis 0 with size 1448
This might be related to the following doc-string comment in PODReader._adjust_clock_drift:
TODO: bad things might happen when scanlines are skipped.
We should add support for Python3 at some point. After having increased the test coverage 😅
It would be great if the klm and pod readers read method would accept gzip compressed files.
I already have an implementation that I would like to share.
The following things need to be or should be fixed in a patch release before I make a conda-forge package:
long_description parameter in setup.py. Since it is markdown you will also need long_description_content_type='text/markdown'.LICENSE.txt (this is the default in conda-forge, not required to rename but eh why not). It should then be included in the MANIFEST.in.After all of the above are complete:
Angles computation do not make use on the TIP Euler angles at the moment.
It would improve the accuracy of the angles if it was.
The lat lon interpolation should skip scan lines which are flagged with the NO_EARTH_LOCATION quality indicator.
See #55 (comment) for an implementation proposal.
Disclaimer: this is probably not an issue in pygac, but because it involves pygac and because of a lack of other place, I would like to have the discussion here.
@helgaweb has brought to my attention four passes from metops A and B where there is a consistent shift in x-direction. Since these are considered KLM satellites, there is no correction applied to the navigation by pygac (or satpy for that matter), so we are wondering where this can come from. @sfinkens @carloshorn any ideas? Attaching some images so you can see.
Recently a bug in pyorbital's sun-earth distance correction was fixed: pytroll/pyorbital#97.
Pygac does not use pyorbital for this, but makes the same mistake:
Line 89 in 05ba35e
I propose to use pyorbital for this.
@ninahakansson @abhaydd @kgkarl This would change the output (to the better).
In section Introduction -> Supported Data Formats of the documentation the POD text links to the KLM user guide and vice versa.
The CLARA team noticed that the masking of scanlines in pygac has changed since an earlier version. I suspect this didn't happen on purpose. In the PR history I only found a refactoring (#72) that shouldn't have changed the behaviour.
@carloshorn @abhaydd IIUC you have already started discussing this. If that is indeed a bug, could you please continue your discussion here? Then we have a proper record of it.
Left: Older version. Right: Current version
While processing 2017/18 data I noticed ~250 errors of the following type:
python bin/pygac-run testbench/NSS.GHRR.M1.D18337.S2304.E0005.B3222526.GC 0 0
[ INFO pygac.gac_reader 2019-02-22 15:36:34.498] Reading NSS.GHRR.M1.D18337.S2304.E0005.B3222526.GC
[ DEBUG pygac.gac_reader 2019-02-22 15:36:34.606] Removed 0 scanline(s) with corrupt scanline numbers
/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/geotiepoints.py:575: RuntimeWarning: invalid value encountered in arccos
90 - rad2deg(arccos(z__/EARTH_RADIUS)),
/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/geotiepoints.py:578: RuntimeWarning: invalid value encountered in arcsin
/ EARTH_RADIUS))))
[ DEBUG pygac.gac_reader 2019-02-22 15:36:36.458] Corrected 0 timestamp(s)
Traceback (most recent call last):
File "bin/pygac-run", line 104, in <module>
reader(args.filename, args.start_line, args.end_line)
File "/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_klm.py", line 579, in main
channels = reader.get_calibrated_channels()
File "/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_reader.py", line 254, in get_calibrated_channels
self.spacecraft_name)
File "/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_calibration.py", line 477, in calibrate_thermal
prt[ifix] = np.interp(ifix[0], inofix[0], prt[inofix])
File "/cmsaf/nfshome/routcm/Modules_CentOS/python/2.7.14/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2061, in interp
return interp_func(x, xp, fp, left, right)
ValueError: array of sample points is empty
This seems to be yet another data corruption, because there are two very similar files with slightly different time stamps from different ground stations:
The errror occurs with the first file, the second one is processed just fine. Currently we would select the first one because it contains more data (according to the timestamps).
I was wondering, why the Earth scene radiance (Ne) does not account for linear radiance estimate Nlin as described in the KLM User's Guide. Following 7.1.2.4-6 on page 7-9, Ne is calculated as: Ne = Nlin + Ncor
Lines 553 to 558 in 2d7fbaf
I have written a script to convert PATMOS-x coefficients from tar balls as provided on their official website to PyGAC json files as introduced in #58.
After testing the script with the coefficient version which should correspond to the current PyGAC coefficients, I noticed some typos which I would like to correct with a PR.
Lines 56 to 57 in 2d7fbaf
0.030 1.224 -0.033 ! ch3a low gain S0,S1,S2
0.213 1.224 -0.033 ! ch3a high gain S0,S1,S2
Line 214 in 2d7fbaf
-7.29 5.76 -0.1157 0.0005882 !ch4 nonlinear rad coef (Ns,b0,b1,b2) (JT)
Line 207 in 2d7fbaf
-7.29 5.76 -0.1157 0.0005882 !ch4 nonlinear rad coef (Ns,b0,b1,b2) (JT)
Line 278 in 2d7fbaf
0.022171 !a1_5
Line 352 in 2d7fbaf
-1.7002941 !a1_3b
Lines 101 to 102 in 2d7fbaf
0.113 0.900 0.000 ! ch1 low gain S0,S1,S2
0.113 0.900 0.000 ! ch1 high gain S0,S1,S2
Lines 109 to 113 in 2d7fbaf
276.659 0.051275 1.363e-06 0.0 0.0 !PRT1 coefficients
Line 162 in 2d7fbaf
-1.7764105 !a1_3b
Line 92 in 2d7fbaf
913.05397 !nu_4
...
913.05397 !nu_5
Lines 86 to 90 in 2d7fbaf
276.659 0.051275 1.363e-06 0.0 0.0 !PRT1 coefficients
I will create a PR, once #58 is merged into master, because I got these differences when comparing the current json files with the one that I produced from the tar ball on the PATMOS-x website.
Currently Pygac has its own implementation of sun-earth distance correction. This could be replaced with pyorbital.astronomy.sun_earth_distance_correction. Note that the value computed by Pyorbital would have to be squared, see pytroll/pyorbital#97.
Couldn't find them on the web. I've asked NOAA NCDC where the pages are located now.
Unittests starting with test__ are not discovered by python setup.py test:
...pygac/pygac/tests> grep test__ *.py
test_klm.py: def test__validate_header(self):
test_pod.py: def test__validate_header(self):
test_pod.py: def test__get_calibrated_channels_uniform_shape(self, get_channels):
test_pod.py: def test__adjust_clock_drift(self, avhrr_gac, get_tle_lines,
test_reader.py: def test__read_scanlines(self):
test_reader.py: def test__validate_header(self):
test_reader.py: def test__correct_data_set_name(self):
test_reader.py: def test__get_calibrated_channels_uniform_shape(self, get_channels):
Some of them are failing.
To avoid an implicit change of the input arrays, I would move the interpolation of missing/wrong data in the ICT and space counts to the POD/KLM specific reader method get_telemetry or use a copy instead of changing the input in-place.
Lines 409 to 425 in 8a7f11e
Remove derived metadata like midnight scanline and missing scanlines. This is something that can be computed by the user using the scanline acquisition times (midnight scanline) and scan line numbers (missing scanlines).
Pygac still carries around an old version of the pytroll geotiepoints. We should instead make use of the updated library.
There are many small things that I think should be improved presentation-wise.
I found some twisted numbers for the metopb coeffs in the calibration.py script.
'metopb'
'bh' 1.478 -> 1.748
'bl' 1.478 -> 1.748
source:
https://cimss.ssec.wisc.edu/patmosx/instrument_files/avhrr_1_instr.dat
I further checked metopa, noaa18 and noaa19 - these are correct!
The KLM data contains angular relationship that could be used instead of computing the angles with pyorbital.
I need it in satpy. Thanks!
Direct Usage:
https://github.com/pytroll/pygac/blob/main/doc/source/usage.rst#direct-usage
says get_reader_cls.
Actual code:
Line 34 in c5dcb32
I'm not familiar with this code otherwise so I'm not sure what else might be wrong in this documentation. One of the maintainers should review all function calls in this documentation.
At the moment, the satpy reader for pygac is dependent on the filename format of the gac files. It would be really good to have a function that takes in a filename, and by looking at the header could determine what type of data it is: pod vs klm, gac vs lac.
Currently, completely missing ICT or space counts lead to ValueError: array of sample points is empty in
Lines 416 to 418 in 8a7f11e
Example product: NSS.GHR.M2.D12306.S06 19.E0709.B3132020.MM
Instead of raising the exception, the resulting array should be filled with NaNs accompanied by a warning.
The Pygac API on readthedocs is missing. I guess the reason is that the repo is checked out but not installed and therefore the version import doesn't work (https://github.com/pytroll/pygac/blob/main/pygac/__init__.py#L27). Looking at satpy a dedicated readthedocs conda environment seems to solve the problem.
The current reader access to the quality indicators bit field, e.g.
Lines 482 to 487 in 2d7fbaf
I noticed, that the script pygac-run does not support gzip files, because it implements the factory function check_file_version. I could have reported this as a bug, but in my opinion it is a matter of software design. How about having a l1c factory class in the back-end?
I started with an implementation, but now I have some questions:
main methods for the different readers, I noticed that there are only two implementations (pod and klm) which are almost identical. Just the channels are provided differently to save_gac. Is there a reason, why the PODReader does not implement a get_calibrated_channels which calls the parent method and then provides six channels as expected by save_gac? If this is not wanted, how about an adapter like Reader._prepare_channels which defaults to call get_calibrated_channels, and is implemented in PODReader?Reader.save method which implements the logic of klm_reader.main and gets the input channels to save_gac form the answer of 2.My goal is a modification of pygac-run to enable it to process entire directories, and eventually entire tar balls.
The sun-earth distance correction factor is calculated both in gac_reader and gac_io at the moment. And when using pygac as reader from satpy, there is no access to this attribute.
The calculation should be removed from gac_io. And the value used should be accessible from satpy. If possible in a general way so that other items can be added later without the need to update satpy.
In pod_reader.py._compute_missing_lonlat(): Function get_tle_lines() will raise an IndexError if the next available tle line is not within a user-defined threshold of n days. Pygac should be able to handle this issue, as tle data are gappy, and a fix has been introduced to deal with such cases. In fact, before a recent commit (3f04306), this error has been caught and tle lines were set to emtpy strings.
One could undo that commit, i.e. put that function back into a try clause like so:
try:
tle1, tle2 = self.get_tle_lines()
except IndexError:
tle1 = tle2 = ''
This would at least allow pygac to continue.
However, this causes new problems in compute_pixels(), which calls pyorbital, thus initializing class Tle. Tle will be unhappy about tle lines being empty strings and thus raise IndexErrors.
As I do not know the code structure well enough: does it make sense to just quit compute_missing_lonlat when there are no tle data within the threshold? Or apply the tle fix here?
pygac raises an IndexError if the timestamp of the processed orbit is newer than the most recent entry in the TLE file.
Example:
NSS.GHRR.M1.D18365.S2329.E0027.B3262324.SVTLE_metopb.txt with the following last two lines:
1 38771U 12049A 18365.81382142 -.00000015 00000-0 13273-4 0 9991
2 38771 98.7294 62.2921 0002057 92.7653 26.0030 14.21477711326215
Traceback:
python bin/pygac-run testbench/NSS.GHRR.M1.D18365.S2329.E0027.B3262324.SV 0 0
[ INFO pygac.gac_reader 2019-02-22 13:57:14.217] Reading NSS.GHRR.M1.D18365.S2329.E0027.B3262324.SV
[ DEBUG pygac.gac_reader 2019-02-22 13:57:14.384] Removed 0 scanline(s) with corrupt scanline numbers
/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/geotiepoints.py:575: RuntimeWarning: invalid value encountered in arccos
90 - rad2deg(arccos(z__/EARTH_RADIUS)),
/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/geotiepoints.py:578: RuntimeWarning: invalid value encountered in arcsin
/ EARTH_RADIUS))))
[ DEBUG pygac.gac_reader 2019-02-22 13:57:16.323] Corrected 1 timestamp(s)
/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_calibration.py:554: RuntimeWarning: invalid value encountered in log
/ np.log(1.0 + nBB_num / Ne))
[ INFO pygac.gac_reader 2019-02-22 13:57:17.839] TLE filename = /cmsaf/nfshome/sfinkens/software/devel/pygac/gapfilled_tles/TLE_metopb.txt
Traceback (most recent call last):
File "bin/pygac-run", line 104, in <module>
reader(args.filename, args.start_line, args.end_line)
File "/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_klm.py", line 580, in main
sat_azi, sat_zen, sun_azi, sun_zen, rel_azi = reader.get_angles()
File "/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_reader.py", line 314, in get_angles
tle1, tle2 = self.get_tle_lines()
File "/cmsaf/nfshome/sfinkens/software/devel/pygac/pygac/gac_reader.py", line 298, in get_tle_lines
if abs(sdate - dates[iindex - 1]) < abs(sdate - dates[iindex]):
IndexError: index 3136 is out of bounds for axis 0 with size 3136
Some ideas:
reader_kwargs)pip install pygac for the latest release or pip install git+... for the latest development versiongit clone && pip install -e .) would be nice, tooThe following lines introduce a larger rounding error than necessary:
Lines 436 to 437 in 5c59f2b
Here, self.utcs is an array of type 'datetime64[ms]', and offset is an array of floating point numbers giving seconds, which requires a conversion to a time delta. However, the conversion to 'timedelta64[s]' strips of the fraction of seconds. In order to stay at the same precision level, the conversion to a time delta should be changed to (1000*offsets).astype('timedelta64[ms]').
Hello,
For our reprocessing activities at EUMETSAT, we are using your AVHR GAC data from ECMWF FS. Unfortunately, some files show an artifact on the zenith angle.
My college Kristina Petraityte has already been in contact with Stephan Finkensieper who advised us to raise a ticket here.
The Error:
Not all files affected, but for example in ECC_GAC_sunsatangles_metopb_99999_20130101T2257085Z_20130102T0039455Z.h5 on image2 at row index 10081, the values for all 409 pixels are about twice the expected value (~12000 instead of ~6000).
We tried to reproduce the file using pygac, but surprisingly in our case, the feature disappeared. This means that the software seems to be correct. However, I would recommend to add the TLE lines into the output, because we are not sure, if we are using exactly the same input as you did.
More details on our reproduction of ECC_GAC_sunsatangles_metopb_99999_20130101T2257085Z_20130102T0039455Z.h5
branch: develop
git hash: bdd31d0
TLE: '1 38771U 12049A 13001.30683292 -.00000109 00000-0 -29853-4 0 9993\n',
'2 38771 098.7267 063.7881 0001633 317.7328 042.3723 14.21473300 15006\n'
pyorbital version: 1.5.0
python: latest anaconda 2.7 64 bit installer
Many thanks in advance!
Unit tests fail on linux (Ubuntu 18.10).
$ env PYGAC_CONFIG_FILE=etc/pygac.cfg python setup.py test
[CUT]
======================================================================
ERROR: test_calibration_ir (pygac.tests.test_calibrate_klm.TestKLMCalibration)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/antonio/projects/pygac/pygac/tests/test_calibrate_klm.py", line 203, in test_calibration_ir
line_numbers=np.array([1, 2, 3]))
File "/home/antonio/projects/pygac/pygac/calibrate_klm.py", line 572, in calibrate_thermal
tprt = np.zeros((float(number_of_data_records)))
TypeError: 'float' object cannot be interpreted as an index
======================================================================
FAIL: test_calibration_vis (pygac.tests.test_calibrate_klm.TestGenericCalibration)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/antonio/projects/pygac/pygac/tests/test_calibrate_klm.py", line 76, in test_calibration_vis
self.assertTrue(np.allclose(ref1, expected[0]))
AssertionError: False is not true
======================================================================
FAIL: test_calibration_vis (pygac.tests.test_calibrate_pod.TestGenericCalibration)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/antonio/projects/pygac/pygac/tests/test_calibrate_pod.py", line 76, in test_calibration_vis
self.assertTrue(np.allclose(ref1, expected[0]))
AssertionError: False is not true
----------------------------------------------------------------------
Ran 22 tests in 6.238s
FAILED (failures=2, errors=1)
Test failed: <unittest.runner.TextTestResult run=22 errors=1 failures=2>
error: Test failed: <unittest.runner.TextTestResult run=22 errors=1 failures=2>
This isn't part of the repo itself, otherwise I'd send a PR.
The project URL is described in the repo as https://pygac.readthedocs.org/, but that leads to a 404. The correct URL is:
http://pygac.readthedocs.io/en/master
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.