nmcdev / nmc_met_io Goto Github PK

1. 为与读取MICAPS读取的dataset统一，在读取CMISS数据中，可否将dataset中datarray的数据名也改为‘data’
2. 可否编写一个类似读取MICAPS数据的get_latest_initTime的函数，获取CMISS中最新时刻模式七宝数据。

你好，用get_fy_awx()读出来的数据，经纬度范围与MICAPS4显示的经纬度不一致，是什么问题？

• lat (lat) float64 55.0 54.95 54.9 54.85 54.8 ... -7.35 -7.4 -7.45 -7.5
• lon (lon) float64 70.0 70.05 70.1 70.15 ... 157.3 157.4 157.4 157.5

ER5数据 怎么配置config文件

使用get_station_data读取站点资料时，如果时间超过当前时间，会出现报错。

data = get_station_data("SURFACE/TMP_MAX_24H_NATIONAL/", "20220118080000.000")
data

IndexError Traceback (most recent call last)
Untitled-2 in
----> 1 data = get_station_data("SURFACE/PLOT_NATIONAL/", "20220118080000.000")
2 data

D:\Anaconda3\envs\metdig\lib\site-packages\nmc_met_io\retrieve_micaps_server.py in get_station_data(directory, filename, suffix, dropna, cache, cache_clear)
674
675 # read the number of stations
--> 676 station_number = np.frombuffer(
677 byteArray[ind:(ind+4)], dtype='i4')[0]
678 ind += 4

IndexError: index 0 is out of bounds for axis 0 with size 0

尝试寻找原因时发现，可能是当时间超过当前时间时，服务器返回代码仍为200，导致解码时出错

service.getData("SURFACE/TMP_MAX_24H_NATIONAL/", "20220120080000.000")
(200, b'\x08\x01\x12\x13NotFoundException()')

我看源代码是从MICAPS4后端的GDServer下载的数据，不是直连Cassandra数据库的，请问下有直接连接Cassandra数据库的版本的么？

调用cimiss_get_obs_latest_time(data_code="SURF_CHN_MUL_HOR")报错：
Exception has occurred: KeyError
'Datetime'

读出来的数值似乎需要转换一下，用该程序读出来的数据在云顶较高的区域数值也较高，参照MICAPS4读出来的数据云顶越高区域数值越低。
并且，datarray中的time似乎有误，并不能精确到分钟，只到了小时，并且，时间为UTC。对应的Cassandra数据文件名为北京时间，且精确到了分钟。
谢谢！

此处Bug，请Fix

nmc_met_io.retrieve_micaps_server.get_radar_mosaic读取雷达数据时间数值有误：

filename_CREF='ACHN.CREF000.20201118.080000.LATLON'
data_dir='RADARMOSAIC/CREF/'
CREF = get_radar_mosaic(data_dir[0], filename=filename_CREF)
print(CREF['time'].values)

['2020-11-19T00:00:00.000000000']

请问如何使用“get_model_grids”命令时，只提取数据，不下载缓存数据？谢谢

===data_daily['date'] = pd.to_datetime(data_daily['date']).dt.tz_convert('Asia/Shanghai')
.dt.tz_convert('Asia/Taiwan') should be dt.tz_convert('Asia/Shanghai')

while i was reading a m4 file with the function, there was an error report：
File "S:\ProgramData\Anaconda3\lib\site-packages\nmc_met_io\read_micaps.py", line 148, in read_micaps_4
txt = f.read().decode('GBK').replace('\n', ' ').split()
AttributeError: 'str' object has no attribute 'decode'

I tried for another file but there was still the error.
The file I read was "r"Z:\data\newecmwf_grib\rain24\rain02\19032820.024"

File "H:/task/develop/python/maintest/atest.py", line 144, in
data = read_micaps_3('Z:/data/surface/jiany_rr/r20/19032708.000')
File "S:\ProgramData\Anaconda3\lib\site-packages\nmc_met_io\read_micaps.py", line 55, in read_micaps_3
txt = f.read().decode('GBK').replace('\n', ' ').split()
AttributeError: 'str' object has no attribute 'decode'

为了读取micaps diamond 1 格式的数据，参考nmc_met_io-master/examples/retrieve_micaps_model.ipynb的相关语句，我使用如下语句读取，但是在遇到错误，不能读取星号数据，建议改进：

>>> from nmc_met_io.read_micaps import read_micaps_1
>>> data = read_micaps_1("/mnt/d/datafile/surface/plot/20020402.000")
Traceback (most recent call last):
  File "pandas/_libs/lib.pyx", line 1926, in pandas._libs.lib.maybe_convert_numeric
ValueError: Unable to parse string "****"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/nmc_met_io-0.1.0-py3.6.egg/nmc_met_io/read_micaps.py", line 95, in read_micaps_1
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/tools/numeric.py", line 150, in to_numeric
    values, set(), coerce_numeric=coerce_numeric
  File "pandas/_libs/lib.pyx", line 1963, in pandas._libs.lib.maybe_convert_numeric
ValueError: Unable to parse string "****" at position 1021

读取symbols数据时经常有预报员姓名夹在数据中，导致不能正常读取。

retrieve_cmadaas中暂无类似UPAR_GLB_MUL_FTM的高空观测数据获取方法，建议新增

我不想直接读取netcdf4然后转成nc存到本地，想直接下载原始的文件，比如EC数据，我想直接下载bz2文件或者grib文件，然后自己在本地进行解析，雷达数据 也是，直接下载bz2或者bin格式，而不是用接口获取dataset数据
请问如何实现这一操作？我看函数里的dataformat默认是json的

采用retrieve_cimiss_history模块中的get_hist_obs获取历史数据一直抛出超时错误，未能成功获取数据，查看数据请求方式与网页上获取数据方法类似，为构造url进行获取，未来是否会提供类似于cma.music.DataQueryClient中的请求方式

>>> valid_times = [6*i for i in range(13)]
>>> levels = [1000, 950, 925, 900, 850, 800, 700, 600, 500, 400, 300, 250, 200, 100]
>>> data = cmadaas_model_3D_grids("NAFP_FOR_FTM_HIGH_EC_ANEA", "2020021512", 24, 'TEM', levels, 100, units="C", scale_off=[1.0, -273.15], 
                                 levattrs={'long_name':'pressure_level', 'units':'hPa', '_CoordinateAxisType':'Pressure'})

参数24，似应改为valid_times，否则报错：int类型不能被迭代。

请问天气雷达数据的读取用什么方法

在读取第一类数据时，出现如下bug:
File D:\Software\Miniconda3\envs\data_get\Lib\site-packages\nmc_met_io\read_micaps.py:84, in read_micaps_1(fname, limit)
82 if (len(txt) % 24) == 0:
83 txt = np.array(txt)
---> 84 txt.shape = [number, 24]
85 else:
86 txt = np.array(txt)
ValueError: cannot reshape array of size 80184 into shape (3084,24)

原因是number=80184既是24也是26的公倍数，我的文件中有26列数据。因此应该将txt的shape变为（3084，26）。

将报错描述中的“if (len(txt) % 24) == 0: ”代码修改如下，即可正常运行
if (len(txt) % 24 == 0) & (number * 24 == len(txt)):

选择天擎气象站年/月/日等数据接口时，如果选择如：最小气压出现的日期等类似要素时，如果最小气压对应了多个日期，原始数据的形式就如：'05-08'，即'5号-8号'，这个时候程序就会报错，显示不能转化为数值型。

错误定位到retrieve_cmadaas.py的153、154行，可以在这两行前加一个try，当出现非数值型字符串时，自动跳过数值型转换

在 用 pip install nmc-met-io 库时，出现以下安装信息：

Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy>=1.17.0 in d:\anaconda3\lib\site-packages (from nmc-met-io) (1.21.5)
Requirement already satisfied: scipy>=1.4.0 in d:\anaconda3\lib\site-packages (from nmc-met-io) (1.7.3)
Requirement already satisfied: pandas>=1.0.0 in d:\anaconda3\lib\site-packages (from nmc-met-io) (1.3.4)
Requirement already satisfied: xarray>=0.16.0 in d:\anaconda3\lib\site-packages (from nmc-met-io) (0.20.2)
Collecting protobuf>=3.12.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/36/16/cfba8fcd817df923827233115df35dc048af12d0afa13df79b303865855a/protobuf-3.20.1-cp37-cp37m-win_amd64.whl (905 kB)
---------------------------------------- 905.1/905.1 KB 4.4 MB/s eta 0:00:00
Requirement already satisfied: urllib3>=1.25.9 in d:\anaconda3\lib\site-packages (from nmc-met-io) (1.26.9)
Requirement already satisfied: tqdm>=4.47.0 in d:\anaconda3\lib\site-packages (from nmc-met-io) (4.64.0)
Requirement already satisfied: python-dateutil>=2.8.1 in d:\anaconda3\lib\site-packages (from nmc-met-io) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in d:\anaconda3\lib\site-packages (from pandas>=1.0.0->nmc-met-io) (2022.1)
Requirement already satisfied: six>=1.5 in d:\anaconda3\lib\site-packages (from python-dateutil>=2.8.1->nmc-met-io) (1.16.0)
Requirement already satisfied: colorama in d:\anaconda3\lib\site-packages (from tqdm>=4.47.0->nmc-met-io) (0.4.4)
Requirement already satisfied: typing-extensions>=3.7 in d:\anaconda3\lib\site-packages (from xarray>=0.16.0->nmc-met-io) (4.2.0)
Requirement already satisfied: importlib-metadata in d:\anaconda3\lib\site-packages (from xarray>=0.16.0->nmc-met-io) (4.11.3)
Requirement already satisfied: zipp>=0.5 in d:\anaconda3\lib\site-packages (from importlib-metadata->xarray>=0.16.0->nmc-met-io) (3.8.0)
Building wheels for collected packages: nmc-met-io
Building wheel for nmc-met-io (setup.py) ... done
Created wheel for nmc-met-io: filename=nmc_met_io-0.1.9.2-py3-none-any.whl size=114006 sha256=34658b7b6e736bdc7f7aff81c33eb94bb3977be966ec3aa32ea027733a8defd1
Stored in directory: c:\users\tom\appdata\local\pip\cache\wheels\c6\c9\0f\58fcb38edef884bd986ac3fb853ac7d7627b5fc18547e5c5b0
Successfully built nmc-met-io
Installing collected packages: protobuf, nmc-met-io
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
gm 3.0.137 requires typing>=3.6.2, which is not installed.
Successfully installed nmc-met-io-0.1.9.2 protobuf-3.20.1

加粗斜线部分显示， 一个无关紧要的 库 gm 3.0.137 requires typing>=3.6.2, which is not installed. 掘金量化sdk库 试图安装。这个库完全和 它 搭不上交道， 感觉 nmc-met-io库安装被人做了手脚。

If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:

1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

这个库好像只有读，没有写啊

FY2G_AMV_IR1_OTG_20230624_2330.AWX怎么进行读取？

File "D:\lss_code\micaps\1.py", line 39, in
f = read_micaps_4(r'D:/lss_code/micaps/data27-19.2.28/ECMWF_HR/CAPE/19022708.000')

File "D:\Anaconda3\lib\site-packages\nmc_met_io\read_micaps.py", line 398, in read_micaps_4
data = (np.array(txt[22:])).astype(np.float)

ValueError: could not convert string to float: '1,124.25'

ec文件中有超过1000的值，其格式为*,*.，read_micaps_4无法正常读取

使用get_station_data从Cassandra读这个路径下的‘SURFACE/PLOT_GLOBAL_ALL_3H/’全球站点数据失败