Add functions for QC data about act

[AdamTheisen]

They are not bit-packed. I'm wondering if the base function we need is just a bit decoder. After thinking about other data sources, I don't know if we can make a one size fits all. We might need to do an arm_qc_utils.py and put arm specific in there. I do think that a standard bit-packed decoder would work well in qc_utils though

[kenkehoe]

OK let's chat at the meeting next week. I think we can use flag_values for non-bitpacked and it will work. It would just be an extension of CF and the functions could auto detect type based on attributes set if we follow CF convention. If you have an example file can you load it here?

[AdamTheisen]

I don't have any examples offhand. I do remember that neon had a separate csv file that had qc information in and I don't know if ameriflux had any. I think they tend to interpolate data when they know it's bad. Not sure though

[kenkehoe]

I'm currently working on migrating some of the DQ QC functions over to ACT. Right now I plan to keep the number of functions to a small number as most users will only need a few. But my question to the group is what will a user typically use them to do and what sort of functions will help them the most. I've got the setting and removing of bits for tests and the display of test numbers from a bit packed number implemented.

I think the most likely use of QC from a user is to mask data. But the bigger question is how will we create the functions to get the data in the format they will use/want. Since there will be a flag_assessments attribute set to standard "Bad", "Indeterminate" we can use that for filtering as well as individual tests. Eventually we'll add DQR support as well. I see Suspect = Indeterminate so we can filter the same. We can return a mask from a function call to either set the values to NaN or return an array with requested data removed (changing the shape of the array). I see both having their place. To do any of this I'm thinking of a generic reader for getting data from the object with the reader doing the work of querying the QC data and DQRs to return an array with filtered data set to NaN. Initially I was thinking of Masked Arrays but I don't think that will work for plotting since we are reading from the object directly with xarray calls. I'm thinking the preferred method is to manipulate the object to get the data the way the user wants and then do plotting, research, ... They can make a copy of the object if they don't want to make changes to the original data.

Any thoughts?

[AdamTheisen]

Hi @kenkehoe

I am going to ping @rcjackson @zssherman and @scollis as well.

As for what would be useful to the end users, I think a mask or way to get back the indices of where specific bits are tripped is useful. For plotting purposes, PyART has a gate filter keyword for many of the variables, but is there a more standard way to do this based on masked data as Ken is wanting to do?

[rcjackson]

We'll probably have to make something similar to the Py-ART GateFilter. Perhaps a separate object that supports masking observations with given variables and stores the mask for later use. In the end, one would be able to use the mask to apply NaNs in regions where the variable is masked in the object.

…

________________________________ From: Adam Theisen <[email protected]> Sent: Monday, June 24, 2019 7:48:54 AM To: ANL-DIGR/ACT Cc: Jackson, Robert; Mention Subject: Re: [ANL-DIGR/ACT] Add functions for QC data (#75) Hi @kenkehoe<https://github.com/kenkehoe> I am going to ping @rcjackson<https://github.com/rcjackson> @zssherman<https://github.com/zssherman> and @scollis<https://github.com/scollis> as well. As for what would be useful to the end users, I think a mask or way to get back the indices of where specific bits are tripped is useful. For plotting purposes, PyART has a gate filter keyword for many of the variables, but is there a more standard way to do this based on masked data as Ken is wanting to do. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#75>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFIQA5BM2B4WWR2HPW3LHODP4C7DNANCNFSM4HVZPS6A>.

[scollis]

Exactly. Gatefilters work great in pyart and the concept can be extended to ACT

…

-sent from a mobile device-

On Jun 24, 2019, at 9:44 AM, Bobby Jackson ***@***.***> wrote: We'll probably have to make something similar to the Py-ART GateFilter. Perhaps a separate object that supports masking observations with given variables and stores the mask for later use. In the end, one would be able to use the mask to apply NaNs in regions where the variable is masked in the object. ________________________________ From: Adam Theisen ***@***.***> Sent: Monday, June 24, 2019 7:48:54 AM To: ANL-DIGR/ACT Cc: Jackson, Robert; Mention Subject: Re: [ANL-DIGR/ACT] Add functions for QC data (#75) Hi @kenkehoe<https://github.com/kenkehoe> I am going to ping @rcjackson<https://github.com/rcjackson> @zssherman<https://github.com/zssherman> and @scollis<https://github.com/scollis> as well. As for what would be useful to the end users, I think a mask or way to get back the indices of where specific bits are tripped is useful. For plotting purposes, PyART has a gate filter keyword for many of the variables, but is there a more standard way to do this based on masked data as Ken is wanting to do. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#75>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFIQA5BM2B4WWR2HPW3LHODP4C7DNANCNFSM4HVZPS6A>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kenkehoe]

OK can you give me some more information on Gatefilters?

[rcjackson]

They are a class with methods that can specify regions to be masked. For example, if I wanted to create a mask excluding all values with temperature < 3, I would do something like: my_filter = act.corrections.gatefilter(my_ds, dims='time') # Would need to specify which dimension(s) we are filtering on my_filter.exclude_below('temperature', 3) This would modify the mask inside my_filter to have all time periods with temperature < 3 to be masked. We probably won't call this a gatefilter, but rather just a datafilter since gatefilter is specific to radar. Any other thoughts on the name? The documentation on how they are implemented in Py-ART is available at: http://arm-doe.github.io/pyart-docs-travis/user_reference/generated/pyart.filters.GateFilter.html pyart.filters.GateFilter — Py-ART Documentation<http://arm-doe.github.io/pyart-docs-travis/user_reference/generated/pyart.filters.GateFilter.html> arm-doe.github.io pyart.filters.GateFilter¶ class pyart.filters.GateFilter (radar, exclude_based=True) [source] ¶. A class for building a boolean arrays for filtering gates based on a set of condition typically based on the values in the radar fields.

…

________________________________ From: Ken Kehoe <[email protected]> Sent: Tuesday, June 25, 2019 6:00:37 PM To: ANL-DIGR/ACT Cc: Jackson, Robert; Mention Subject: Re: [ANL-DIGR/ACT] Add functions for QC data (#75) OK can you give me some more information on Gatefilters? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#75>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFIQA5CCS6YEL4FYXSNHBBDP4KPRLANCNFSM4HVZPS6A>.

[kenkehoe]

OK thanks for the update. This seems very similar to numpy masked array functions already available.

I like the method to do the masking but I think we should use QC variables to store the result. That way multiple masks can be applied and retrieved independently by storing the result of a test ( or mask) in a bit-packed array format. Otherwise the result will be impossible to reverse or untangle if the user decided to do something else. Also if we save the result internally as a CF QC variable it can be written to a file and used later or shared with others.

I would prefer we have a function that uses the flage_meanings or flag_assessments under a QC variable to determine what should be added to the numpy mask and returns the result. I see QC as more complicated and more user variable than the corollary gatefilter ideas.

[kenkehoe]

I'm moving away from a qc_utility.py module and used a method approach instead. It should be able to handle both bit-packed and integer QC styles as long as they follow CF. If they don't we can add some additional methods to clean method to update the other institutions to our internal standard that closely follows CF.

[kenkehoe]

With the recent merge of #81 into the repo I think this issue can be closed. We will continue to add more QC tests.