Is an ARTS without master grids possible? about arts

[riclarsson]

As a small comment. If you want an atmosphere that can be extrapolated or interpolated only in some range of altitudes, I really think that that range should be part of the atmosphere itself. That is the AtmField of #552 should just contain a pair of minimum and maximum altitudes within which it is valid to compute an AtmPoint.

I think introducing a new Wsv just confuses the matter; the atmospheric field knows itself where it is valid and can thus check its data for validity within that range

[erikssonpatrick]

My suggestion implies that the atmosphere is defined to infinite altitude. So "z_space" could be seen as a variable flagging where the path calculations should start/stop. But no problem with seeing this as a part of AtmField.
However, in the downward direction, the lower boundary of the atmosphere is defined by the surface altitude, and we don't need any additional variables. Today there can be a gap between the surface and atmosphere (for e.g. limb sounding) but with my suggestion that will never be the case. (By the way, I am planning to replace z_surface with a GField2 called surface_elevation, a change in line with what we discuss here).

[erikssonpatrick]

Alll: No comments on the overall level? This has to be sorted out before Richard and I can continue on your work on AtmField and Ppath, respectively.

[m-brath]

I think this is so far ok and I like Richard's idea to add the altitude limits to the atmfield. But I have some issues with the extrapolating.
I think we should distinguish between constant atmospheric properties, which are defined as a scalar, and non-constant atmospheric properties, which are define as vector, matrix... For the scalar, I think it is totally fine to "extrapolate" them, but I found it difficult to extrapolate non-constant atmospheric properties in altitude, because this can have a big influence on the results, which for a user would not be so clear. I think setting the quantities to zero outside the given data would be safer.
For latitude/longitude direction extrapolating is OK for me. Maybe, we should not extrapolate, because of the discontinuities at the backside, but we could interpolate since we are on a closed surface. By doing this we would automatically have some kind of periodic boundaries and we would not have any discontinuities.
Considering the particle properties, I still need to think about.

[olemke]

Freddy, Richard and I discussed this in our coffee/ARTS break today and one concern that manifested pretty quickly was that a single method to handle the extrapolation for all fields is likely not sufficient. Just for VMR as an example, it would be useful to not only have a nearest neighbor extrapolation, but also be able to set the field to zero outside the defined range. For other quantities a linear extrapolation might be more useful. Additionally, you might want to treat extrapolation towards the surface different from extrapolation towards space. Therefore, we came up with a solution where the atmospheric fields have two additional extrapolation properties (one for upwards, one for downwards) which could have the following values:

NONE: Trying to extrapolate this field throws an error
ZERO: Data outside the field is set to zero
NEAREST: Use the nearest value
BAROMETRIC: Extrapolation for pressure according to the barometric formula
EXTRAPOLATE: Linear extrapolation, longitudinal it will be cyclic interpolation

For scalar values NEAREST would be the default.

Proper cyclic extrapolation (over the pole) for latitudes would need to be implemented later.

BAROMETRIC would make it possible to only provide one single pressure point or layer.

NONE could be used for cases where the user explicitly wants to make sure that the calculation is only using the given data and avoid unexpected use of extrapolated value.

[stefanbuehler]

Sounds nice. :-)

…

Am 14.11.2022 um 12:37 schrieb Oliver Lemke ***@***.***>: Freddy, Richard and I discussed this in our coffee/ARTS break today and one concern that manifested pretty quickly was that a single method to handle the extrapolation for all fields is likely not sufficient. Just for VMR as an example, it would be useful to not only have a nearest neighbor extrapolation, but also be able to set the field to zero outside the defined range. For other quantities a linear extrapolation might be more useful. Additionally, you might want to treat extrapolation towards the surface different from extrapolation towards space. Therefore, we came up with a solution where the atmospheric fields have two additional extrapolation properties (one for upwards, one for downwards) which could have the following values: NONE: Trying to extrapolate this field throws an error ZERO: Data outside the field is set to zero NEAREST: Use the nearest value BAROMETRIC: Extrapolation for pressure according to the barometric formula EXTRAPOLATE: Linear extrapolation, longitudinal it will be cyclic interpolation For scalar values NEAREST would be the default. Proper cyclic extrapolation (over the pole) for latitudes would need to be implemented later. BAROMETRIC would make it possible to only provide one single pressure point or layer. NONE could be used for cases where the user explicitly wants to make sure that the calculation is only using the given data and avoid unexpected use of extrapolated value. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were assigned.

[erikssonpatrick]

Thanks for the comments. Yes, different kinds of extrapolation could be nice. But in my mind, we have not yet clarified if we even want extrapolation at all (except allowing scalar values). I really think this has to be settled in a top-to-bottom approach. How do we want ARTS to work on the grand scale?

Two overlaying questions are:
1: Do we want that the atmosphere always is defined globally (up to some specified altitude)? That is, all lat and lons are always covered. Or do we want to have clear lat and lon limits for the defined atmosphere, as we have now?
2. Shall the cloudbox be a sub-domain (as now) or do we abandon that concept?

It seems that the general feeling for 2 is that we want to keep the cloudbox. But there should be a proper decision about it.

For 1, I am much less sure. Always having a global atmosphere would be nice, but opens up for mistakes. You can then easily end up doing a simulation outside the area where you have a proper definition of the atmosphere. Maybe we can devise checks for it, but we should then look at that before going in this direction.

The outcome of 1 will to some extent governs how easy it is to allow different types of extrapolation. But what extrapolation we want and can handle, is for me a later question. In short, easier to discuss when we know how far we will be extrapolating.

Maybe there are further top-level issues to be clarified before we go into details ...

(To be clear, I can not join the meeting Fri (due to teaching))

[riclarsson]

I have never used the cloud stuff so I will ignore that part.

We should internally treat the atmospheric field as globally known. The user decides if this is bad through their extrapolation method. The extrapolation question is not for later, it decides the answer to the question.

Since the user chooses their extrapolation methods for all six directions for each atmospheric quantity, there's no reason to consider bad latitude or longitude coverage beyond what the user finds acceptable. Their choices are simply respected fully* even if it means a bad atmosphere**.

*. So only two possible errors can be thrown: the user chooses to not allow extrapolation for some quantity, or some derived point value is not accepted (only for T<=0 K; P<0 Pa as far as I am concerned). We do not need any user-level extra checks for this.

**. We will need user-level derived data debug mechanisms, such as easy ways to produce plots of the atmospheric points along a propagation path in pyarts for this to work. It is extremely important to provide after-the-fact user-friendly diagnostics/plotting tools for the multiple issues that might arise here. We lack a lot of these today.

[erikssonpatrick]

Anything said about this at the Zoom meeting?

[erikssonpatrick]

OK, I got an answer today by Simon.
I still would prefer to discuss this from top-to-bottom. So I just make some remarks:

• That this has implications for the Jacobian calculations. This including extrapolation.
• That there must be a way to do IBA calculations, i.e. easily create a 1D atmosphere from the data sampled along the ppath and use the 1D atmosphere in a separate RT calculation.
• What about ybatch-type calculations? How to do that? (That said, no problem for me to remove this. I have never seen the point. And now this seems as something for Python)
• For completeness, a repeat. Waht to do with the cloud box?
• Something that I have missed?
  I am favor of introducing AtmField and don't say what is suggested will not work (except maybe some extrapolation option), just raising points to be considered.

[riclarsson]

@erikssonpatrick

1. There are no problems with the Jacobian calculations. For the atmospheric fields, these are always computed at the atmospheric grid point and regardless of extrapolation methods, you will have an atmospheric grid point. I brought up the issue of mapping between OEM and AtmField some meetings ago and you also said this would work OK so long as I provide a method to compute the Del operator on the AtmField. This I will implement, it will return three AtmPoint partial derivatives when used as gradient.
2. We have 0D-3D atmospheres going forward. You can extrapolate a 1D list of atmospheric points from this.
3. @olemke had some questions here. I do not, I also am not using it.
4. The cloud box should be similar to today is what I understood. What is the problem of giving an altitude range instead of pressure range?

[olemke]

• What about ybatch-type calculations? How to do that? (That said, no problem for me to remove this. I have never seen the point. And now this seems as something for Python)

Can you elaborate why you see a problem with ybatch calculations? I don't see why they would not work with the proposed changes. IMHO, it would be beneficial to keep the ybatch mechanism because it is very efficient to use if you have to deal with hundreds or thousand of atmospheric profiles. This is one of the most efficient parallelisations we have in ARTS since it scales so well even with a high number of cores. @m-brath for example just used this on the DKRZ supercomputer with over a hundred cores. While it of course could be done on the Python side, the overhead would be significantly higher when using Python's multiprocessing compared to utilizing OpenMP inside ARTS.

[erikssonpatrick]

It simply feels awkward to have two ways of specifying atmospheres, but OK-ish now as the two ways have a one-to-one mapping. (One way to notice that having two ways is bad is that the documentation of atm_fields_compact still refers to the old PSD system.) But atm_fields_compact and AtmField will work in different ways and atm_fields_compact will be very restricted compared to AtmField (i.e. no extrapolation). And I think that will cause confusion, and frustration after a while.
If running batch calculations from Python really is considerably less efficient than having a batch feature in ARTS, why not consider fixing this more nicely if we anyhow turn things upside down? For me, this sounds as if we should have a fourth dimension after all! I was against a dim 4, to not make this change too big. But this has anyhow grown and now I feel that adding a dimension is not a big addition.
We can here kill two birds with one stone. You can take the fourth dimension as time literally. Or you place your batch atmospheres at times 0, 1, 2, 3 ... and you can still use ybatch_index to extract one separate atmosphere at a time (just need a conversion from Index to time). And you can e.g. set O2 VMR for all batch cases with a single value.

[stefanbuehler]

I like the idea of a fourth dimension for batches. /Stefan

…

Am 23.11.2022 um 19:55 schrieb Patrick Eriksson ***@***.***>: It simply feels awkward to have two ways of specifying atmospheres, but OK-ish now as the two ways have a one-to-one mapping. (One way to notice that having two ways is bad is that the documentation of atm_fields_compact still refers to the old PSD system.) But atm_fields_compact and AtmField will work in different ways and atm_fields_compact will be very restricted compared to AtmField (i.e. no extrapolation). And I think that will cause confusion, and frustration after a while. If running batch calculations from Python really is considerably less efficient than having a batch feature in ARTS, why not consider fixing this more nicely if we anyhow turn things upside down? For me, this sounds as if we should have a fourth dimension after all! I was against a dim 4, to not make this change too big. But this has anyhow grown and now I feel that adding a dimension is not a big addition. We can here kill two birds with one stone. You can take the fourth dimension as time literally. Or you place your batch atmospheres at times 0, 1, 2, 3 ... and you can still use ybatch_index to extract one separate atmosphere at a time (just need a conversion from Index to time). And you can e.g. set O2 VMR for all batch cases with a single value. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were assigned.Message ID: ***@***.***>

[riclarsson]

This is in the ARTS3 dev branch