IERS EOP Rates, Formats

Orekit development
1

Hi,

I saw an email that the IERS will add rates to the x, y polar motion parameters this November. Current code in ITRFProvider assumes the rate of the polar motion correction is zero. Is consideration of PM rates something that Orekit should add? @luc?

The IERS email also mentioned that the formats of the XML files will change and that the IERS will add a new JSON format. Should Orekit support the new formats?

IERS message: https://datacenter.iers.org/data/2/message_490.txt

Best Regards,
Evan.

2

Hi @evan.ward , thanks for the info, I missed this.

Yes, I think Orekit should support everything IERS produces, so we should check the new data format, the EOP history container and the ITRF provider.
Supporting the json format would be a plus, but I am really reluctant to add any dependency to Orekit, so I wonder if we should parse it the hard way, as we do for example for the XML format.

3

I am not sure however how to handle having both rates and the existing interpolation scheme (see EOPHistory.java line 462).

4

Agreed :slight_smile: !

@luc you’re only talking about the API right?

I wonder if we should extend EOPEntry or add an hasPoleCorrectionRates() method in EOPEntry and EOPHistory.

Since Hermite interpolation supports derivatives, maybe replacing:
final double[] interpolated = interpolate(date, entry -> entry.getX(), entry -> entry.getY());
With:

final double[] interpolated;
if (hasPoleCorrectionRates()) {
    interpolated = interpolate(date, entry -> entry.getX(), entry -> entry.getY(), entry -> entry.getXdot(), entry -> entry.getYdot());
} else {
    interpolated = interpolate(date, entry -> entry.getX(), entry -> entry.getY());
}

Would work?
(With the 1st interpolate using both values and rates in the Hermite interpolator)

Was that your interrogation @luc or am I off-topic?

5

This is what I had in mind. Your proposal seems fine to me.
We should take care, when we load EOP from a set of different files, to set the rate flag to false as soon as one of the files misses rates, so the flag remains a global one.

6

Yes I was wondering how we should handle “mixed” data

7

I have looked at this, it is not really straightforward.

I have pushed an iers-message-490 branch, but it only adds rates to EOPEntry and to interpolator, but no parser fills these fields (except PredictedEOPHistory but it is not a parser).

I gave a look at the XML files but the only new XML file published here is an EOP C04 and we do not support XML format for EOP C04. I am not sure how to understand file readme.txt. Will really only EOP C04 be affected in XML and not finals.daily.xml that we support (thanks to RapidDataAndPredictionXMLLoader)?

I also looked at bulletin A, but it seems not really straightforward either. Maybe we could rely on the first line that give columns titles, assuming the new files will have titles “x_rate” and “y_rate” whereas old files won’t have these. So counting the titles, we could parse both old and new files.

What do you think? Should we merge the iers-message-490 branch within develop because it changes the constructor of EOPEntry (hence must be done in a major release), despite currently no parser fills this? Should we also add the missing csv parsers, counting columns? Should we wait until IERS makes sure which files are affected and which are not?

8

+1 for merging. Because we will not have additional information before November 6, it is the best decision.

It can be done during a minor release, right?

I appreciate the “release early, release often” mantra. So I’m in favor of releasing as it. If the IERS change breaks the APIs, we will probably have to patch many Orekit versions. In other words patching N or N + 1 versions will represent the same amount of work. That’s just my opinion. I think that for the question any opinion is welcome.

9

Yes, only the public API for EOPEntry needs to be changed in a major release.

Yes, but patching older versions that miss the rates in EOPEntry will be slightly different from patching 12.X versions. For older versions, we will only parse the columns and ignore them, taking care not to stall on new files and just accept them, whereas for new versions we would need to use the derivatives.

I also wonder whereas we should make the number of interpolation points customizable. Currently we always use 4 points. Some coworker asked recently to IERS about the interpolation scheme that is recommended and the answer was that it is a frequently asked question that unfortunately has no answers: IERS does not recommend any method. We have seen people using Lagrange 4 (Orekit), we have seen people using Lagrange 8 (some internal proprams at work). With derivatives, should we go to Hermite 4, or Hermite 2 in order to avoid too large degrees? Should we have a different number of points for dx,dx that do not have rates and UT1/LOD (since LOD is a scaled derivative of UT1) or xp and yp that do have derivatives? How do we specify the interpolation scheme to use if we make it customizable.

I have the bad feeling that diving into this uncovers problems we did not anticipate.

10

Hello,

I agree with Bryan on the “release early, release often” philosophy.

If we need a patch for all Orekit versions because they do not work with the new files then we can apply the same patch for all.
And, in a second time, treat the pole derivatives in version 12.1.
Would you be okay with that @luc?

I do think it’s a good idea (for a 12.1 or further version :wink: ) to allow users to choose the number of interpolation points and (eventually) the interpolation method.
Since it’s an advanced feature we could put the default values in LazyLoadedEop like it’s done for the eopContinuityThreshold?
So that a new user won’t have to guess what value is needed for those.

11

This changes EOPEntryAPI, so it can’t be postponed to a minor version. This is the reason why I started with that in the branch.

Maybe. I’ll have a look at this

12

I did not recall it at first, but we already started working on this for the EOP C04 files.
See issue 1031.
We however only allowed parsing the file without error at that time, we ignored the rates.
So I will reopen the issue to complete this work.

13

Hi!

I wrote the following email to IERS

Dear IERS,

I would like to ask you some questions according to Message No. 490.

1. Will the .txt files be impacted by the addition of x rate, y rate, sigma x rate, and sigma y rate?

2. Will these parameters be optional or mandatory for parsing the EOPs files?

3. If we released a software N years ago, do we have to patch it after November 6 in order to allow it to parse the new EOPs files?

Thank you very much.

Regards,
Bryan Cazabonne

Please fin below the answer

Dear Mr. Cazabonne,

The format change we announced in the IERS Message No. 490 only affects the .csv and .xml file formats.
This is an adjustment of both file formats to the format change of the original .txt file "EOP_20_C04_one_file_1962-now.txt" / "eopc04_20.1962-now".
As part of the implementation of IERS 2020 C04, pole rates have been added to this format.
This was announced in February 2023 in IERS Message No. 471 (see https://datacenter.iers.org/products/publications/messages/message_471.txt).

For the xml files the new parameters are optional. If your software parses the xml files, there will be no need to patch it.
Concerning the csv files, headers and fields for the new parameters will be added for all EOP products (see example files https://datacenter.iers.org/products/eop/new_format_examples/csv/eopc04_20.1962-now.csv, https://datacenter.iers.org/products/eop/new_format_examples/csv/gpsrapid.daily.csv ...). If your software parses the csv files, the need for a patch depends on how the fields are addressed in the software. If the fields are addressed by the names of the headers, no change will be necessary. If the fields are addressed differently (for example by counting certain characters) your software will have to be adapted.

If you process the .txt file "EOP_20_C04_one_file_1962-now.txt" / "eopc04_20.1962-now" or any of the .csv files for EOP Products, you should check if your routines still work correctly with the new format and adapt them if necessary.
For this purpose we recommend the use of the example files at https://datacenter.iers.org/products/eop/new_format_examples/.

If you are not able to patch your software by November 6, you may use the csv files in the old format for a few more months, which we will provide in separate folders at https://datacenter.iers.org/products/eop/.

I hope we could help you. If you have any further questions, please do not hesitate to contact us.

Best regards,
Sonja Geist
IERS Central Bureau

To summarize, I don’t think there is an impact to Orekit. The release of 12.0 version can starts :smiley:

2 Likes
14

This is factually false. Optional parameters trigger parsing errors if they are not expected and explicitly ignored. This is a change in the parsing software.

I disagree

15

Yes, I agree. But does Orekit support the IERS .xml files?

Could you detail your disagreement?

16

Yes, at least the rapid data : RapidDataAndPredictionXMLLoader

EOP data is mandatory for all operational missions. Not being able to load fresh data will have a devastating effect on many operational systems. This change, with only a few weeks warning, is a big threat. If we decide to ship version 12.0 without full support for it, we will force our users to do big updates to their systems twice in a short time : once for 12.0, once for handling the new IERS files. This will cost time, money, and may put missions in jeopardy (yes, I know I am over-reacting).

My point is: it is now the time to handle these files.

In fact, I already have written a csv parser and by a lucky fluke, it appears several IERS csv files can be handled by one parser, as they all use similar keys in the first line, so we can easily identify the columns for the keys we want. I did not finish the tests and I was planning to do something else this week-end (some metal slab to scrap to a very high precision flat surface in my home workshop), but as IERS change seems too important to me, I’ve put aside my plans and will do this. So I will implement what is needed. This way, our users will be able to make just one big change from 11.X to 12.0 that will support the new IERS kludge.

Going to 12.0 will already be difficult for most Orekit users, it is a big version with lot of changes, so I really don’t want them to do this kind of chore twice in a few months. I really don’t know how users who can’t switch to 12.X will handle IERS changes. Perhaps they will be able to patch their software to use some 11.3.4, 10.3.2 or 9.3.2 we may provide with compatible changes to just ignore the rates, I don’t know. But as far as 12.0 is concerned, it must be complete in my humble opinion.

17

Another point: the example files for the new format only have new header keys for x_rate and y_rate, but the corresponding fields are empty…
So I guess we will only have real data after November 6th. So much for example files.

18

OK, I have added support for all versions of CSV and XML files (bulletin A, bulletin B, EOP C04), with or without rates, and updated the default configuration so people can throw in such files in their orekit-data.
I hope this will help users use the new files when IERS publishes them.

I now remove my concerns about 12.0 publication.

19

Thinking again, just wait a few hours more before starting 12.0.
I am going to add two features related to EOP derivatives, hopefully it will not take long.

20

Great @luc !!! :clap: :clap:

Do you think your commits can apply as a patch for former Orekit versions that may become unusable with the new formats?
(we may want to wait for the publication of the new files before applying the patches though, just to avoid two patches per version if IERS files happen to differ from what’s expected)