Should the SP3Parser be more SMART?

Orekit development
1

Hi, all,

I am using sp3 file provided by ILRS ACs, ASI,BKG,DGFI,ESA,GFZ,JCET,NSGF, and ilrsa, ilrsb (see details, ftp://edc.dgfi.tum.de/pub/slr/products/orbits/).

The are somewhat flaws according to the sp3 file format.
Is there need to enhance the SP3Parser to be more SMART?

(1) ilrsa and ilrsb: (a) “%/*” is used as an HEADER_COMMENTS, instead of “/*”; (b) no ‘EOF’.

%i    0    0    0    0      0      0      0      0         0
%/* ilrsa.orb.lageos1.160709.v35.sp3 Reference TRF: SLRF2008
%/* Input orbits: ASI v35, DGFI v35, ESA v35, GFZ v35,
%/* JCET v35, NSGF v35,    
%/* Combination details in README_CC.ilrsa                  
*  2016  7  3  0  0  0.00000000
PL51  -3935.038756   5359.461638 -10292.322587 999999.999999
VL51  10627.335527  54038.852939  24357.640751 999999.999999

(2) asi and dgfi: (a) the minute may be ‘60’, the hour may be ‘24’ (which should be 0).

*  2016  7  6 16 60  0.00000000
PL51  11948.228978   2986.113872   -538.901114 999999.999999
VL51   4605.419303 -27972.588048 -53316.820671 999999.999999
*  2016  7  6 17  2  0.00000000
PL51  11982.652569   2645.786926  -1177.549463 999999.999999
VL51   1128.248622 -28724.293303 -53097.358387 999999.999999

*  2016  7  6 23 58  0.00000000
PL51   3215.382310  -7958.586164   8812.395707
VL51 -18058.659942 -45834.335707 -34496.540437
*  2016  7  7 24  0  0.00000000
PL51   2989.229334  -8494.421415   8385.068555
VL51 -19617.027447 -43444.824985 -36706.159070
*  2016  7  7  0  2  0.00000000
PL51   2744.983592  -9000.639164   7931.904779
VL51 -21072.925764 -40899.633288 -38801.567078

(3) gfz: (a) the second in DATA_EPOCH is just 4 digits, which should be 8 digits (F11.8).

*  2016  7  3  0  0  0.0000
PL51  -3935.038808   5359.461691 -10292.322543
VL51  10627.335290  54038.853128  24357.641280

(4) jcet: (a) no clock record but left some blank (space) in DATA_POSITION and DATA_VELOCITY.

*  2016  7  3  0  0   0.00000000                            
PL51  -3935.038648   5359.461640 -10292.322619              
VL51  10627.335826  54038.852638  24357.640559

So, the suggested enhancements are followings:
(1) DATA_EPOCH: use the rest as seconds.

final double second = Double.parseDouble(line.substring(20, 31).trim());
==>
final double second = Double.parseDouble(line.substring(20).trim());

(2) HEADER_COMMENTS: both “%/*” and “/*”

HEADER_COMMENTS("^/\\*.*")
==>
HEADER_COMMENTS("^[%]/\\*.*")

(3) clock record in DATA_POSITION and DATA_VELOCITY:

// clock (microsec)
pi.latestClock = line.length() <= 46 ?
                                        DEFAULT_CLOCK_VALUE :
                                        Double.parseDouble(line.substring(46, 60).trim()) * 1e-6;
==>
// clock (microsec)
pi.latestClock = line.trim().length() <= 46 ?
                                             DEFAULT_CLOCK_VALUE :
                                             Double.parseDouble(line.substring(46, 60).trim()) * 1e-6;

There are much more complicate about handling the case of minute=60 and hour=24, and no EOF.

Any idea?

Best regards,
LiRW

2

Hi @lirw1984

Thank you very much for reporting those issues. We always considered that Orekit must be very strict when writing a file but can accept some laxity when parsing a file. Therefore, all the issues you highlighted must be fixed. Your proposals look good. For minute=60 and hour=24, I think we can read the value and convert it to the one accepted by Orekit. Something like:

pi.latestEpoch = new AbsoluteDate(year, month, day,
                                  hour == 24 ? 0 : hour, // sometimes equal to 24 instead of 0
                                  minute == 60 ? 0 : minute, // sometimes equal to 60 instead of 0
                                  second, pi.timeScale);

Are you interesting in contributing all these fixed?
If yes:

  1. Thank you very much!
  2. Could you open an issue on the GitLab issue tracker: Issues · Orekit / Orekit · GitLab
  3. Could you follow the steps in the contributing guide to contribute the fix (fork, etc.)? Orekit – Contributing to Orekit. It will help us to include this contribution in Orekit easily.
  4. Please take care that for contributions, we need an Individual Contributor License Agreement (ICLA). You can find more about this in the Orekit governance document. Could you complete the ICLA and send it to me in private message?

Thank you and best regards,
Bryan

3

Do we need to take carry into account when rolling back 24h to 0h (going to next day) and 60 minutes to 0 minutes (going to next hour)?

4

Looking at the data, yes we should take carry.
Beware however that this must be done starting from least significant fields and propagated forward, for example 2015 12 31 23 60 0.00000000 should be 2016 01 01 01 00 0.00000000.

5

Looking even more closely, carry is not even consistent!

One example has 2016 7 6 16 60 0.00000000 followed by 2016 7 6 17 2 0.00000000, so carry applies and the next example has 2016 7 6 23 58 0.00000000 followed by 2016 7 7 24 0 0.00000000 so carry does not apply.

I wonder if in this case we should parse two dates, one with and one without carry, and then select one according to the previous date.

6

Looking at the provided examples, we need to take care for minutes (i.e., 16 60 means 17 00). But not for hours ( 7 24 means 7 00).

7

Thank you all for reply.

I’ve done more test.
For asi, 16 60 means 17 00. For dgfi, 24 means 00. And it seems that the bugs are only in the hist files. So maybe we just do this wired fix.

‘no EOF’ is consistent for ilrsa and ilrsb.

8

This would be a ad hoc solution, we should do something more definitive. If both asi and dgfi experience similar bugs, we could expect other providers to also have that. The pattern is simple and the dates can be predicted.

I did not try it, but could you try the following patch?
sp3.patch (6.7 KB)

9

Hi @bcazabonne and @luc ,

About the missing “EOF”, is that OK if we just check the number of epochs?

That is, if “pi.nbEpochs == pi.file.getNumberOfEpochs()” means the file is OK, regardless of EOF exists or not.

            // initialize internal data structures
            final ParseInfo pi = new ParseInfo();

            int lineNumber = 0;
            Stream<LineParser> candidateParsers = Stream.of(LineParser.HEADER_VERSION);
            for (String line = br.readLine(); line != null; line = br.readLine()) {
                ++lineNumber;
                final String l = line;
                final Optional<LineParser> selected = candidateParsers.filter(p -> p.canHandle(l)).findFirst();
                if (selected.isPresent()) {
                    try {
                        selected.get().parse(line, pi);
                    } catch (StringIndexOutOfBoundsException | NumberFormatException e) {
                        throw new OrekitException(e,
                                                  OrekitMessages.UNABLE_TO_PARSE_LINE_IN_FILE,
                                                  lineNumber, source.getName(), line);
                    }
                    candidateParsers = selected.get().allowedNext();
                } else {
                    throw new OrekitException(OrekitMessages.UNABLE_TO_PARSE_LINE_IN_FILE,
                                              lineNumber, source.getName(), line);
                }
                if (pi.done) {
                    if (pi.nbEpochs != pi.file.getNumberOfEpochs()) {
                        throw new OrekitException(OrekitMessages.SP3_NUMBER_OF_EPOCH_MISMATCH,
                                                  pi.nbEpochs, source.getName(), pi.file.getNumberOfEpochs());
                    }
                    return pi.file;
                }
            }

            // we never reached the EOF marker
            throw new OrekitException(OrekitMessages.SP3_UNEXPECTED_END_OF_FILE, lineNumber);
10

Hi,

Because the line just before the EOF key is always the last epoch, I think that’s an interesting workaround to fix the issue.

Regards,
Bryan