Replacing TLEs with OMM (XML?)

matevzb · June 13, 2024, 6:49pm

Hi,

I’m trying to prepare the script for obtaining orbital elements for the given satellite using space-track.org and use it in Orekit for propagator. Since TLE-based API is marked as “To Be Deprecated”, I’m trying to use OMM instead. Space-track can only export it as json or xml (apart from csv and html) and I can’t really wrap my head around to how to approach this correctly in Orekit.

So, based on topic from 2021 on the new framework for CCSDS messages, the new implementation should be capable of automatically detecting between the two formats.

When I try to parse the XML, I get “org.orekit.errors.OrekitException: unsupported format for file …”

Moreover, all test files for CCSDS (on github under /src/test/resources/ccsds/) are in KVN format, none is in XML. Is XML supported? Does anybody use space-track.org API for obtaining orbital elements for use in orekit (apart from TLEs)?

luc · June 13, 2024, 7:47pm

There are test files in XML, for example OMM-with-units.xml, so yes XML is supported, in fact both for parsing and for writing.

Could you give an example of which file cannot be parsed and the stack trace of the error, with the full error message?

matevzb · June 14, 2024, 5:48am

I used the API
https://www.space-track.org/basicspacedata/query/class/gp/NORAD_CAT_ID/59035/orderby/EPOCH%20desc/format/xml/limit/1/emptyresult/show to retrieve the OMM: starlink.xml (2.0 KB)

When I run the following

from org.orekit.files.ccsds.ndm import ParserBuilder
from org.orekit.data import DataSource
print(ParserBuilder().buildOmmParser().parseMessage(DataSource("starlink.xml")))

I get

---------------------------------------------------------------------------
JavaError                                 Traceback (most recent call last)
Cell In[10], line 3
      1 from org.orekit.files.ccsds.ndm import ParserBuilder
      2 from org.orekit.data import DataSource
----> 3 print(ParserBuilder().buildOmmParser().parseMessage(DataSource("starlink.xml")))

JavaError: <super: <class 'JavaError'>, <JavaError object>>
    Java stacktrace:
org.orekit.errors.OrekitException: unsupported format for file starlink.xml
	at org.orekit.files.ccsds.section.XmlStructureProcessingState.processToken(XmlStructureProcessingState.java:64)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.process(AbstractMessageParser.java:220)
	at org.orekit.files.ccsds.utils.lexical.XmlLexicalAnalyzer$XMLHandler.startElement(XmlLexicalAnalyzer.java:206)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)
	at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(XMLDTDValidator.java:744)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1358)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$ContentDriver.scanRootElementHook(XMLDocumentScannerImpl.java:1288)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3131)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:851)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:601)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:504)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
	at org.orekit.files.ccsds.utils.lexical.XmlLexicalAnalyzer.accept(XmlLexicalAnalyzer.java:79)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.parseMessage(AbstractMessageParser.java:156)

Emiliano · June 14, 2024, 7:27am

Hello @matevzb
Beware that space-track is not always compliant with CCSDS standards when building NDMs. For example, their OPMs require fields that are not in the CCSDS handbooks (e.g., USER_DEFINED entries in the header, etc.). In the past, I had instances where CDMs gotten from space-track could not be parsed by Orekit because a separating blank space was missing between a value and its units. It’s possible that in your case the OMM is not built following the CCSDS format. In my case I had to write my own crude parser for the CMD.

luc · June 14, 2024, 7:52am

There are two problems here.

The first problem is that this file is not an OMM message, it is an NDM message that contains an OMM message. So if you parse it using the NDM parser, it parses much more entries.

The second problem is that among the entries, there is a USER_DEFINED parameter named DECAY_DATE that is empty. I think this is a violation of the ODM standard as paragraph 7.5.1 (in CCSDS 502.0-B-3 from april 2023) states that

A non-empty value field must be assigned to each mandatory keyword
except for *‘_START’ and *‘_STOP’ keyword values.

However, USER_DEFINED is a mandatory parameter only in the USER_START/USER_STOP section of OCM, not OMM. In OEM, OPM and OMM, USER_DEFINED are optional parameters, not mandatory parameters. I find strange to put a mandatory parameter with empty value (they could just remove the parameter), but as it obviously happens, we could manage this.

Could you open an issue for this?

luc · June 14, 2024, 8:31am

Well, in fact there is no need to open an issue for this, we already have a generic feature that was in fact created exactly for that: token filters.
Between the construction of the ParserBuilder and the call to buildNdmParser, you should insert a filter that would replace empty user defined tokens with tokens having a value set to “unknown”. Here is how to do that:

 NdmParser parser = new ParserBuilder().
                    withFilter(token -> {
                       if (token.getName().startsWith("USER_DEFINED") &&
                           (token.getRawContent() == null || token.getRawContent().isEmpty())) {
                           // replace null/ empty entries with "unknown"
                           return Collections.singletonList(new ParseToken(token.getType(), token.getName(), "unknown",
                                                                           token.getUnits(),
                                                                           token.getLineNumber(), token.getFileName()));
                       } else {
                          return Collections.singletonList(token);
                       }
                    }).
                    buildNdmParser();

matevzb · June 15, 2024, 7:59am

Ok, getting somewhere. If I manually edit the source xml to change the user defined DECAY_DATE parameter to unknown (as is done in the filter), I get the result. A bit awkward in Python due to castings needed, but it works.

It appears as if Python wrapper for ParserBuilder doesn’t implement .withFilter method. Does anybody has any thoughts on that?

from org.orekit.data import DataSource
from org.orekit.data import PythonReaderOpener
from java.io import StringReader

class MyReaderOpener(PythonReaderOpener):
    def __init__(self, msgStr):
        self.msgStr = msgStr
        super(MyReaderOpener, self).__init__()
        
    def init(self, m):
        pass

    def openOnce(self):
        return StringReader(self.msgStr)


objectOMM = '<?xml version="1.0" encoding="utf-8"?><ndm xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://sanaregistry.org/r/ndmxml_unqualified/ndmxml-3.0.0-master-3.0.xsd"><omm id="CCSDS_OMM_VERS" version="3.0"><header><COMMENT>GENERATED VIA SPACE-TRACK.ORG API</COMMENT><CREATION_DATE>2024-06-13T02:09:14</CREATION_DATE><ORIGINATOR>18 SPCS</ORIGINATOR></header><body><segment><metadata><OBJECT_NAME>STARLINK-31207</OBJECT_NAME><OBJECT_ID>2024-038N</OBJECT_ID><CENTER_NAME>EARTH</CENTER_NAME><REF_FRAME>TEME</REF_FRAME><TIME_SYSTEM>UTC</TIME_SYSTEM><MEAN_ELEMENT_THEORY>SGP4</MEAN_ELEMENT_THEORY></metadata><data><meanElements><EPOCH>2024-06-12T07:08:39.506208</EPOCH><MEAN_MOTION>15.25907591</MEAN_MOTION><ECCENTRICITY>0.00019410</ECCENTRICITY><INCLINATION>43.0019</INCLINATION><RA_OF_ASC_NODE>8.4567</RA_OF_ASC_NODE><ARG_OF_PERICENTER>268.8898</ARG_OF_PERICENTER><MEAN_ANOMALY>91.1730</MEAN_ANOMALY></meanElements><tleParameters><EPHEMERIS_TYPE>0</EPHEMERIS_TYPE><CLASSIFICATION_TYPE>U</CLASSIFICATION_TYPE><NORAD_CAT_ID>59035</NORAD_CAT_ID><ELEMENT_SET_NO>999</ELEMENT_SET_NO><REV_AT_EPOCH>1772</REV_AT_EPOCH><BSTAR>-0.00011876000000</BSTAR><MEAN_MOTION_DOT>-0.00003363</MEAN_MOTION_DOT><MEAN_MOTION_DDOT>0.0000000000000</MEAN_MOTION_DDOT></tleParameters><userDefinedParameters><USER_DEFINED parameter="SEMIMAJOR_AXIS">6866.198</USER_DEFINED><USER_DEFINED parameter="PERIOD">94.370</USER_DEFINED><USER_DEFINED parameter="APOAPSIS">489.396</USER_DEFINED><USER_DEFINED parameter="PERIAPSIS">486.731</USER_DEFINED><USER_DEFINED parameter="OBJECT_TYPE">PAYLOAD</USER_DEFINED><USER_DEFINED parameter="RCS_SIZE">LARGE</USER_DEFINED><USER_DEFINED parameter="COUNTRY_CODE">US</USER_DEFINED><USER_DEFINED parameter="LAUNCH_DATE">2024-02-25</USER_DEFINED><USER_DEFINED parameter="SITE">AFETR</USER_DEFINED><USER_DEFINED parameter="DECAY_DATE">unknown</USER_DEFINED><USER_DEFINED parameter="FILE">4343479</USER_DEFINED><USER_DEFINED parameter="GP_ID">259443997</USER_DEFINED></userDefinedParameters></data></segment></body></omm></ndm>'

from org.orekit.data import DataSource
from org.orekit.files.ccsds.ndm import ParserBuilder
from org.orekit.files.ccsds.ndm import Ndm
from org.orekit.files.ccsds.ndm.odm.omm import Omm

parser = ParserBuilder().buildNdmParser()

ro = MyReaderOpener(objectOMM)
ds = DataSource("ommProvider", MyReaderOpener(objectOMM))
msg = parser.parseMessage(ds)
ndmMsg = Ndm.cast_(msg)
OMM = [Omm.cast_(OMMentry) for OMMentry in ndmMsg.getConstituents()]
tle = OMM[0].generateTLE()

print(tle)

petrus.hyvonen · June 16, 2024, 8:25am

Hi @matevzb

It is the casting that is not fully 1:1 with java. You probably need to cast the parser back to a ParserBuilder to get visibility of the withFilter method.

ParserBuilder.cast_(parser).withFilter(…)

Note that for the filter function, you will need to subclass the PythonFunction from org.orekit.python package, and implement the apply method.

Greetings

matevzb · June 16, 2024, 8:31am

There are multiple with___ methods on the parser available, but no withFilter

['__class__',
 '__delattr__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getstate__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '_jobject',
 'boxfn_',
 'buildAcmParser',
 'buildAemParser',
 'buildApmParser',
 'buildCdmParser',
 'buildNdmParser',
 'buildOcmParser',
 'buildOemParser',
 'buildOmmParser',
 'buildOpmParser',
 'buildTdmParser',
 'cast_',
 'class',
 'class_',
 'conventions',
 'dataContext',
 'defaultInterpolationDegree',
 'defaultMass',
 'equals',
 'equatorialRadius',
 'flattening',
 'getClass',
 'getConventions',
 'getDataContext',
 'getDefaultInterpolationDegree',
 'getDefaultMass',
 'getEquatorialRadius',
 'getFlattening',
 'getMissionReferenceDate',
 'getMu',
 'getParsedUnitsBehavior',
 'getRangeUnitsConverter',
 'hashCode',
 'instance_',
 'isSimpleEOP',
 'missionReferenceDate',
 'mu',
 'notify',
 'notifyAll',
 'of_',
 'parameters_',
 'parsedUnitsBehavior',
 'rangeUnitsConverter',
 'simpleEOP',
 'toString',
 'wait',
 'withConventions',
 'withDataContext',
 'withDefaultInterpolationDegree',
 'withDefaultMass',
 'withEquatorialRadius',
 'withFlattening',
 'withMissionReferenceDate',
 'withMu',
 'withParsedUnitsBehavior',
 'withRangeUnitsConverter',
 'withSimpleEOP',
 'wrapfn_']

petrus.hyvonen · June 16, 2024, 8:42am

Hm. not sure,

It seems like the ParserBuilder object is the one having the withFilter method, so it should likely be applied prior to the buildNdmParser(), you are right the NdmParser object cannot be cast back to a ParserBuilder.

petrus.hyvonen · June 16, 2024, 8:49am

Seems a bit odd this. The withFilter is in the orekitwrapper java code, but I cannot see it on the ParserBuilder object in Python.

I can also confirm that the method is visible in the alternative orekit_jpype wrapping.

petrus.hyvonen · June 16, 2024, 9:32am

Ok, this is an wrapper issue. ParserBuilder method withFilter not available in Python (#464) · Issues · Orekit Labs / Orekit Python Wrapper · GitLab

The methods are not part of the generated code from the reflection of the java code to identify methods.

Update:
There’s a new build released now, orekit wrapper 12.0.1 build 8 on conda-forge servers. This includes the wrapped methods. Please let us know how this progresses.

matevzb · June 16, 2024, 6:27pm

I appreciate your effort, but I would need a bit more help to get through this one.

This is the code I have at the moment, just returning the tokens as-is.

from org.orekit.data import DataSource
from org.orekit.data import PythonReaderOpener
from java.io import StringReader

class MyReaderOpener(PythonReaderOpener):
    def __init__(self, msgStr):
        self.msgStr = msgStr
        super(MyReaderOpener, self).__init__()
        
    def init(self, m):
        pass

    def openOnce(self):
        return StringReader(self.msgStr)


from org.orekit.files.ccsds.utils.lexical import ParseToken
from typing import List, Optional

objectOMM = '<?xml version="1.0" encoding="utf-8"?><ndm xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://sanaregistry.org/r/ndmxml_unqualified/ndmxml-3.0.0-master-3.0.xsd"><omm id="CCSDS_OMM_VERS" version="3.0"><header><COMMENT>GENERATED VIA SPACE-TRACK.ORG API</COMMENT><CREATION_DATE>2024-06-13T02:09:14</CREATION_DATE><ORIGINATOR>18 SPCS</ORIGINATOR></header><body><segment><metadata><OBJECT_NAME>STARLINK-31207</OBJECT_NAME><OBJECT_ID>2024-038N</OBJECT_ID><CENTER_NAME>EARTH</CENTER_NAME><REF_FRAME>TEME</REF_FRAME><TIME_SYSTEM>UTC</TIME_SYSTEM><MEAN_ELEMENT_THEORY>SGP4</MEAN_ELEMENT_THEORY></metadata><data><meanElements><EPOCH>2024-06-12T07:08:39.506208</EPOCH><MEAN_MOTION>15.25907591</MEAN_MOTION><ECCENTRICITY>0.00019410</ECCENTRICITY><INCLINATION>43.0019</INCLINATION><RA_OF_ASC_NODE>8.4567</RA_OF_ASC_NODE><ARG_OF_PERICENTER>268.8898</ARG_OF_PERICENTER><MEAN_ANOMALY>91.1730</MEAN_ANOMALY></meanElements><tleParameters><EPHEMERIS_TYPE>0</EPHEMERIS_TYPE><CLASSIFICATION_TYPE>U</CLASSIFICATION_TYPE><NORAD_CAT_ID>59035</NORAD_CAT_ID><ELEMENT_SET_NO>999</ELEMENT_SET_NO><REV_AT_EPOCH>1772</REV_AT_EPOCH><BSTAR>-0.00011876000000</BSTAR><MEAN_MOTION_DOT>-0.00003363</MEAN_MOTION_DOT><MEAN_MOTION_DDOT>0.0000000000000</MEAN_MOTION_DDOT></tleParameters><userDefinedParameters><USER_DEFINED parameter="SEMIMAJOR_AXIS">6866.198</USER_DEFINED><USER_DEFINED parameter="PERIOD">94.370</USER_DEFINED><USER_DEFINED parameter="APOAPSIS">489.396</USER_DEFINED><USER_DEFINED parameter="PERIAPSIS">486.731</USER_DEFINED><USER_DEFINED parameter="OBJECT_TYPE">PAYLOAD</USER_DEFINED><USER_DEFINED parameter="RCS_SIZE">LARGE</USER_DEFINED><USER_DEFINED parameter="COUNTRY_CODE">US</USER_DEFINED><USER_DEFINED parameter="LAUNCH_DATE">2024-02-25</USER_DEFINED><USER_DEFINED parameter="SITE">AFETR</USER_DEFINED><USER_DEFINED parameter="DECAY_DATE">unknown</USER_DEFINED><USER_DEFINED parameter="FILE">4343479</USER_DEFINED><USER_DEFINED parameter="GP_ID">259443997</USER_DEFINED></userDefinedParameters></data></segment></body></omm></ndm>'

from org.orekit.data import DataSource
from org.orekit.files.ccsds.ndm import ParserBuilder
from org.orekit.files.ccsds.ndm import Ndm
from org.orekit.files.ccsds.ndm.odm.omm import Omm

from org.orekit.python import PythonFunction

class MyFilter(PythonFunction):
    def apply(self, token: ParseToken) -> List[ParseToken]:                
        return [token]

myFilt = MyFilter()
parser = ParserBuilder().withFilter(myFilt)
ndmparser = parser.buildNdmParser()

ro = MyReaderOpener(objectOMM)
ds = DataSource("ommProvider", MyReaderOpener(objectOMM))
msg = ndmparser.parseMessage(ds)
ndmMsg = Ndm.cast_(msg)
OMM = [Omm.cast_(OMMentry) for OMMentry in ndmMsg.getConstituents()]
tle = OMM[0].generateTLE()

When the parseMessage is called on the parser, I get the type error in the apply method.

---------------------------------------------------------------------------
JavaError                                 Traceback (most recent call last)
Cell In[28], line 40
     38 ro = MyReaderOpener(objectOMM)
     39 ds = DataSource("ommProvider", MyReaderOpener(objectOMM))
---> 40 msg = ndmparser.parseMessage(ds)
     41 ndmMsg = Ndm.cast_(msg)
     42 OMM = [Omm.cast_(OMMentry) for OMMentry in ndmMsg.getConstituents()]

JavaError: <super: <class 'JavaError'>, <JavaError object>>
    Java stacktrace:
java.lang.RuntimeException: type error
	at org.orekit.python.PythonFunction.apply(Native Method)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.process(AbstractMessageParser.java:200)
	at org.orekit.files.ccsds.utils.lexical.XmlLexicalAnalyzer$XMLHandler.startElement(XmlLexicalAnalyzer.java:206)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)
	at com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator.startElement(XMLDTDValidator.java:744)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1358)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$ContentDriver.scanRootElementHook(XMLDocumentScannerImpl.java:1288)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3131)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:851)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:601)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:504)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
	at org.orekit.files.ccsds.utils.lexical.XmlLexicalAnalyzer.accept(XmlLexicalAnalyzer.java:86)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.parseMessage(AbstractMessageParser.java:156)

petrus.hyvonen · June 16, 2024, 7:10pm

This seems to work, following Luc’s example

from java.util import Collections

class MyFilter(PythonFunction):
    def apply(self, token: ParseToken) -> List[ParseToken]:    
        return Collections.singletonList(token)

matevzb · June 16, 2024, 7:52pm

Thank you!

I haven’t used Python wrappers for Java code before and I’m still learning the tricks.
For reference, here is the code that loads the xml file exported from space-track and generates a TLE.

from org.orekit.data import DataSource
from org.orekit.files.ccsds.utils.lexical import ParseToken
from typing import List, Optional

from org.orekit.files.ccsds.ndm import ParserBuilder, Ndm
from org.orekit.files.ccsds.ndm.odm.omm import Omm

from org.orekit.python import PythonFunction
from java.util import Collections

class MyFilter(PythonFunction):
    def apply(self, token: ParseToken) -> List[ParseToken]:
        token = ParseToken.cast_(token)        
        if (token.getName().startswith("USER_DEFINED") and (token.getRawContent() is None or token.getRawContent() == "")):            
            # Replace null/empty entries with "unknown"
            new_token = ParseToken(token.getType(), token.getName(), "unknown", token.getUnits(), token.getLineNumber(), token.getFileName())
            return Collections.singletonList(new_token)
        else:
            return Collections.singletonList(token)            


myFilt = MyFilter()
parser = ParserBuilder().withFilter(myFilt)
ndmparser = parser.buildNdmParser()

msg = ndmparser.parseMessage(DataSource('starlink.xml'))
ndmMsg = Ndm.cast_(msg)
OMM = [Omm.cast_(OMMentry) for OMMentry in ndmMsg.getConstituents()]
tle = OMM[0].generateTLE()

print(tle)