Validation Exception parsing full space-track catalog

bradh · November 27, 2022, 5:13am

I’m trying to use Orekit 11.3 (maven dependency) to parse out the space-track catalog. I had hoped to use the NDM parser, but I’m getting a validation exception.

The parsing code is:

Ndm ndm = new ParserBuilder().buildNdmParser().parseMessage(new DataSource("/home/bradh/Downloads/spacetrack.xml"));

The trace looks like this

Exception in thread "main" org.orekit.errors.OrekitException: value for key OBJECT_ID has not been initialized
	at org.orekit.files.ccsds.section.CommentsContainer.checkNotNull(CommentsContainer.java:76)
	at org.orekit.files.ccsds.ndm.odm.CommonMetadata.validate(CommonMetadata.java:66)
	at org.orekit.files.ccsds.ndm.odm.omm.OmmParser.finalizeMetadata(OmmParser.java:212)
	at org.orekit.files.ccsds.section.XmlStructureKey.lambda$static$3(XmlStructureKey.java:51)
	at org.orekit.files.ccsds.section.XmlStructureKey.process(XmlStructureKey.java:82)
	at org.orekit.files.ccsds.section.XmlStructureProcessingState.processToken(XmlStructureProcessingState.java:69)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.process(AbstractMessageParser.java:180)
	at org.orekit.files.ccsds.ndm.NdmParser.processToken(NdmParser.java:205)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.process(AbstractMessageParser.java:180)
	at org.orekit.files.ccsds.utils.lexical.XmlLexicalAnalyzer$XMLHandler.endElement(XmlLexicalAnalyzer.java:200)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:618)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1728)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2899)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
	at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:542)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:889)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:825)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1224)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:637)
	at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
	at org.orekit.files.ccsds.utils.lexical.XmlLexicalAnalyzer.accept(XmlLexicalAnalyzer.java:77)
	at org.orekit.files.ccsds.utils.parsing.AbstractMessageParser.parseMessage(AbstractMessageParser.java:139)
	at net.frogmouth.demo.mavenproject1.Mavenproject1.main(Mavenproject1.java:50)
Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:982)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:929)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:457)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:294)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

I’m not at all familiar with the CCSDS standards, but from a brief look at the code, possibly the problem is this entry (and others like it), where there is no content to the OBJECT_ID element:

  <omm id="CCSDS_OMM_VERS" version="2.0">
    <header>
      <COMMENT>GENERATED VIA SPACE-TRACK.ORG API</COMMENT>
      <CREATION_DATE>2022-11-19T17:27:15</CREATION_DATE>
      <ORIGINATOR>18 SPCS</ORIGINATOR>
    </header>
    <body>
      <segment>
        <metadata>
          <OBJECT_NAME>TBA - TO BE ASSIGNED</OBJECT_NAME>
          <OBJECT_ID/>
          <CENTER_NAME>EARTH</CENTER_NAME>
          <REF_FRAME>TEME</REF_FRAME>
          <TIME_SYSTEM>UTC</TIME_SYSTEM>
          <MEAN_ELEMENT_THEORY>SGP4</MEAN_ELEMENT_THEORY>
        </metadata>
        <data>
...
      </data>
      </segment>
    </body>
  </omm>

I don’t particularly need that entry, and would be fine with just skipping over it. Is there a way to selectively ignore validation? Or another way to get out the rest of the data in the catalog?

luc · November 27, 2022, 9:52am

Hi @bradh welcome to the forum.

This seems to be a violation of CCSDS standard to me. I looked at both the latest version (CCSDS 502.0-B-2) and the pink book for the upcoming 3.0 standard, not publicly available yet. Both states that in Key-Value Notation “A non-empty value field must be specified for each obligatory keyword.” (see section 6.5.1 in the latest version, it will be in section 7.5.1 in the upcoming 3.0 standard). The XML format is specified with reference to the KVN format, for example in the pink book there will be a sentence that reads: “The keywords in the metadata and data sections shall be those specified in Section 4. The rules for including any of the keyword tags in the OMM/XML are the same as those specified for the OMM/KVN in Section 4.”

So the value must be non-empty.

I would first suggest that you refer to space-track and ask them to fix the files. This would be the best solution, but would probably take a very long time to do, if they accept and this would probably imply changing existing database entries.

Another thing you could do is preprocess the files when you read them, and replace <OBJECT_ID/> with <OBJECT_ID>unknown</OBJECT_ID> before parsing it with Orekit.

The last thing I could propose is that you could open a feature request in the forge issue tracker asking for a filtering capability at token level. This would be to add a addTokenFilter(Function<ParseToken, ParseToken> filter) so user can change tokens on the fly. In your case, the filter would check if the token name is OBJECT_ID and the raw content is null or emtpy, and in this case would return a new token with a value set to unknown for example. In all other cases, your filter would just return the input token. This change is a simple one (I have already started looking at it), and fortunately as the next version will be a major version (12.0), we can add methods to public interfaces.

luc · November 27, 2022, 2:30pm

OK, I have created issue 991 for this.
The corresponding merge request is merge request 314

bradh · November 28, 2022, 7:03am

Thanks for the quick response on the forum, for creating the issue for me, and for the MR!

Very much appreciated (and very impressed).

luc · November 28, 2022, 3:07pm

Thanks for the kind words, it’s really appreciated.

Beware that I will change the API slightly. After some thoughts, setting up the filters is probably best configured at the parser builder level rather than at the parser itself. This change could perhaps allow to include the feature in the 11.X series rather than having to wait for 12.0.

bradh · November 30, 2022, 11:48pm

I have also followed up with space-track and they are responsive, but its not yet clear what the outcome or timeframe will be.

bradh · December 2, 2022, 2:34am

Some further information: The parts that cause problem are from the analyst object range - there is tracking information, but not lineage to a launch. In this case, there isn’t an international identifier or object ID assigned by 18SDS.

@luc do you have a recommendation or preference for what should be inserted in this case? I suggested a UUID or a constant text entry, but maybe there is a better way you can suggest?

Serrof · December 2, 2022, 5:23pm

In that case “Analyst Object” could be a good default string.
So you’re able to access OMM for those? I thought space-track didn’t publish elements on them.

Best,
Romain.

bradh · December 2, 2022, 9:38pm

The analyst objects are available in OMM format from the catalog page (requires login):

https://www.space-track.org/#recent

There are downloads with well-tracked analyst objects available as part of the full catalog, and as a distinct subset (e.g. https://www.space-track.org/basicspacedata/query/class/gp/EPOCH/>now-30/NORAD_CAT_ID/80000--89999/orderby/NORAD_CAT_ID/format/xml/emptyresult/show)

luc · December 2, 2022, 10:40pm

The CCSDS standard reads:
There is no CCSDS-based restriction on the value for this keyword, but it is
recommended that values be the international spacecraft designator as published
in the SPACEWARN Bulletin (reference [2]). Recommended values have the format
YYYY-NNNP{PP}

At Orekit level, the id is just a string, but if one of the method getLaunchYear, getLaunchNumber or getLaunchPiece is called on the metadata, it assumes the id is an international designator and attempts to parse it to extract the desired information. If the id is not an international designator, then it will generate an error with a NOT_VALID_INTERNATIONAL_DESIGNATOR message.

So if you don’t call these methods, you can use whatever you want as a replacement string. I would suggest either a UUID or a hash.

bradh · February 18, 2023, 3:28am

I got an update from the Space-Track support team:

Following up with you to let you know that analyst TLEs coming from 18th Space Defense Squadron without an international designator (OBJECT_ID), now have the CCSDS-compatible value “UNKNOWN” placeholder value instead of NULL.