java.lang.OutOfMemoryError: GC Overhead limit exceeded

Orekit Python Wrapper usage
1

Hi, all

I am doing orbit determination using large measurement.

It is not convergence in some case. Here is an example of exception trace.

orekit.JavaError: <super: <class 'JavaError'>, <JavaError object>>
    Java stacktrace:
org.orekit.errors.OrekitException: maximal count (30) exceeded
        at org.orekit.estimation.leastsquares.BatchLSEstimator.estimate(BatchLSEstimator.java:441)
Caused by: org.hipparchus.exception.MathIllegalStateException: maximal count (100) exceeded
        at org.hipparchus.util.Incrementor.lambda$static$0(Incrementor.java:41)
        at org.hipparchus.util.Incrementor.increment(Incrementor.java:238)
        at org.hipparchus.optim.nonlinear.vector.leastsquares.LevenbergMarquardtOptimizer.optimize(LevenbergMarquardtOptimizer.java:449)
        at org.orekit.estimation.leastsquares.BatchLSEstimator.estimate(BatchLSEstimator.java:435)

After a bit long time, it will raise java.lang.OutOfMemoryError: GC Overhead limit exceeded.

What should I do to handle this issue?

I tried google, it says,’ Prevention : Increase the heap size and turn off it with the command line flag -XX:-UseGCOverheadLimit.’.
https://www.geeksforgeeks.org/understanding-outofmemoryerror-exception-java/

Is that right? If so, how to Increase the heap size by using python wrapper.

Thanks,
RW

2

Hi @lirw1984

I performed some performance tests of Orekit a year ago with big number of measurements (i.e., about 10 milions for some tests) and I encountered the same issue. The purpose of my simulation was to progressively reproduce a full GNSS constellation orbit determination in a batch least squares process.

As far as I remember, in my computer, I reached this exception for about 200k measurements. I tried using a more powerful computer and with it I reached the issue for about 1 million measurements.

I have two questions:

  1. How many measurements do you have?

  2. How many parameters do you estimate?

Unfortunately, I think this issue is just related to the performance of your computer…

We are more an more thinking about adding decentralized computing orbit determination in Orekit. But, we didn’t started implementing this feature.

Best regards,
Bryan

3

Hi @bcazabonne,
Thanks for your reply.

What I’m doing is determining orbit for different objects, using different arcs, for about hundreds times. The number of measurements for every OD is small. And just one parameter is estimated.

So I fill it’s another issue.

I found an old topic. I’ll try to follow it. It may be caused by improperly using of java Objects in python.
https://forum.orekit.org/t/python-wrapper-outofmemoryerror-gc-overhead-limit-exceeded/531

4

Hi, @bcazabonne and all,

I’ve found a bug which leads to a memory leak, in tutorial.

I followed the orbit determination tutorial. It’s very wonderful, helping me starting to do an orbit determination.

Due to this GC overhead limit exceeded error, I google and google.

I tried to extract all java object in python following the previous mentioned topic. The error remains.

I noticed that someone says a WoW tool named VisualVM (https://visualvm.github.io/), which can be used to monitor the memory of VM. It’s really a powerful tool actually.

Here’s a dumped heap. We can see that there is a loop reference between lsBuilder and observer.

358438206
3584382061920×864 175 KB

I checked the src of tutorial.

In the class org.orekit.tutorials.estimation.common.AbstractOrbitDetermination, new an org.orekit.tutorials.estimation.common.OrbitDeterminationObserver instance and set it as the estimator’s observer.

            // Set the observer and estimate orbit
            estimator.setObserver(new OrbitDeterminationObserver(initialGuess,
                                                                 logStream,
                                                                 estimator));

In the method setObserver of class org.orekit.estimation.leastsquares.BatchLSEstimator, the property observer is assigned. That is, the estimator refers to the observer.

   /** Set an observer for iterations.
     * @param observer observer to be notified at the end of each iteration
     */
    public void setObserver(final BatchLSObserver observer) {
        this.observer = observer;
    }

And in the construct of class OrbitDeterminationObserver, the property estimator is assigned by the parameter estimator. That is, the observer refers to the estimator. A loop occurs.

    public OrbitDeterminationObserver(final Orbit initialGuess,
                                      final PrintStream logStream,
                                      final BatchLSEstimator estimator) {
        this.previousPV = initialGuess.getPVCoordinates();
        this.logStream  = logStream;
        this.estimator  = estimator;
        String header = FORMAT_HEADER;
        header = addParametersNames(header,
                                    estimator.getOrbitalParametersDrivers(true),
                                    estimator.getPropagatorParametersDrivers(true),
                                    estimator.getMeasurementsParametersDrivers(true));
        System.out.format(Locale.US, header);
        if (logStream != null) {
            logStream.format(Locale.US, header);
        }
    }

It does not matter in simple mode, or just one run orbit determination. For a massive batch orbit determination, it leads to memory out error.

The reference of observer to the estimator should be removed, by deleting the observer’s property estimator.

5

Hi @lirw1984,

That’s a very nice catch ! Thank you !

Did you try without the circular reference and observed some improvements in memory usage ?

Could you please open an issue in the Orekit tutorials issues’ tracker and link this forum thread to it ?

Thanks,
Maxime

6

Hi @MaximeJ ,

I’ve created an issue. It’s my pleasure.

Well, I removed the reference of estimator in observer. The memory error never occurs.

1 Like
7

Very nice !
I’m pretty sure @bcazabonne will be very interested in your findings.

8

That’s great! Congratulations!

Do you want to contribute the fix?

Bryan

9

Sure.

The changes are all in OrbitDeterminationObserver.java.

I commented the property estimator of OrbitDeterminationObserver. So in the method evaluationPerformed, the parameter evaluationsProvider is used to fetch all EstimatedMeasurement, instead of estimator.getLastEstimations().entrySet().

OrbitDeterminationObserver.java (13.9 KB)

PS: How to delete this uploaded file?
OrbitDeterminationObserver.java (13.9 KB)

10

Hi, was this issue already fixed in a release? I didn’t find anything related in the release notes.

11

Hi @guija,

Yes the issue was fixed with version 11.1 of the tutorials: the estimator attribute of OrbitDeterminationObserver was removed from the class.

Usually, the release notes of the tutorials are pretty concise: we just list the new tutorials. That’s why it didn’t appear.

1 Like