Random job failures in Orekit CI/CD

Orekit development
1

Several of us have noticed random (but recurring) failures of the “verify” job in Orekit’s CI/CD pipeline. These are due to tests failing for reasons unrelated to the change made. When this happens, we usually solve the problem by restarting the job. The second execution is successful, even though no new code changes have been made. Sometimes, the job has to be restarted two or three times before it runs successfully.

A few months ago, I opened a ticket about this issue, which was processed and closed:

https://gitlab.orekit.org/orekit/orekit/-/issues/1579

I don’t know if the problem we are facing today is a resurgence of the same problem, or if it is the overall testing strategy that needs to be reviewed.

Here is an example:

2

Hi Sebastien,

I still suspect the default DataContext is the culprit, whose content depends on the execution order.
One thing I would like to try is the following, in ExceptionalDataContextTest:

    @AfterAll
    static void tearDown() {
        DataContext.setDefault(new LazyLoadedDataContext());
    }

Cheers,
Romain.

3

It’s pushed on develop. Let’s see. So far the only unexplained failure I’ve seen is on the patch branch which doesn’t have this.

4

FYI I think it solved the issue.

Cheers,
Romain.

2 Likes