As you know, so far and for historical reasons, the continuous integration of Orekit and related software (Rugged, Hipparchus) has been entrusted to Jenkins. Since we use Gitlab CE and are convinced from experience that Gitlab CI/CD, the continuous integration orchestrator provided with Gitlab CE, is much more efficient and robust than Jenkins, it became clear over time that we had to abandon the second in favour of the first.
But Gitlab CI/CD only handles continuous integration. It doesnāt offer a dashboard for monitoring code quality. For this reason, Gitlab CI/CD couldnāt single-handedly replace Jenkins. We needed to provide you with another tool for code quality monitoring. We chose SonarQube because it provides extensive and detailed quality monitoring.
Another side effect of the Jenkins abandonment was that we needed an artifact repository manager because we were going to run all the continuous integration jobs in separate environments, inside Docker containers. We chose Nexus OSS Edition.
This new platform has been operational for a few weeks. We invite you to discover:
Pipelines. The .gitlab-ci.yml script performs the following tasks (i.e. ājobsā according to Gitlab vocabulary):
verify: Orekit build, testing and code analysis.
deploy:artifacts: Artifacts upload on the repository manager (only for master, releases and develop branches).
deploy:site: Technical documentation (Javadoc, Maven static site) publication on the website (only for master, releases and develop branches).
Quality report. The quality indicators are remarkable for a project with such a history. This report highlights the project quality requirement.
Published artifacts. The artifacts corresponding to Orekit official versions are published in the āmaven-releasesā repository. Those corresponding to the development branch (snapshots) are published in the āmaven-snapshotsā repository.
Any change requires a little period of adaptation and we will do our best to provide you with the necessary support if these tools are new to you. But we bet that you will be quickly convinced by this new platform.
Now we have to deal with Rugged (which will use the same platform) and Hipparchus (which canāt use Gitlab CI/CD because its repository is hosted on Github).
Thank you very much for the work you have done with @gbonnefille.
From a developer point of view, using Giltlab CI + SonarQube is much more user-friendly than the old Jenkins. Moreover, itās very nice to have a direct visual access to the CI result with the green or red icons. This allows to see direclty whether the job went well or not.
Yes, except for Hipparchus which repository is hosted on Github, the Community Edition of Gitlab fit our needs. The same goes for the community editions of SonarQube and Nexus.
No, we donāt have to. As I mentioned in another post, we have a fallback solution for Hipparchus thanks to Gitlab Actions ((but we could also have relied on Travis CI, Circle CI, Drone or another hosted CI platform).
Itās up to you, Orekit contributors, to decide. Thanks to the SonarQube community branch plugin, the quality control can be done on all branches (and we have therefore activated it on all branches). However, be aware that SonarQube handles the so-called āshort-livedā branches differently from the ālong-livedā ones. By configuration, the long-lived branches are master, develop and release-*, with the noticeable exception of the release-10.1 branch, identified as a short-lived branch by SonarQube due to a configuration error on my part.
Yes, there is something still in progress.
Currently, the effective release process (to Maven Central) is still a manual operation. The reason is effectively related to the signature process. The bot does not have a validated public key to sign its production. Thus, it only publish to the private repository, without signature verification. When the official release process take place, a rebuild is necessary. The published artifacts on Maven Central are not the one plublished on Nexus releases repository.
Currently, we should consider the Nexus releases repository as a staging repository, prior to the staging repository offered by Maven central.
In the future, as this should be discussed, the bot can have an official signing key and a direct access to Maven central. It will then be able to produce official (signed) packages. Such packages will be deployed on Nexus for review, and when the vote is accepted, the exact same packages could be deployed to Maven Central.
But this is still a TODO.
Any though about this? What about providing key to a bot?
From my point of view, there are two cases to distinguish:
If the change is minor and involves only a few lines of code, then I think itās a bit excessive to go through a pull request before doing a merge in develop. Indeed, it is unlikely that a minor change would imply a degradation of the quality of the existing code in develop.
If the change involves a lot of lines of code (e.g. a new feature), then it is interesting to create a dedicated branch for the change to push it to the repository so that SonarQube can check it before doing the merge in develop. Indeed, the probability of introducing a drop in code quality (i.e. code duplication, spotbugs, etc.) is much higher than in the first case.
Correcting spotbug warnings before a release is a not a very interesting step. Sometimes it is expensive in terms of correcting time. So, if we can limit this it would be cool
I donāt like the idea of giving the bot the release signing key. To me signing a release should be done by a person with a password. But I also donāt want to create extra rules if they donāt improve quality/security.
My understanding is that currently only a subset of committers have access to the release signing key. Giving the bot access would also give all committers access since they could cause the bot to do a build and sign it. If all committers should be able to make releases then we should make sure they have access to it, or move to using a KEYS file in the repository the way Apache does.
Yes, giving keys to a bot is a significant and scary change (SKYNET, Terminator, Matrixā¦ )
One of the most interest of such change is automation and infrastructure as code. I donāt blame anybody, but the last release revealed that release managers have various environment (JDK version at least). And we all know the limit of a list of manual commands: error like such environment.
Using a bot for the final part will ensure the process of the release.
Concerning the idea to keep the release process under control, we have to focus on what Gitlab has to offer. And its offering is quite limited: we can protect some branches (master?) and so, only a subset of people (Administrator) has the ability to push changes on this branch.
So, the bot will be activated only after action (merge+push) of a selected set of people.
If this is not acceptable, we can also imagine a more complex, but still automated, process.
The final publication can be managed by another project (lets call it orekit-release). In this project, we only have the script for the publication.
When the vote is OK, based on a version picked from the Nexus releases repository, someone activate the pipeline on orekit-release. This pipeline will:
download the version from the Nexus releases repository,
eventualy sign agin the package with its own key
publish on Maven Central
With such organisation, we can decouple the Maintainer of orekit (allowed to prepare release) and the release team (allowed to publish such release).
Taking a closer look at SonarQube it seems that many of the rules are disabled for Orekit.[1] In particular I think we should enable āUntrusted XML should be parsed with a local, static DTDā since that has been an issue in the past.
Iām guessing that your point is not directed at me, but Iāll clarify it anyway: the actual SonarQube configuration and set of activated rules are a first draft resulting from the Checkstyle configuration file available in the Orekit repository. Everything can be changed. It is up to you, the contributors, to discuss which rules should be added, ignored or modified.
For your information, the first time we launched SonarQube, we used its default configuration and this resulted in around 3000 ācode smellsā. This number of warnings seemed a bit too frightening and counterproductive for an introduction. So we deactivated the rules that reported really minor issues and generated a lot of warnings. After that, Luc and others started correcting the most important issues. It is now up to all of you to decide how far you want to go, what rules you are willing to assume and which ones you do not want to hear about.
Beyond the rules to apply, you will note that SonarQube keeps all quality reports, even those related to branches that have been deleted from the repository and obsolete tags (10.1-RCx). Fortunately, we can clean them up so that it only makes sense for us.
We could therefore delete all reports related to:
short-lived branches (at least the branches which do not exist anymore in the git repository)
Indeed, we have to discuss together (i.e.between Orekit developers) about the rules that have to be activated or not. Current SonarQube configuration is just a first try and have to be improved.
I think these discussions have to be done in a new post of the forum.
I agree with @bcazabonne that Gitlab CI + SonarQube are nicer to use than Jenkins.
Just a thought on this, @sdinot correct me if Iām wrong.
If you push a branch on the repository it will go through CI + Sonar.
However if you fork and go through the merge request process then the CI will not be activated.
Since we have a small configuration (only one job running on the CI at a time) we donāt want to be overflowed by all usersā forks on the CI.
So if you have developer access and want your work to be checked before integration it seems better to push a separate branch before merging it on develop.
In my humble opinion, systematically using a Merge-Request is a really good practice as it will avoid breaking the develop due to a bad commit (too quickly pushed as it was considered so trivialā¦).
Using a fork or having developer access is an option. Technically, the gitlab-runner can be shared between known (and estimated) forks.
In order to optimize actual process, I would suggest:
configure develop branch as the default one, in order to preset the right branch in the merge-request. Currently, master is set as the default branch, which require manual action when opening merge-request. The side effect will be for future clone, but I suspect that a user deciding to clone Orekit is also effectively interested on develop and not master.
when working on trivial fixes, push them in a dedicated branch, open a merge-request and immediatelly click on merge when pipeline succeeds. It seems to be a really good trade-off: you donāt have to expect after CI for merging and the work is controlled before being merged.
The CI is triggered even in the context of a merge request, but the result of the CI will depend on the origin of the merge request:
If the merge request comes from another branch of the Orekit repository, the pipeline will be launched, but the deployment jobs will be skipped because of the context restriction (merge_request is not listed in the only parameter).
If the merge request comes from another repository, the pipeline will not be executable as it will not have access to any Gitlab Runner (CI executor) by default. Once the timeout reached, the pipeline will be stated as failed (or ācanceledā, I canāt remember right now). But we can grant access to a Gitlab Runner to another repository when it turns out that its author really wants to contribute to Orekit (we donāt do this by default because the platform is open and public, and spammers have a field day).
It is not an issue. If we need more resources, weāll ask for them.
In light of what I just explained, it doesnāt matter, but working from another branch of the Orekit repository saves us a bit of sys. admin.
This new environment is great, many thanks to everyone involved !
I agree. From my experience, commits that seem trivial have a tendency to break many things. Complex new features are usually scrutinized carefully before being merged, but it is just human nature to underestimate the potential side effects of minor changesā¦
This would indeed make the creation of merge-requests less prone to errors.
Iāve worked like this on a project, and it is very comfortable. Most of the time it is just āfire and forgetā. And in the rare occurrence where all hell breaks loose, I was glad that the issue was caught before the merge.
)
when opening merge-request. The side effect will be for future clone, but I suspect that a user deciding to clone Orekit is also effectively interested on develop and not master.