Automatically deploy Maven Site

In order to automatize Orekit’s releasing process, one topic is the generated Maven site.

On the target site, it seems we host:

  • each RELEASE version
  • one SNAPSHOT version (the last one)

The question is like a similar question for the whole website: what is the expected pipeline?

I started reading the releasing.md and I realized there is a call for vote. Technically, I understand the committee must review everything from a sort of staging.

Based on this documentation, I propose the following:

  • push on develop will update the SNAPSHOT version of the maven site (note that this also trigger a publication of the package in the staging maven repository hosted at https://packages.orekit.org/).
  • push on release-X.Y will update the RELEASE version of the maven site, but as the website is not updated, it is not directly accessible from external users (note that this also trigger a publication of the package in the release maven repository hosted at https://packages.orekit.org/).
  • push on master will update the RELEASE version of the maven site and will update the link site-orekit-latest.

Concurrently, the website must be modified manually and will follow the pipeline described previously:

So, during the review:

  • orekit is updated on its release-X.Y branch (and tested via http://packages.orekit.org)
  • website is updated on its develop branch

When the vote is OK:

  • orekit’s master branch is updated (and someone has to push manually into Maven Central)
  • website’s master branch is updated

Is it correct/acceptable?

I think it would be great!
What about the non-maven artifacts hat are generated for review, and after vote put to the downloads page?
Currently, the manual provess for release candidates is to set up temporary staging/donwloads directory on the server (ssh, mkdir, vi index.html…), and to copy the new artifacts manually to the official download directory when the vote succeeds.

What artifacts are you talking about?

A generic answer: depending on the review process expected, but artifacts can be hosted directly by Gitlab, associated to the relevant pipeline. Please, look at my experiments: https://gitlab.orekit.org/gbonnefille/orekit/pipelines?scope=branches&page=1 This view focus on branches status. As you can see in the right, a button allow to download anything declared as artifact during the pipeline. When selecting a pipeline and then a job, you are also able to browse the artifacts avoiding a full download.

Having the full list of the artifacts that must be published and reviewed can help to draw a detailed picture.

These ones: Download or Orekit – Downloads. The most important one is the orekit-X.Y-sources.zip which is completely independant from the maven artifacts (the sources.jar maven artifact is only an extract for IDE debugging, not a complete distribution allowing to rebuild the project from sources).

These artifacts are nevertheless built by maven, when the mvn assembly:single goal is run. They are simply not uploaded to maven central, which cannot do anything with them.

OK, we have to add a step to deploy these artifacts.

As I recently also worked on website auto-publishing, I suggest a different approach.
Instead of using an explicit staging/download directory, I propose to publish the new maven site directly to its final destination (since my recent modifications, it is now /var/www/mvn-site/site-orekit-X.Y) and play with the new staging homepage (https://test.orekit.org/).
Process become:

  1. push a new release candidate → bot will publish maven site on /var/www/mvn-site/site-orekit-X.Y and Maven packages to the project’s Nexus Release repository
  2. update the whole website (mostly adding the new release to the release list) and push on develop branch → bot will publish website on https://test.orekit.org/
  3. call for vote: people can check the website and the release
  4. if the vote failed, iterate from 1.
  5. if vote OK, merge website to master branch → bot will publish on https://www.orekit.org
  6. if vote OK, merge orekit’s changes into master branch → bot will (re)publish on /var/www/mvn-site/site-orekit-X.Y (perhaps there was a trivial change…)
  7. if vote OK, deploy Maven packages manually to Maven Central

Is such process acceptable?

Looking deeper, it seems this part is slightly different currently. It seems that:

  • artifacts are stored as attachments to the release publication on GitLab
  • links to downloads from website are a mix between specific links into maven site and Redirect rules to point to Gitlab dedicated artifacts.

I suppose all of this changes were made by @sdinot.

Before breaking anything, I’m interested in understanding why artifacts are stored in GitLab and not directly in the website (like maven site). It seems that such solution brings new complexity as Gitlab’s attachments use a not predictable internal id in the URL. Furthermore, there is now a Nexus repository that can be used to store raw artifacts. Perhaps it could be an option to store these artifacts with a predictable URL.

The reason was an historical one. Before Gitlab, we used Redmine/Chiliproject which had a files download feature. When we switched to Gitlab, we tried:

  1. to reproduce the older Redmine feature
  2. to make use of Gitlab releases feature Releases · Orekit / Orekit · GitLab

the Gitlab releases feature allows to attach file to the comment about the release, which we used. This feature uses non-predictable URL, hence the redirection added recently.

I would suggest some rationalization to help automation.

I noticed that the three packages (jar, sources.jar and javadoc.jar) are already hosted on Maven central:

I also noticed that the sources.zip is also available via this repository:

So, content is already hosted. Furthermore, all these URLs are predictable. What about directly reusing these contents, avoiding (complex) duplication on our own side?

The missing part of this maze is the staging one. But I suspect I’m able to reproduce this solution on the Orekit’s Nexus. Would it be acceptable?

This is not the same as sources.jar.

The sources.jar package as a maven artifact is a stripped-down version of sources.zip, created for a different purpose. It only contains the sources themselves (subdirectories of src/main/java) for use in an IDE, for debugging purposes. The sources.ziparchive contains everything needed to build the package (sources, tests, resources, build files, plugins configuration files…)

Both are needed.

Yes, I simply noticed both are already hosted in the maven central.

It is an error for sources.zip in version 10.0. The file is not present in earlier versions
(see https://repo1.maven.org/maven2/org/orekit/orekit/9.3.1/).

I don’t really want to force people using maven only, and sources.zip is not really a maven artifact, so its kind of awkward for me to use this as our primary download area. I would really prefer to have sources.zip handled by ourselves. However, this is only my opinion, if other people prefer otherwise, I’d like to read their point of view too.

As a tradeoff, what about storing these artifacts in the Orekit’s Nexus, in the repository RELEASE?

I will check that:

  • I can deploy them all in this repository
  • I can forge a link for a direct download.

This seems fine to me.

Humh… since Gitlab offers direct links for downloading a release as a tar/zip/tar.gz/tar.bz2, is there still any interest to produce a orekit-x.y-sources.zip from Maven? (See https://gitlab.orekit.org/orekit/orekit/-/releases#10.0)

Please, note that the packages provided by Gitlab can be obtained locally with git archive -o orekit-x.y-sources.zip --format zip HEAD. The main differences with the zip provided by maven is that it can embed local files not managed by git, while git archive is strictly limited to versioned files.

I did some changes to have the Maven site automatically deployed on spoutnik.

The only remaining actions are:

  • to update the symbolic link between site-orekit-development and site-orekit-X.Y-SNAPSHOT,
  • to remove the previous *-SNAPSHOT after a release.

My understanding is: such actions are casual enough to remain manuals.