Using Jenkins multibranch pipeline for continuous integration

Orekit development
1

Dear all,

Continuing the migration of the development platform, I deployed a new instance of Jenkins for Orekit:

https://ci.orekit.org/

Now, before migrating the continuous integration jobs, I would propose you a Pipeline as Code approach, based on the Pipeline plugins family, in particular, the Multibranch Pipeline plugin.

Here are the required definitions (diped into the Jenkins documentation):

  • Pipeline: Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code” via the Pipeline domain-specific language (DSL) syntax.

  • Multibranch Pipeline: the Multibranch Pipeline project type enables you to implement different Jenkinsfiles for different branches of the same project. In a Multibranch Pipeline project, Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control. This eliminates the need for manual Pipeline creation and management.

  • Jenkinsfile: This is a text file that contains the definition of a Jenkins Pipeline and is checked into source control.

I have created a first draft of this script for Orekit in a dedicated branch:

https://gitlab.orekit.org/sdinot/orekit/tree/feature/jenkins-multibranch-pipeline

You can see the result of such pipeline in the new instance of Jenkins, through:

I could open a merge request but I need your opinion and I think that my Jenkinsfile can be largely improved (for example, to get back and process the coverage report).

Reviews and ideas are welcome!

More generally, what is your opinion about this approach?

Sébastien

2

I think the multi-branch pipeline approach is a good one. We would probably want it to run the equivalent of mvn verify site to check the build and the site build work. I have some experience with Jenkinsfiles and could help set one up.

I also noticed that GitLab has its own CI feature. Would it make sense to use GitLab CI so that there is closer integration and less tools to maintain? Or is Jenkins better than GitLab CI?

3

Within my capacity, help is welcome. As I doesn’t know Java and Maven (I am a C++ addict), I really have no opinion about the right way to do the job in the Orekit context. So, do not hesitate to express your point of view and your needs.

You are right, Gitlab comes with its own CI tool and the latter works fine, in particular with containers. I already use it on a couple of personal projects and some of my colleagues use it on internal projects. But this tool lacks of reporting capabilities. It does not collect, process and display the reports generated by the QA tools like JUnit, Jacoco or Checkstyle. From this point of view, it represents a regression.

So, for the moment, we decided to keep Jenkins but the situation is far from being immutable! :wink:

Sébastien

4

I took a look at the Jenkinsfile, but it looks like you already made it awesome. :slight_smile: So I don’t think there anything I can add to it.

Evan

5

I just discover Jenkinsfile syntax.

Why using “if/then/else” for branch name checking and not using the when+branch approach?
I have the feeling that a declarative approach should avoid looking like a shell script. (a script can be hard to parse/understand while a declarative approach simplify this understanding)

6

Upon reflection, with a view toward a devops approach, I think that I did a wrong choice:

  • We need to deploy a new version of the binary packages and the related documentation when we release a new official version of Orekit, what is first did on a release branche. For example:

    • The future version 9.3 will be published first of all on “release-9.3” branch, then on the “master” branch.

    • If a severe bug is detected on the version 9.2, the patched version (9.2.1) will be published on the “release-9.2” branch. But this version 9.2.1 will not be published on the “master” branch if the version 9.3 was published in the meantime.

So, I think that the “assembly:single” target must be played on all the “release-*” branches, not on the “master” branch.

Additionally, as some people asked us some kind of “nightly build” and as we need an on-line documentation synchronized with the source code, I think that the “assembly:single” target must be played on the “develop” branch.

Am I mistaken?

7

There is neither “else” nor “else when” instructions with the “when” directive. So, if you use such directive, you must declare two (alternative) stages:

stage('Build on master') {
    when {
        branch "master"
    }
    steps {
        ...
    }
}

stage('Build on other branch') {
    when {
        not { branch "master" }
    }
    steps {
        ...
    }
}

And the code inside the “steps” directives could be duplicated. I don’t like this approach.

8

More precisely, with the “when” directive, the code would be:

stage('Build on master') {
    when {
        branch "master"
    }
    steps {
        withEnv(["JAVA_HOME=${tool 'openjdk-8'}",
                 "PATH+MAVEN=${tool 'mvn-default'}/bin:${env.JAVA_HOME}/bin"]) {
            sh 'mvn verify assembly:single'
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
            archiveArtifacts artifacts: 'target/*.zip', fingerprint: true
            checkstyle pattern: 'target/checkstyle-result.xml'
            junit 'target/surefire-reports/*.xml'
            jacoco execPattern:'target/**.exec', classPattern: '**/classes', sourcePattern: '**/src/main/java'
        }
    }
}

stage('Build on other branch') {
    when {
        not { branch "master" }
    }
        withEnv(["JAVA_HOME=${tool 'openjdk-8'}",
                 "PATH+MAVEN=${tool 'mvn-default'}/bin:${env.JAVA_HOME}/bin"]) {
            sh 'mvn verify site'
        }
    }
    post {
        always {
            archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
            checkstyle pattern: 'target/checkstyle-result.xml'
            junit 'target/surefire-reports/*.xml'
            jacoco execPattern:'target/**.exec', classPattern: '**/classes', sourcePattern: '**/src/main/java'
        }
    }
}
9

A Jenkinsfile following this strategy was added in the “develop” branch of the Orekit repository.

10

I set up a job in Jenkins. You can see the result through:

At the moment, there is only a Jenkinsfile script in the “develop” branch. In anticipation of patch releases, it would be useful to copy this script in the “release-*” branches (at least).

Sébastien