Continuous Integration

From Gephi:Wiki
Jump to: navigation, search

Overview

This page is about the Mavenization of Gephi and the creation of an Continuous_integration process at Gephi.

The aim of this project is to automate as much things as possible in the daily development process, as well as the release process. Tools and techniques exists to automate building, testing and releasing code and this project is about implementing them as tuning them for our needs. It's currently difficult to release a new version of Gephi and make sure everything works as expected. The process is semi-automated but still rely on manual tasks, editing a set of configuration files and build archives. It's also difficult to easily test the current development version.

Objectives

  • Convert the existing ant-based Netbeans Platform application to Maven.
  • Build, test and release in one Maven command. That includes localization, NBMs and installers.
  • Deploy a continuous integration serveur (Jenkins) and wire it to our code repository.
  • Produce nighly builds binaries and NBMs.
  • Publish Gephi artifacts on Maven Central.
  • Convert the Gephi Toolkit project.
  • Adapt plugin development to Maven.
  • Write documentation.

Mavenization of Gephi

Maven logo.png

Gephi is a Netbeans Platform application with more than 80 modules. The Netbeans Platform is originally ant-only but a Maven support have been added in the last few years. In general, Netbeans IDE supports Maven very well and recent versions have made things really easy. Documentation has also improved, thanks to pioneers like Fabrizio. Thanks to him and the rest of the Netbeans Platform team we can transition the application from Ant to Maven.

The Netbeans Platform Maven support is done using only traditional Maven features. Whereas the Ant implementation relied on a lot of custom platform scripts, the Maven implementation follows the standard multi-module Maven projects and relies on a single Maven plugin: nbm-maven. In other words a Netbeans Platform application built with Maven can be built anywhere, with or without Netbeans. The ant platform made a difference between the platform modules (Lookup, Utilities, ...) and the modules of your application. Maven simplifies that a lot and all dependencies are traditional Maven dependencies. It also simplifies reusing modules between projects.

We can sum up the pros for Netbeans Platform on Maven:

  • Fits into the standard Maven model. Works on Eclipse or command-line out of the box.
  • All JARs are dependencies to Maven Central or 3rd party repositories. No library JARs are directly included in the source codebase.
  • Facilitate reuse of modules in other projects.
  • Easier to switch to a different version of the Netbeans Platform.
  • Large number of Maven plugins to customize things.
  • In development, the application is booting instantaneously when already built.

Some drawbacks as well:

  • It takes longer to build compared to Ant
  • Netbeans GUI is not as good for Maven-based Netbeans Platform application.
  • Learning curve a little bit steeper.
  • Netbeans don't discover automatically what modules should be rebuilt.

One step release

Releasing a new version of Gephi needs to happen every so often and the process should be 100% automated. The current ant-based scripts take care of a few things and the rest is done manually. Roughly for every release we need to:

  • Integrate the localization files and modules
  • Integrate configuration files
  • Build the zip, dmg (Mac OS X) and tar.gz
  • Build the Windows installer
  • Create the Javadoc and sources archive
  • Sign and create the NBMS
  • Upload the binaries, NBMs and Javadoc

Maven supports all these tasks through various plugins and configurations settings. Moreover it's possible to run Ant scripts with the antrun Maven plugin if necessary.

Continuous integration

Jenkins

Jenkins logo.png

Jenkins is the leading continuous integration system and can help us achieve a few things:

  • Automated builds every day or at every push on GitHub
  • Automate unit tests

Resources:

Nexus

Nexus200x50.png

Jenkins works well with Nexus to store our artifacts. Nexus is a repository for artifacts, which could either be librairies Gephi is using or release binaires.

  • Nexus stores all our JARs. No more needed to add JARs manually.
  • Good place to store releases and distribute NBMs.

Resources:

Nighly builds

Jenkins can easily run daily and produce a version of Gephi based on the last version of the code. Binaries and installers can be available to be tested at any time. Nighly builds will help us to test things earlier and at a larger scale.

Along with binaries it is also possible to release a new set of NBMs every day and push them to our update center automatically. NBMs are Netbeans packages and the way updates in Gephi currently work. Every so often, Gephi clients check our NBM repository to see if new versions are available. We could easily distribute a 'testing' version of Gephi which would check every day the NBM repository. Beta testers could therefore test something a day after it has been pushed just by updating their Gephi.

Gephi Toolkit

The Toolkit is based entirely on the main Gephi source code and is a subset of the Gephi modules. Because each module would be built by Maven and artifacts produces, it would very easy to create a POM for the toolkit that depends on the core modules.

Plugins development

TBD

Converting Gephi to Maven

This is a multi-step process with no ability to revert. The structure of the project should be quite different after the migration.

File structure

Module

The Maven structure is a bit different but simplified with no more Netbeans-only files. An important difference with Maven is the separation between the source files (.java) and the resources (Icons, Bundle.properties...). They are in separate folders.

Ant Maven
DataLaboratoryPlugin (Name of the module without spaces)
  • build.xml (Ant build script, shouldn't be touched)
  • manifest.mf (Module version and configuration)
  • nbproject
    • build-impl.xml (Ant build script, shouldn't be modified)
    • genfiles.properties (Used by Netbeans, shouldn't be modified)
    • project.properties (Module metadata like author or license)
    • project.xml (Dependencies and public-packages go here)
    • suite.properties (Generated by Netbeans, shouldn't be modified)
  • private (Private folder where Netbeans caches some stuff)
  • src (Sources and resources)
DataLaboratoryPlugin(Name of the module without spaces)
  • pom.xml (Configuration, dependencies, public-packages)
  • src
    • main
      • java (Java sources only)
      • resources (All Bundle.properties, icons or other resources files)
      • nbm
        • manifest.mf (Module version and configuration)
        • module.xml (Module metadata like author or license)

Project structure

The project structure is also different. Maven is always configured through pom.xml files and each module or project has its pom. The pom files inherit each other and Maven have concepts like superpom to reuse configuration as much as possible. For a Netbeans Platform application, there is pom.xmlat the root at the project and one for each module. The pattern here is to configure things in the pom at the root folder, which we can call the parent pom and put as little things as possible in each module. Except dependencies and public-packages there shouldn't really be anything that is special from one module to another. Configuration about compilation, packaging and so on can be defined in the parent pom. All properties defined in the parent pom are accessible in each module's pom.

The way the application is configured is different with Maven. Instead of a nbproject folder there is an application module with its pom and configuration files. Similarly, there is brandingmodule which contains everything about the application's branding.

Ant Maven
gephi
  • Algorithms (module)
  • ... (module)
  • WorkspaceUI (module)
  • nbproject
    • build-impl.xml (Netbeans file)
    • genfiles.properties (Netbeans files)
    • gephi.conf (Configuration file used for releases)
    • Info.plist (Mac OS X application bundle configuration)
    • platform.properties (Netbeans Platform version and configuration)
    • platform.xml (Ant target to update the platform)
    • private (Private folder used by Netbeans)
    • project.properties (Project name and enabled modules)
    • project.xml (Netbeans file)
  • branding (Icons and branding resources)
gephi
  • modules
    • AlgorithmsPlugin (module)
    • application (Main application module)
      • src
        • main
          • app-resources
            • gephi.icns (Icon)
            • gephi.ico (Icon)
            • gephi.iss (InnoSetup configuration)
            • gephi48.png (Icon)
            • gephifile128.png (Icon)
            • Info.plist (Mac OS X app bundle)
    • ... (module)
    • WorkspaceUI (module)
  • pom.xml (Parent POM file)
  • src
    • assemble (Assembly configuration files)
      • javadoc.xml (Assembly configuration for Javadoc)
      • sources.xml (Assembly configuration for Sources)
    • keystore (Keystore folder)
      • keystore.ks (Keystore file)
    • main
      • javadoc
        • overview.html (Overview HTML)
        • stylesheet.css (Javadoc CSS)

Application layout

Technically speaking modules are independent and you can go to each module folder and create a JAR, ZIP or NBM of this module using Maven. The complete application is defined as a set of modules using the reactor plugin. The parent pom simply defines the list of modules it want to have included. In the case one would like to build a 'light' version of Gephi which uses just some modules, a new parent pom would be created and the relevant set of modules listed.

When asked to build the complete application, Maven builds a dependency tree using the project dependencies (the modules) and the dependencies of these modules (other modules or librairies). Cyclic dependencies aren't allowed and Maven will resolve dependency versions issues in the case two modules depend on a different version of another modules.

When opening the mavenized Gephi project, there are three important project opening:

  • gephi-parent: This is the parent project defined by the pom.xml at the root of the project. All the other modules including application and branding are listed as child modules.
  • gephi-application: This is the Netbeans Platform application module. It contains the configuration for the complete application and list all modules as dependencies.
  • gephi-branding: The module which contains all the branding elements.

The NetBeans Platform Quick Start Using Maven tutorial gives a bit more details:

  • app: The app project enumerates included modules; permits interactive runs; produces various kinds of packaging, such as ZIP by default, but optionally JNLP, NBMs, and in the future perhaps OSGi; holds functional tests. Each module project specifies its own compile dependencies, while the app project's dependencies are anything additional that should be present at runtime. For example, that includes, by default, the whole platform cluster, that is, the set of modules constituting the "platform" cluster. The app project could include other clusters, or subsets of clusters, or whatever plugins you want included in your app which are not used as compilation dependencies. At a minimum, core.startup and its transitive dependencies are included.
  • branding: This project contains the global resources used for branding the application, such as the splash screen.
  • parent: This project is a Maven reactor project for the NetBeans Platform application, which lists the modules to include and the location of the project's repositories. This project does not contain any sources. The IDE generates the modules containing the sources and resources in sub-directories of this project.

Dependencies

Maven has great dependency management system. Dependencies are declared in the pom and look like this:

<dependency>
   <groupId>com.sun.mail</groupId>
   <artifactId>javax.mail</artifactId>
   <version>1.4.5</version>
</dependency>

To be resolved, dependencies should be present in any of the configured repository or Maven Central.

These are the repository we use:

   <repositories>
       <repository>
           <id>netbeans</id>
           <name>NetBeans</name>
           <url>http://bits.netbeans.org/maven2/</url>
       </repository>
       <repository>
           <id>gephi</id>
           <name>Gephi 3rd Party</name>
           <url>http://nexus.gephi.org/nexus/content/repositories/thirdparty/</url>
       </repository>
       <repository>
           <url>http://download.java.net/maven/2/</url>
           <id>beans-binding</id>
           <layout>default</layout>
           <name>Repository for library Library[beans-binding]</name>
       </repository>
   </repositories>

Dependencies are transitive so if module A depends on module B and C depends on A, C will automatically depend on B as well. Note that dependencies of the type 'nbm' (the Netbeans modules) are not transitive.

Like ant-based projects, Maven-based projects still use the concept of public packages and you can't depend on a module's class which is not defined in public packages. For instance if the module CollectionUtils depends on the fastutil library and the Graph module depends on CollectionUtils it requires to put fastutil's packages in CollectionUtils' public packages to allow Graph to use them. Building will fail with a Private classes referenced from module otherwise.

For Gephi, there are three types of dependencies: libraries, Netbeans Platform module and gephi module.

A Netbeans dependency has the current netbeans.version value defined in the parent pom and a nbm type. As nbm dependency are not transitive all netbeans modules should be explicitely added as dependencies. This is not different compared to the ant-based projects but differs from the traditional Maven way.

   <dependency>
       <groupId>org.netbeans.api</groupId>
       <artifactId>org-openide-util-lookup</artifactId>
       <version>${netbeans.version}</version>
   </dependency>

A module to module dependency is straightforward and reuse the current group id and project version.

   <dependency>
       <groupId>${project.groupId}</groupId>
       <artifactId>graph-api</artifactId>
        <version>${project.version}</version>
   </dependency>

Libraries

Like most of large Java projects, Gephi uses libraries. Ant-based Netbeans Platform projects had a special type of modules called "Library Wrapper" designed to expose a library's packages to other modules. For instance the Batik Wrapper would include Batik's JARs and add their packages as public packages. Maven modules add direct dependencies to libraries and no JARs are directly copy-pasted in the repository. All JARs should be online and Maven will download them. Each artifact has a group.id, artifact.id and version.

However not all libraries we are using are available as Maven artifacts on Maven Central. Therefore we created artifacts on our Nexus 3rd party repository. Nexus has an interface to upload JARs and create artifacts given the group.id, artifact.id and version.

With Maven the concept of a library wrapper still exist as it is still required to expose packages as public to be able to reuse the library's classes in dependent modules. For that reason we created the core-library-wrapper and ui-library-wrapper modules which contains all the librairies so core and ui modules can add only one dependency for all librairies. Though it is not entirely modular it avoids to duplicate the same library dependency in different modules and make sure we don't depend on two different versions of the same library.

Build and release in one click

We'll review some of the features added to the POM to support painless release process. It's useful to review what Maven is, the getting-started and the [1] POM reference]. If you don't have Maven installed (type mvn -version to find out), follow the instructions. Linux packages are available as well.

Everything is mostly defined in two different POMs:

  • Parent pom (pom.xml)
  • Application pom (modules/application/pom.xml)

How Maven works

Maven build lifecycle works with phases, namely:

  • validate - validate the project is correct and all necessary information is available
  • compile - compile the source code of the project
  • test - test the compiled source code using a suitable unit testing framework. These tests should not require the code be packaged or deployed
  • package - take the compiled code and package it in its distributable format, such as a JAR.
  • integration-test - process and deploy the package if necessary into an environment where integration tests can be run
  • verify - run any checks to verify the package is valid and meets quality criteria
  • install - install the package into the local repository, for use as a dependency in other projects locally
  • deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects.

This is the default model the pom.xml is customizing. Each phase is then divided into goals which are basically tasks. A goal is bound to a phase so it will be executed during that phase only. Every time we're configuring something in the POM it is done for a particular phase and goal. This is also how plugins are executed. For instance, if I want the plugin A to be executed during the packaging phase I can bind its configuration to the phase package and the goal jar.

Parent POM

The parent POM's role is to defined what is common to every module including application and branding. Version numbers, repository URLs and general compilation parameters lie there.

The main sections are:

  • properties: A list of properties to configure different things. We want to avoid to have configuration values everywhere in the POM so all configuration happen here.
  • repositories: The list of the repositories we're using. These are locations we can download artifacts.
  • distributionManagement: The list of repositories we can upload artifacts to. These are our Nexus repositories. One for snapshots (development versions) and releases.
  • pluginManagement: A list of all the Maven plugins we're using.
  • plugins: This is the section each plugin is configured.
  • modules: The list of all gephi modules.

NBM Plugin

The nbm-maven-plugin is an important element of the build process. Its main goal is to create the NBM file for every module and the NBM application for the application module. All the concepts of the Netbeans Platform and the custom build logic the ant-based platform had are packaged into that plugin.

Notably it has goals for:

  • nbm:autoupdate: Create the autoupdate XML file along with all the NBMs.
  • nbm:build-installers: Build the installers
  • nbm:nbm: Build the nbm
  • nbm:run-platform: Run the application in development mode.

Application POM

The application POM is located in the modules/application folder and its role is to configure the application and what artifacts are produced. This is the location where localization is configured, as well as the installers and the Mac OS X application bundle.

When building and packaging gephi, the application module is the last module to be built as it has dependencies to all the other modules. It is responsible for producing final release artifacts such as binaries, installers and autoupdate folder. For example, for the 0.8.2 version, it produces the following artifact:

  • gephi-0.8.2-SNAPSHOT.dmg (Mac OS X application)
  • gephi-0.8.2-SNAPSHOT.zip (Zip)

Development version

One of the main motivation to deploy a continuous integration environment is testing. Our aim is to improve the quality of the releases and increase the number of beta-testers.

Jenkins is configured to create nightly build artifacts. In other words the complete application is compiled, build and deployed to Nexus every night. In the Maven world this is called a SNAPSHOT. A version is suffixed with SNAPSHOT when in development. For instance 0.8.2-SNAPSHOT is the development version which will end with the release of the 0.8.2 version. Releases are not prefixed or suffixed.

The autoupdate site is the set of NBMs files Gephi is using to get updates. Every Gephi release has its set of NBMs files, one per module. Updating NBMs is the easiest way to deliver updated modules to clients. This is the mechanism we're using to keep beta-testers with the latest development version. Instead of downloading and installing the nightly build, testers can update modules from the NBMs. Clients normally acknowledge a new version of a module if the specification version is incremented. The parent POM has been customized to override the specification version of all modules so it can be incremented at will. By default, the specification version is the project version and is supposed to remain the same during development (snapshot phase).

The application POM is configured to prepare and upload the entire NBM repository to Nexus.

Building Gephi

Development

The development version is the version used by developers when working on the code. Typically the developer has the project opened in Netbeans and is working on some modules and need to test the software. The startup time should be as quick as possible and must include all latest changes.

Netbeans automatically recognizes the platform project and will properly run Gephi when asked to 'Run' the project. However the project needs to be built before. After every change to the code the project needs to be rebuilt to see the changes when run. This is not done automatically when run.

Running the application (need to build first)

Command line in the modules/application folder:

mvn nbm:run-platform

Building and running the application'

Command line at the root of the project:

mvn --also-make --projects modules/application package

Nightly Build

The nightly build produces the most recent snapshot artifacts and upload them to nexus. In addition of the jar and nbm, the sources and javadoc are also produced for each module. The application modules produces a zip and dmg file.

For instance, for the 0.8.2 version, in the gephi-parent module:

  • gephi-0.8.2-SNAPSHOT-sources.zip (Sources)
  • gephi-0.8.2-SNAPSHOT-javadoc.zip (Javadoc)

In the gephi-application module:

  • gephi-0.8.2-SNAPSHOT.dmg (Mac OS X)
  • gephi-0.8.2-SNAPSHOT.zip (Zip)

In every other module (ex: RankingAPI):

  • ranking-api-0.8.2-SNAPSHOT-javadoc.jar (Javadoc)
  • ranking-api-0.8.2-SNAPSHOT-sources.jar (Sources)
  • ranking-api-0.8.2-SNAPSHOT.jar (Jar)
  • ranking-api-0.8.2-SNAPSHOT.nbm (Nbm)

The update center files are also updated so testing version so Gephi can be updated.

Build and deploy artifacts

mvn clean deploy -P deployment,create-dmg,deploy-dmg,upload-updates -Dkeystore.password=XXX

The NBMs files are signed thanks to the keyword password.

Sources and Javadoc'

Custom goals are used to aggregate and deploy the Javadoc and Sources.

mvn javadoc:aggregate assembly:single deploy:deploy-file wagon:upload -P export-javadoc
mvn assembly:single deploy:deploy-file -P export-sources

Latest artifacts

Links for latest artifacts:

Release

The release process is similar to the nightly build, except some locations and parameters. The most notable difference is the separation between the linux, windows and mac os releases. Because the Windows release needs to be done on a Windows computer (due to the installer) and the Mac OS X release done on a Mac OS X computer (due to the signing), custom profiles exists for each platform.

Update versions'

The development versions are tagged with SNAPSHOT. For a release, this is removed and only the numerical versions remains. We use the versions plugin to automatically set the version on the complete project, including sub-modules.

mvn versions:set -DnewVersion=0.8.2
mvn versions:commit

Build and deploy artifacts

mvn clean deploy -P deployment,release,upload-updates -Dkeystore.password=XXX

Sources and Javadoc

mvn javadoc:aggregate assembly:single deploy:deploy-file wagon:upload -P export-javadoc,release
mvn assembly:single deploy:deploy-file -P export-sources,release

Windows

mvn clean deploy -P deployment,release,release-extra,release-windows -Dkeystore.password=XXX

Mac OS X

mvn clean deploy -P deployment,release,release-extra,release-macos,create-dmg -Dkeystore.password=XXX

Resources