Specification - GSoC Automated Build & Maven

From Gephi:Wiki

Jump to: navigation, search
This article is work in progress.



Contents

Introduction

This proposal describes implementation details of converting ant based gephi to a maven based build, automating release cycle via shell scripts and creating an automated build system. As mentioned in the Gephi GSoC 2011 wiki page, Netbeans Platform, which Gephi is built upon, hides a lot of complexity and can build in one step Windows executable, Mac OS X application and archives. Several more manual steps need to be done manually to achieve a complete release. And it is needed to focus on dealing with different platforms and installers, localization and update center. Initial objective of this project will be to evaluate the benefits and drawbacks of a move to Maven and implement it if its a good thing. Further goals include, researching for a more robust way to release the Toolkit replacing the current choice of using custom ANT targets. Next step will be identifying the best lightweight solution to achieve an automated build of the application and the creation of installers, from the large number of automated build solutions available for Java.

Student: Keheliya Gallaba

Mentor: Julian Bilcke

Proposal Details

List of deliverables

  • Report on feasibility/benefit analysis of migration to Maven from Ant
  • Set of scripts for the manual tasks when possible
  • A Maven repository for Gephi including Gephi toolkit deployment
  • Report on comparison of automated build solutions
  • Setting up of an automated build server
  • Updated documentation
  • Frequent blog posts about the progress

Quantifiable results for the Gephi project

  • Significant reduction in build time
  • An organized/automated build procedure
  • Quick notification of build failures via the automated build system
  • Positive feedback from developers
  • Increased downloads & usage from the community

Detailed description

Gephi is built on top of Netbeans platform, which is a generic framework for Swing Applications. Main characteristics of the Netbeans platform are

  • User interface framework
  • Data editor
  • Customization display
  • Wizard framework
  • Data systems
  • Internationalization
  • Help system

Following benefits are achieved by using a platform like that.

  • Reduction in development time
  • User interface consistency
  • Updating
  • Platform independence
  • Reusability and reliability

The runtime container is the minimal subset of modules that NetBeans Platform applications require. It consists of the following five modules:

  • Bootstrap: This module is executed before any other module. It carries out all registered command handlers and prepares a boot classloader, which loads the Startup module.
  • Startup: This module deploys the application, at which point it initializes the module system and the file system.
  • Module System: This module is responsible for the definition of modules, as well as their settings and dependencies.
  • File System: This module makes a virtual data system available, with platform-independent access. Primarily it is used to load module resources into an application.
  • Utilities: This module provides basic components, such as those required for intermodular communication.

Following diagram shows the dependencies between these five basic modules in the runtime container.

The Netbeans Runtime Container

Gephi contains features split into modules. Modules depend on each other. Plugins developers create new modules that contains their code, add dependencies to Gephi modules, and distribute their plugins by creating an NBM package. Following diagram shows a list of gephi modules categorised by their functionality.

A list of gephi modules categorised by their functionality.

Since Netbeans platform uses Apache Ant by default for building, Gephi also follows with a set of customized targets to compile, build and package. But since Netbeans Platform can now be built using Maven and it has grown significantly with documentation, this proposal considers the feasibility of moving gephi to a Maven-based build. This will allow deployment of Gephi and Gephi toolkit in local, public and central maven repositories. This projects goals include evaluating the benefits and drawbacks of a move to Maven and perform the task if it is beneficial in the long run. More details are included in the approach section of this document.

In Software Engineering it is advised to follow Continuous integration (i.e. by using an automated build system) for integrating early and often to avoid the pitfalls of integration hell. This will eventually reduce rework and thus reduce cost and time in Software development. Gephi currently does not have an automated build server. And it is mentioned in the documentation that since the updates can be deployed through the AutoUpdate system, there is no a critical need of a build server producing Nightly Builds. But on the other hand, although Netbeans Platform hides a lot of complexity and can build in one step Windows executable, Mac OS X application and archives several more manual steps need to be done manually to achieve a complete release. There is a need to deal with different platforms and installers, localization and update center. As mentioned in the release check list these manual tasks include

  • Updating App Version
  • Checking localization
  • Javadoc packaging
  • Removing unnecessary logging
  • Updating the Javadoc branding
  • Updating AutoUpdate URL
  • Setting visibility of modules
  • Adding keystore to sign NBMs
  • Creating and upload new NBMs

Some scripts could be written to speed-up the whole process of release by automating some of the tasks. Setting up an automated build server will make the development process more robust. Following best practises are followed in a typical Continuous Integration scenario.

  • Maintaining a code repository
  • Automating the build
  • Making the build self-testing
  • Everyone commits to the baseline every day
  • Every commit (to baseline) should be built
  • Keeping the build fast
  • Testing in a clone of the production environment
  • Making it easy to get the latest deliverables
  • Everyone can see the results of the latest build
  • Automating deployment

Implementation details will be discussed later in the Approach section of the document.

Approach

Phase 1

Netbeans Module m2 plugin can be used to create NetBeans module artifacts. It registers a new packaging type nbm. Any project with this packaging will be automatically turned into a netbeans module project. Additionally it allows to create clusters of modules, generate an autoupdate site content or build and assemble an application on top of NetBeans platform. The plugin is also capable of populating the local maven repository with module jars and NBM files from a given NetBeans installation. Since gephi is a Platform-based application with multiple sets of NetBeans modules, we can use a project with nbm-application packaging to perform the final application assembly. This packaging type (defined in nbm-maven-plugin) has module projects and all dependencies of the target NetBeans Platform included as dependencies. The nbm-application project/packaging defines a build lifecycle that creates a final application from the NBM files in local/remote repotories and bundles them in a ZIP file (also uploadable to the repository). In addition to that we can configure the project to generate an autoupdate site and webstartable binaries of the applications (typically in a deployment profile) Few more goals that will be useful

  • nbm:autoupdate
  • nbm:webstart-app
  • nbm:cluster
  • nbm:branding
  • nbm:run-ide
  • nbm:run-platform
Some important exceptions to focus on

There will be some special cases where special attention is needed

  • For example when creating localization modules using Maven bundling localized resources in all modules will not work, because these are usually searched for starting with the original module's ClassLoader, which cannot "see" all the other modules. In DevFaqMavenL10N it is suggested to use the existing nbm:branding goal, which despite its name can handle localization as well as branding.
  • System-scope dependencies, like provided scope, are never packaged with a project's artifacts. So a system scope for a library wrapper module can’t be used. MNBMODULE-120 has a discussion and alternative solutions for this.

Phase 2

An another important goal of this project is to identify the best lightweight solution to achieve an automated build of the application and the creation of installers. There are few possible options for continuous integration.

Apache Continuum
  • Provides good support for Apache Maven and Apache Ant.
  • Supports executing shell scripts
  • SCM via CVS, Subversion, Bazaar, Mercurial etc.
  • Written in Java
  • Cross-platform
  • Open Sourced under Apache 2.0 licence
  • can run standalone or the Continuum WAR can be deployed in a servlet container
  • For other features http://continuum.apache.org/features.html
Hudson / Jenkins (A fork of Hudson)
  • Good support for Apache Ant and Apache Maven based projects
  • supports SCM tools including CVS, Subversion, Git and Clearcase etc.
  • can execute shell scripts and Windows batch commands
  • Written in Java
  • Released under MIT license
Other options

Cabie

  • Written in perl
  • Cross-platform
  • Released under GNU/GPL

After comparing characteristics of above solutions, an automated build system can be implemented. It will be able to achieve the following major tasks.

  • Checking out source from source control
  • Running clean build
  • Deploying to integration server
  • Running unit tests

While completing all of the above tasks, I will explain to the members of the community how to use the new system, by updating the documentation and writing frequent blog posts.

Schedule

This is an approximated schedule that I would like to follow:

  • April 8 - April 25 : Community Bonding, Research on Build tools, Gephi code base
  • April 25 - May 16 : Researching & Reporting on feasibility/benefit analysis of migration to Maven from Ant. Start creating .POM files. Hosting some of the dependencies as Maven artifacts in a local repository.
  • May 16 - May 30 : Setting up a full Maven repository for Gephi including Gephi toolkit deployment / Writing shell scripts for the manual tasks when possible
  • May 30 - June 13 : Researching & Reporting on comparison of automated build solutions. Getting feed back from Developer community on extra features to implement in the build server. Eg: Emailing Dev Mailing lists about the success of daily/nightly builds
  • June 13 - July 15 : Setting up of an automated build server
  • July 15 - August 15 : Testing, Bug fixing, Updating documentation, Explaining to members of the community how to use the new system

These milestones may slightly vary since I’m hoping to get feedback from the Gephi community about the features to implement, before starting each small iteration.

Discussion/FAQs

What is your choice of the build server? 

From what I found out the best option would be Hudson. From NetBeans version 6.7, Hudson is integrated into the NetBeans IDE. So the hudson jobs can be natively managed from the IDE without opening the browser. Following links show the supported features.

A good example is Fabrizio Giudici of forceTen using Maven+Hudson for Continuous Integration of Swing and NetBeans Platform desktop applications successfully. He has posted some tutorials and screencasts.

Other options are using Jenkins or Apache Continuum. As pros we can see all of these solutions support

  • bazaar as SCM.
  • maven or Ant building
  • executing shell scripts
  • written in Java

So suiting Gephi very well. You can know more about the Hudson Vs Jenkins debate here. Since it's a fresh approach there might be Gephi specific aspects I'm missing out. So please enlighten me on those. I would love to research on them and refine this solution.

Our Windows installer uses InnoSetup, is it possible to make this work on a unix based server? 

Innosetup can be run on a linux server using wine. This blogpost explains how.

Another option is NSIS which has a linux compiler MakeNSIS which creates installers for Windows.

Launch4j and Installjammer are other alternatives for creating cross-platform Windows native EXEs out of Java executables.

References

Video Tutorials

Articles

Retrieved from "http://wiki.gephi.org/index.php/Specification_-_GSoC_Automated_Build_%26_Maven"
Personal tools