Release management

This page describes how to create and deploy the SIS Maven artifacts, binary bundle, javadoc and list of API changes. The Release FAQ page describes the foundation wide policies. The instructions on this page provide a summary suitable to SIS releases, together with SIS-specific steps. The intended audiences are SIS release managers.

Configure

Before to perform a release, make sure that the following conditions hold:

  • Commands will be executed in a Unix shell.
  • Subversion, GNU GPG, ZIP, Maven, Ant, Java and the Java compiler are available on the path.
  • The release management setup steps have been executed once.

For all instructions in this page, $OLD_VERSION and $NEW_VERSION stand for the version number of the previous and the new release respectively, and $RELEASE_CANDIDATE stands for the current release attempt. Those versions shall be set on the command line like below (Unix):

export OLD_VERSION=0.8
export NEW_VERSION=1.0
export RELEASE_CANDIDATE=1
export SIGNING_KEY=<your key ID>
export BOOTCLASSPATH=<path to the rt.jar file of a JDK7 installation>

The key ID value is an hexadecimal numbers with 8 digits (may be the last 8 digits of a 40 digits number). It can be found be executing gpg --list-keys.

The boot class path value is highly system-dependent. Some examples are:

  • /usr/lib/jvm/java-7/jre/lib/rt.jar on some Linux distributions (replace java-7 by the actual directory name).
  • /Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre/lib/rt.jar on MacOS.

Starts the GPG agent if not already done. This is for avoiding to be prompted many time for the passphrase in every modules to be built by Maven.

eval $(gpg-agent --daemon)

Directory layout

The steps described in this page assume the following directory layout (some directories will be created as a result of the steps). Any other layout can be used. However if the layout differs, then the relative paths in this page shall be adjusted accordingly.

<any root directory for SIS>
├─ master
├─ branches
│  └─ $NEW_VERSION
├─ tags
│  └─ $NEW_VERSION
│     └─ target
│        └─ distributions
├─ non-free
│  └─ sis-epsg
├─ distribution
│  └─ $NEW_VERSION
│     └─ RC$RELEASE_CANDIDATE
└─ site
   └─ content
      └─ apidocs

Java versions

While we need a JDK7 installation for the bootclasspath option, we do not recommend to use it for generating the binaries since JDK7 has reached its end of public updates (but not the end of commercial support). We recommend to create the release with JDK8 instead (note however that JDK8 javadoc tool seems to have a bug that prevent its use with SIS). Make sure that the JDK installation to be used for the release has been updated recently, since compilers may receive critical bug fixes. For example the javadoc tool of older JDK versions was known to generate HTML pages with a vulnerability in them.

Prepare web site and source code before branching

Before to start the release process, we need to test more extensively the master. The tests described below often reveal errors that were unnoticed in daily builds. It is better to detect and fix them before to create the branch. First:

  • Review and update the README and NOTICE files on the source code repository.
  • Review the javac and javadoc warnings reported by Jenkins and fix at least the typos.

Prepare release notes

We update JIRA first because doing so is sometime a reminder of uncompleted tasks in source code. Update JIRA tasks and prepare release notes as below:

  • Ensure that the Fix Version in issues resolved since the last release includes this release version correctly.
  • Ensure that all open issues are resolved or closed before proceeding further.
  • On the site source code repository, create a content/release-notes/$NEW_VERSION.html file with all the features added.
  • Use content/release-notes/$OLD_VERSION.html as a template, omitting everything between the <body> and </body> tags.
  • The release notes can be obtained from JIRA, by clicking on the Roadmap tab → the version number → Release notes and then configuring the release notes to display HTML format and copying it. A suggested approach would be to reorganize the release notes as New Features, then Improvements, then Bug Fixes, then Tests and finally Tasks. The Sub Tasks can be classified according the category of their parent issue.

Web site

Review and update the content/DOAP.rdf file on the site source code repository. Add a new <release> block for the new release with the estimated release date.

Update the version numbers from the old one to $NEW_VERSION in the following files.

  • content/index.mdtext
  • content/gdal.mdtext
  • content/downloads.mdtext (need also to update $NEW_VERSION-SNAPSHOT to the next snapshot version)
  • content/command-line.mdtext
  • content/epsg.mdtext (be aware that the version number may sometime be behind the SIS version number)

Commit:

svn add content/release-notes/$NEW_VERSION.html
svn commit --message "Prepare documentation for the $NEW_VERSION release."

The new web site will be published in the staging area. It will not yet be published on http://sis.apache.org.

Test master extensively

Build the project with the apache-release profile enabled. This profile performs the following actions:

  • Enable extensive tests (i.e. it will set the org.apache.sis.test.extensive property to true).
  • Generate Javadoc. This may fail if the source code contains invalid Javadoc tags or broken HTML.
  • Generate additional binary artifacts (*.pack.gz and *.oxt files). This will fail if duplicated class files or resources are found. Consequently building the .*pack.gz file is an additional test worth to do before deployment.
  • Sign the artifacts.

Each of those additional products may cause a failure that did not happen in normal builds.

cd ../master
mvn clean install --activate-profiles apache-release --define bootclasspath=$BOOTCLASSPATH

Verify signature for all files:

find . -name "sis-*.asc" -exec gpg --verify '{}' \;

If the SIS_DATA environment variable was set during above build, unset it a try again. Ideally the build should be tested in both conditions (SIS_DATA set and unset).

Build the non-free group of modules. Those modules will not be part of the distribution (except on Maven), but we nevertheless need to ensure that they work. Before to test, verify that their version numbers and their SIS dependencies are the snapshot tested in this section.

cd ../non-free
svn update
mvn clean install --define bootclasspath=$BOOTCLASSPATH

Run the integration tests. Before to test, verify that their version numbers and their SIS dependencies are the snapshot tested in this section. Note that execution of those tests may be slow.

cd ../release-test
svn update
cd maven
mvn clean test --define bootclasspath=$BOOTCLASSPATH

Source code

Replace the $OLD_VERSION number by $NEW_VERSION in the following literals. Those changes need to be applied on the development branch and merged with all other branches and master.

  • DOWNLOAD_URL in application/sis-console/src/main/java/org/apache/sis/console/ResourcesDownloader.java file.
  • <sis.non-free.version> in root pom.xml file.

Commit and merge to other branches and to master.

git commit --message "Set the EPSG geodetic dataset URL to its expected location after release."
git checkout geoapi-3.1
git merge
# Repeat the above commit and the merge commands for other branches until the changes are merged to master.

Create branch and tag

Execute the following command:

git checkout -b $NEW_VERSION

It is okay to checkout the branch in a separated directory if desired. The BRANCH_DIR environment variable will specify that directory.

export BRANCH_DIR=`pwd`

We need to update the SIS version numbers not only in the pom.xml files, but also in a few Java files. The following command performs the replacement using Ant. Note that this command does not yet remove the -SNAPSHOT suffix in version numbers (this will be done later, at tag creation time):

ant -buildfile core/sis-build-helper/src/main/ant/prepare-release.xml branch -Dbranch.version=$NEW_VERSION -Dsis.version=$NEW_VERSION

Open the root pom.xml file in an editor and perform the following changes:

  • Remove the whole <pluginRepositories> block (including comment), since it should not be needed for releases (and is actually not allowed).

Remove the modules that are not yet ready for a release. This may require removing <module> elements in the parent pom.xml file.

git rm storage/sis-shapefile
git rm application/sis-javafx

Validate with git diff, ensure that the build is not broken (we will run the tests later), then commit the changes on the branch:

mvn clean package --define skipTests=true --define bootclasspath=$BOOTCLASSPATH
git commit --message "Prepare branch for $NEW_VERSION release."

Tag the branch

In theory, the next commands would be a real (non-dry) run of mvn release:prepare, followed by mvn release:perform. However we perform those steps manually rather than relying on Maven for the following reasons:

  • Perform some additional changes on the tag.
  • Give more opportunities to take action before commit in case of problems.
  • We need a separated install phase first, for proper installation of sis-build-helper plugin.

First, create the tag:

git tag $NEW_VERSION

It is okay to checkout the tag in a separated directory if desired. The TAG_DIR environment variable will specify that directory.

export TAG_DIR=`pwd`

Update versions number in pom.xml files (like what mvn release:prepare would have done), plus some additional files:

ant -buildfile core/sis-build-helper/src/main/ant/prepare-release.xml tag -Dbranch.version=$NEW_VERSION -Dsis.version=$NEW_VERSION

Validate with git diff, search SNAPSHOT in the whole source directory in case we missed some, then commit:

git commit --message "Fix version number to $NEW_VERSION (no -SNAPSHOT)."

Verify content

Build and test the project in the same way than we did on master. In addition to testing, execution of an install phase before deployment is required for our custom sis-build-helper plugin.

mvn install --activate-profiles apache-release --define bootclasspath=$BOOTCLASSPATH
find . -name "sis-*.asc" -exec gpg --verify '{}' \;

Open the pom.xml file of all modules in the non-free group. Set their version numbers and their SIS dependencies to $NEW_VERSION without -SNAPSHOT suffix. Commit and built:

cd ../../non-free
svn commit --message "Set version number and dependencies to $NEW_VERSION."
mvn clean install --define bootclasspath=$BOOTCLASSPATH

Open the pom.xml file of integration tests. Set its version numbers and its SIS dependencies to $NEW_VERSION without -SNAPSHOT suffix. Commit and built (note that execution may be slow).

cd ../release-test
svn commit --message "Set version number and dependencies to $NEW_VERSION."
cd maven
mvn clean test --define bootclasspath=$BOOTCLASSPATH

Deploy Maven artifacts

If above verifications succeeded, deploy:

chmod a-w target/distribution/*.zip
mvn deploy --activate-profiles apache-release --define bootclasspath=$BOOTCLASSPATH

Verify and close the Nexus release artifacts

Verify the staged artifacts in the Nexus repository. The artifacts can be found under Build PromotionStaging repositories, and searching for org.apache.sis in the Repository column. Navigate through the artifact tree and make sure that all javadoc, source, test and jar files have .asc (GPG signature) and .md5 files.

Additional references about signatures:

Select any *-javadoc.jar file and click on the Archive Browser tab on the right side. Select any *.html file which is known to use some of the custom taglets defined in the sis-build-helper module. Click on that file and verify that the custom elements are rendered properly. In particular, all Java code snippets are missing if the @preformat taglet had not be properly registered, so try to see at least one code snippet.

Additional cleaning:

  • Delete all org/apache/sis/parent/$NEW_VERSION/parent-$NEW_VERSION-source-release.zip.* files on the Nexus repository. They should not be there - source release will be deployed on an other repository later.

Close the Nexus staging repository:

  • Click the checkboxes for the open staging repositories (org.apache.sis-<id>) and press Close in the menu bar.
  • In the description field, enter "SIS $NEW_VERSION-RC$RELEASE_CANDIDATE" (replace $NEW_VERSION and $RELEASE_CANDIDATE by appropriate values. This will not be done automatically since this field box is not our bash shell!).
  • Click on the org.apache.sis-<id> link in order to get the URL to the temporary Maven repository created by Nexus.

We will announce later (in the Put the release candidate up for a vote section) on the dev@ mailing list the availability of this temporary repository for testing purpose.

Deploy non-free resources to Nexus

We deploy the non-free resources in a separated Nexus repository for allowing deployment of those resources to Maven Central before deployment of the rest of Apache SIS. This allows to test that the URL to those resources in Apache SIS are correct.

Go to the directory that contains a checkout of https://svn.apache.org/repos/asf/sis/data/non-free/sis-epsg. Replace occurrences of <version>$OLD_VERSION</version> by <version>$NEW_VERSION</version> in the root pom.xml. Deploy:

mvn deploy --activate-profiles apache-release --define bootclasspath=$BOOTCLASSPATH

In Nexus:

  • Delete all *-source-release.zip.* files since they duplicate the *-source.zip.* files.
  • In the sis-epsg-$NEW_VERSION.jar file, verify that META-INF/LICENSE contain the EPSG terms of use.
  • Close the repository and take note of its URL.

Test the Nexus release artifacts

Go to the test Maven project. Open the root pom.xml file and set the <version> number to the SIS release to be tested. Then go to the <url> declaration of the first <repository> and replace value by the URL of the temporary Maven repository created by Nexus. Usually, only the 3 last digits need to be updated.

cd ../../release-test/maven
vim pom.xml
mvn compile
svn commit -m "Test project for SIS $NEW_VERSION-RC$RELEASE_CANDIDATE"

Create a temporary directory where Apache SIS will write the EPSG dataset.

mkdir target/SpatialMetadata
export SIS_DATA=`pwd`/target/SpatialMetadata

Clear the local Maven repository in order to force downloads from the Nexus repository, then test. This will also verify the checksums.

rm -r ~/.m2/repository/org/apache/sis
mvn package --strict-checksums

Verify that the EPSG dataset has been created, then cleanup:

ll $SIS_DATA
mvn clean

Stage the source, binary and javadoc artifacts

Generate the Javadoc. While not mandatory, we suggest to use JDK 8 or above for getting the new look and feel, getting the Javadoc enhancements expected in future JDK releases (more dynamic pages), avoiding the security vulnerability discovered in the Javadoc tools of older JDK releases, and keeping the diff smaller on the SVN repository of SIS web site. If JDK8 is not used, then omit the cp command below.

cd $TAG_DIR
mvn javadoc:aggregate
cd target/site
zip -9 -r apache-sis-$NEW_VERSION-doc.zip apidocs
cd ../..

Note: if Javadoc fails because of JDK-8061305 bug, a workaround is to temporarily copy the OpenGIS annotations into the sis-metadata module (only the time needed for building the javadoc):

mkdir core/sis-metadata/src/main/java/org/opengis
cd core/sis-metadata/src/main/java/org/opengis
ln -s <path to GeoAPI project>/geoapi/src/main/java/org/opengis/annotation
cd -

Initialize the distribution directory

Create the directory for the new version and release candidate within the distribution directory. The $RELEASE_CANDIDATE variable shall be the number of current release attempt.

cd ../../distribution
svn update
mkdir -p $NEW_VERSION/RC$RELEASE_CANDIDATE
svn add $NEW_VERSION
cd $NEW_VERSION/RC$RELEASE_CANDIDATE
export DIST_DIR=`pwd`

Copy the HEADER.html file from the previous release. Update the file content if necessary.

svn copy https://dist.apache.org/repos/dist/release/sis/$OLD_VERSION/HEADER.html .

Move the files generated by Maven to the distribution directory:

mv $TAG_DIR/target/sis-$NEW_VERSION-* .
mv $TAG_DIR/target/site/apache-sis-$NEW_VERSION-* .
mv $TAG_DIR/application/sis-console/target/distribution/apache-sis-$NEW_VERSION.zip .

Rename the files to something more conform to the convention seen in other Apache projects:

mv apache-sis-$NEW_VERSION.zip             apache-sis-$NEW_VERSION-bin.zip
mv sis-$NEW_VERSION-source-release.zip     apache-sis-$NEW_VERSION-src.zip
mv sis-$NEW_VERSION-source-release.zip.asc apache-sis-$NEW_VERSION-src.zip.asc

Sign and commit

Sign the source, javadoc and binary artifacts:

shasum --algorithm 512 apache-sis-$NEW_VERSION-src.zip > apache-sis-$NEW_VERSION-src.zip.sha
md5sum                 apache-sis-$NEW_VERSION-src.zip > apache-sis-$NEW_VERSION-src.zip.md5

gpg --use-agent --armor --detach-sign --default-key $SIGNING_KEY apache-sis-$NEW_VERSION-doc.zip
shasum --algorithm 512 apache-sis-$NEW_VERSION-doc.zip > apache-sis-$NEW_VERSION-doc.zip.sha
md5sum                 apache-sis-$NEW_VERSION-doc.zip > apache-sis-$NEW_VERSION-doc.zip.md5

gpg --use-agent --armor --detach-sign --default-key $SIGNING_KEY apache-sis-$NEW_VERSION-bin.zip
shasum --algorithm 512 apache-sis-$NEW_VERSION-bin.zip > apache-sis-$NEW_VERSION-bin.zip.sha
md5sum                 apache-sis-$NEW_VERSION-bin.zip > apache-sis-$NEW_VERSION-bin.zip.md5

Verify checksums and signatures:

find . -name "*.md5" -exec md5sum --check '{}' \;
find . -name "*.sha" -exec shasum --check '{}' \;
find . -name "*.asc" -exec gpg --verify '{}' \;

Commit:

svn add apache-sis-$NEW_VERSION-*
cd ../..
svn commit --message "SIS release candidate $RELEASE_CANDIDATE"

Update online Javadoc

Copy the Javadoc to the web site staging directory:

cd ../site/content
rm -r apidocs
unzip $DIST_DIR/apache-sis-$NEW_VERSION-doc.zip

Execute svn add for new files and svn remove for deleted files:

cd apidocs
svn status | gawk '/^\?.*/ {print $2}' | xargs svn add
svn status | gawk '/^\!.*/ {print $2}' | xargs svn remove
svn commit --message "Update javadoc for SIS $NEW_RELEASE."

Test the release

Execute the following commands in any temporary directory for testing the sources:

wget --no-check-certificate https://dist.apache.org/repos/dist/dev/sis/$NEW_VERSION/RC$RELEASE_CANDIDATE/apache-sis-$NEW_VERSION-src.zip \
                            https://dist.apache.org/repos/dist/dev/sis/$NEW_VERSION/RC$RELEASE_CANDIDATE/apache-sis-$NEW_VERSION-src.zip.asc
# Test
gpg --verify apache-sis-$NEW_VERSION-src.zip.asc
unzip apache-sis-$NEW_VERSION-src.zip
cd sis-$NEW_VERSION
mvn install

Execute the following commands in any temporary directory for testing the binary:

wget --no-check-certificate https://dist.apache.org/repos/dist/dev/sis/$NEW_VERSION/RC$RELEASE_CANDIDATE/apache-sis-$NEW_VERSION-bin.zip \
                            https://dist.apache.org/repos/dist/dev/sis/$NEW_VERSION/RC$RELEASE_CANDIDATE/apache-sis-$NEW_VERSION-bin.zip.asc
# Test
gpg --verify apache-sis-$NEW_VERSION-bin.zip.asc
unzip apache-sis-$NEW_VERSION-bin.zip
cd apache-sis-$NEW_VERSION
unset SIS_DATA
bin/sis about --verbose
bin/sis metadata https://github.com/opengeospatial/geoapi/raw/master/geoapi-netcdf/src/test/resources/org/opengis/wrapper/netcdf/NCEP-SST.nc
bin/sis crs http://svn.apache.org/repos/asf/sis/trunk/core/sis-referencing/src/test/resources/org/apache/sis/referencing/crs/ProjectedCRS.xml --format WKT

Prepare OpenOffice add-in

The OpenOffice add-in source code can be distributed under Apache 2 license only. But the binary may contain non-free resources, notably the EPSG geodetic dataset. For now, the release manager should apply the following local modifications before to build the add-in:

  • Move to the application/sis-openoffice directory.
  • Replace the src/main/unopkg/license.txt file by EPSG terms of use. A copy is found in the non-free group of SIS modules.
  • Run mvn package --activate-profiles non-free.

Put the release candidate up for a vote

  • Create a VOTE email thread on dev@ to record votes as replies. A template is available here.
  • Create a DISCUSS email thread on dev@ for any vote questions. A template is available here.
  • Perform a review of the release and cast your vote:
  • a -1 vote does not necessarily mean that the vote must be redone, however it is usually a good idea to rollback the release if a -1 vote is received. See Recovering from a vetoed release below.
  • After the vote has been open for at least 72 hours, has at least three +1 PMC votes and no -1 votes, then post the results to the vote thread:
    • Reply to the initial email and prepend to the original subject "[RESULT]"
    • Include a list of everyone who voted +1, 0 or -1.

Recovering from a vetoed release

Reply to the initial vote email and prepend to the original subject:

[CANCELED]

Delete the svn tag created by the release:perform step:

svn delete https://svn.apache.org/repos/asf/sis/tags/$NEW_VERSION --message "deleting tag from rolled back release"

Drop the Nexus staging repository:

  • Go to Nexus repositoryBuild PromotionStaging repositories and search for org.apache.sis in the Repository column.
  • Right click on the closed staging repositories (org.apache.sis-<id>) and select Drop.

Make the required updates that caused the vote to be canceled during the release cycle.

Finalize the release

The artificats in the repository are not yet mirrored and available for Maven to download. Promote the staged Nexus artifacts, by releasing them.

  • Go to Nexus repositoryBuild PromotionStaging repositories and search for org.apache.sis in the Repository column.
  • Click the checkboxes of the closed staging repositories (org.apache.sis-<id>) and press Release in the menu bar.

Check in the source and binary artifacts into distribution svn which will be pulled by all mirrors within 24 hours. The dist/dev svn is not mirrored, but the dist/release is. From any directory:

svn move https://dist.apache.org/repos/dist/dev/sis/$NEW_VERSION/RC$RELEASE_CANDIDATE \
         https://dist.apache.org/repos/dist/release/sis/$NEW_VERSION \
    --message "Committing SIS Source and Binary Release Candidate $RELEASE_CANDIDATE for SIS-$NEW_VERSION."

svn delete https://dist.apache.org/repos/dist/dev/sis/$NEW_VERSION \
    --message "Delete SIS $NEW_VERSION staging repository after release."

Update JIRA:

  • Update the JIRA versions to mark the version as "released".
  • Set the date to the date that the release was approved.
  • Prepare for the next release:
    • Make a new release entry in JIRA for the next release.
    • Update the Roadmap wiki page.

Verify release signatures

Download all source and binary artifacts into a new directory, then execute in that directory:

find . -name "*.asc" -exec gpg --verify '{}' \;

The output shall report only good signatures.

Announce the release

  • WAIT 24 hours after committing releases for mirrors to replicate.
  • Publish the web site updates:
  • Make an announcement about the release on the dev@, users@, and announce@ mailing lists. A template is available here. The email needs to be sent from an @apache.org email address.

Delete the prior version:

svn delete https://dist.apache.org/repos/dist/release/sis/$OLD_VERSION \
    --message "Archive SIS-$OLD_VERSION after release of SIS-$NEW_VERSION."

Update master for the next development cycle

On the development branch (usually JDK8), update the version numbers in the pom.xml files on master with the following command:

mvn clean
mvn release:update-versions --define autoVersionSubmodules=true

This change will need to be merged manually on the JDK7 branch and on the master. Then on the development branch:

  • Edit the version number in the application/sis-console/src/main/artifact/README file.
  • Edit the value of the MAJOR_VERSION or MINOR_VERSION constant in the core/sis-utility/src/main/java/org/apache/sis/internal/system/Modules.java file.

Delete old artifacts on Maven snapshot repository

Login in the Nexus repository. In the Repositories tag, select Snapshots of type hosted (not to be confused with Snapshots of type group). Navigate to the org/apache/sis directory and delete all directories starting with the old version number. The sub-directories need to be cleaned too.