3. Release Engineering

3.1. Nightly build on Eclipse infrastructure

The N4JS IDE, headless n4jsc.jar, and the N4JS update site is being built on the Eclipse Common Build Infrastructure (CBI). For this purpose the N4JS project is using a dedicated Jenkins instance, referred to as a "Jenkins Instance Per Project" (JIPP) in Eclipse CBI documentation. At this time, the N4JS project’s JIPP is running on the "old" infrastructure, not yet using docker. This will be migrated at a later point in time.

The N4JS JIPP is available at: https://ci.eclipse.org/n4js/

The nightly build performs the following main steps:

  1. compile the N4JS implementation,

  2. build the n4jsc.jar, the IDE products for MacOS, Windows, Linux, and the update site,

  3. run tests,

  4. sign the IDE product for macOS and package it in a .dmg file,

  5. deploy to n4jsc.jar, IDE products and update sites to Eclipse download server (i.e. download.eclipse.org),

  6. move all artifacts older than 7 days from download.eclipse.org to archive.eclipse.org.

Details about all the above steps can be found in the Jenkinsfile eclipse-nightly.jenkinsfile, located in the root folder of the N4JS source repository on GitHub.

The most accurate documentation for our JIPP can be found at https://wiki.eclipse.org/IT_Infrastructure_Doc. Note that many other documents do not apply to our JIPP, at the moment, as they refer to the new infrastructure, e.g. https://wiki.eclipse.org/CBI and https://wiki.eclipse.org/Jenkins.

3.2. Build the N4JS IDE from command line

Ensure you have

  • Java 11

  • Maven 3.2.x and

  • Node.js 8

installed on your system.

Clone the repository

git clone https://github.com/Eclipse/n4js.git

Change to the n4js folder:

cd n4js

Run the Maven build:

mvn clean verify

You may have to increase the memory for maven via export MAVEN_OPTS="-Xmx2048m" (Unix) or set MAVEN_OPTS="-Xmx2048m" (Windows).

Available optional maven profiles are:

buildProduct

create IDE products (Windows, macOS, Linux) and a jar for headless compilation
execute-plugin-tests

run OSGi tests (without UI)
execute-plugin-ui-tests

run UI-based OSGi tests
execute-ecmas-tests

run ECMA test suite
execute-smoke-tests

run generated tests using corrupted source code as input
execute-accesscontrol-tests

run generated tests for checking accessibility of class/interface members
execute-hlc-integration-tests

run integration tests using the headless jar (requires docker!)

Available system properties:

noTests

suppress execution of all tests
startAndKeepVerdaccio

enforce starting and suppress stopping of the test verdaccio (see Test Verdaccio containing n4js-libs)

3.2.1. Publish maven-tooling org.eclipse.n4js.releng.util

For extending the N4JS-language in a different project, the org.eclipse.n4js.releng.util module needs to be published as a maven-plugin. You can deploy this SNAPSHOT-artifact to a local folder by providing the local-snapshot-deploy-folder-property pointing to an absolute path in the local file system: 
mvn clean deploy -Dlocal-snapshot-deploy-folder=/var/lib/my/folder/local-mvn-deploy-repository

The existence of local-snapshot-deploy-folder will trigger a profile enabling the deploy-goal for the project org.eclipse.n4js.releng.util

3.2.2. Test Verdaccio containing n4js-libs

If profile execute-hlc-integration-tests is active, a local verdaccio instance is started and populated with freshly-compiled n4js-libs (the libraries located under top-level folder /n4js-libs) and is stopped before the end of the build. The verdaccio instance is started as a docker container called n4js-test-verdaccio.

When giving -DstartAndKeepVerdaccio on the command line, such a test verdaccio will always be started/populated but never stopped, regardless of whether profile execute-hlc-integration-tests is active or not. This is useful to enforce starting of the test verdaccio (even without running integration tests) and then reusing it in subsequent builds.

3.2.3. Generation of Eclipse help for spec and design document

The HTML pages for N4JSSpec and N4JSDesign documents are generated from the Asciidoc sources in the project org.eclipse.n4js.spec org.eclipse.n4js.design by Asciispec. 

Creating Eclipse help for N4JSSpec
Figure 1. The process of creating Eclipse help for N4JSSpec

Figure The process of creating Eclipse help for N4JSSpec shows the generation process for N4JSSpec document. The process for N4JSDesign (and other adoc documents) is the same. The following explains the diagram.

  • Asciispec is used to compile the source N4JSSpec Asciidoc into a single large N4JSSpec.html file which contains all the chapters. The use of the custom parameter -a eclipse-help-mode indicates that a special header and footer styles as well as CSS style should be used (i.e. no table of content menu, no download links etc.). Here, we are using the possibility provided by Asciidoctor to configure header/footer as well as CSS style via parameter :docinfodir: and :stylesheet:.

  • Our custom tool Chunker splits N4JSSpec.html (and other documents) into multiple chunked HTML files, each of which corresponds to either the index file or a chapter. It automatically re-writes internal links.

  • Another custom tool EclipseHelpTOCGenerator takes to Docbook file N4JSSpec.xml and generates an XML file describing the table of content (TOC) in the Eclipse format. This TOC file references the chunked HTML files above.

  • Another custom tool IndexTocGenerator takes to Docbook file N4JSSpec.xml similar to EclipseHelpTOCGenerator, but it generates an HTML fragment which can be embedded into the index.html page generated by the Chunker (Thus it has to run before the Chunker in that case).

3.3. Updating frameworks and dependencies

3.3.1. Update of Eclipse, EMF, Xtext, etc.

For updating the N4JS IDE to a new version of Eclipse, EMF, Xtext, etc. follow these steps:

  1. Create a new branch.

  2. Bump versions of all dependencies mentioned in file N4JS.setup:

    1. Update all labels that refer to the version of the Ooomph setup (search for "label!" to find them).

    2. Choose a new Eclipse version and define this in N4JS.setup.

    3. For those other dependencies that come with Eclipse (e.g. EMF, Xtext) find out which version matches the chosen Eclipse version and define that version in N4JS.setup.
      Tip: use the contents list of the SimRel you are targeting, e.g. https://projects.eclipse.org/releases/2019-03

    4. For those other dependencies that are available via the Eclipse Orbit, find out which version is the latest version available in the Orbit and define that version in N4JS.setup.
      Tip: contents of the Eclipse Orbit can be found at https://download.eclipse.org/tools/orbit/downloads/
      (choose the correct link for the chosen Eclipse version!)

    5. For all remaining dependencies (i.e. unrelated to Eclipse and not in Orbit), choose a version to use and define it in N4JS.setup.

  3. Check Require-Bundle sections of MANIFEST.MF files by searching for related bundle names or for ;bundle-version=":

    1. There should be at most one version constraint for a specific bundle
      NOTE: the version constraints in the MANIFEST.MF files are just lower bounds and - at this time - we do not bump them to the latest version, in most cases.

    2. There should be no version constraints to our bundles (i.e. org.eclipse.n4js…​)

  4. Review parent pom.xml files, i.e. releng/org.eclipse.n4js.parent/pom.xml:

    1. Update property xtext-version.

    2. Check all other *-version properties and update them where needed.

  5. Update target platform file org.eclipse.n4js.targetplatform.target using Ooomph’s auto-generation:

    1. Start the Eclipse Installer.

    2. Update the Eclipse Installer (using the button with the turning arrows).

    3. On the second page, add the N4JS.setup file from your branch to the Eclipse Installer, using a GitHub raw(!) URL:
      https://raw.githubusercontent.com/eclipse/n4js/BRANCH_NAME/releng/org.eclipse.n4js.targetplatform/N4JS.setup

    4. Ooomph a new development environment with this setup.

    5. In the new Eclipse workspace created by Ooomph, the target platform file should have uncommitted changes:

      1. carefully review these changes, to be sure they make sense, and then

      2. commit & push those changes to your branch.

  6. Thoroughly test the new versions, including some manual(!) tests:

    1. Run Jenkins builds.

    2. Ooomph another N4JS development environment with Eclipse Installer. This time, after Ooomphing is completed, the target platform file should no longer have any uncommitted changes.

    3. Ensure the following types of tests can be executed locally in the newly installed Eclipse:

      1. plain JUnit tests (e.g. org.eclipse.n4js.lang.tests).

      2. Plugin tests.

      3. Plugin UI tests.

      4. SWTBot tests.

      5. Xpect tests (individual files and entire bundles; e.g. org.eclipse.n4js.spec.tests).

      6. Xpect UI tests.

    4. Ensure an N4JS IDE product can be launched from within the newly installed Eclipse using the launch configuration provided in the n4js repository.

    5. After launching the N4JS IDE product, refresh the workspace and review/commit any changes in file N4JS__IDE.launch.

    6. Download a product created in a Jenkins CI build and test it manually.

    7. After merging to master: download a product created in a nightly build and test it manually. Ensure signing and JRE bundling are still working properly.

All the above steps need to be performed in the n4js-n4 repository, accordingly (e.g. file N4JS-N4.setup).

3.3.2. Update of the embedded JRE

For updating the embedded JRE inside the N4JS IDE follow these steps:

  1. Given a new JRE download location for Linux, MacOS and Windows with a common new version

  2. Update the location related properties in the pom.xml files of

    1. n4js/builds/pom.xml

    2. n4js/builds/org.eclipse.n4js.jre.linux.gtk.x86_64/pom.xml

    3. n4js/builds/org.eclipse.n4js.jre.macosx.cocoa.x86_64/pom.xml

    4. n4js/builds/org.eclipse.n4js.jre.win32.win32.x86_64/pom.xml

  3. Update the versions at all following locations:

    1. n4js/builds/org.eclipse.n4js.jre.linux.gtk.x86_64/META-INF/MANIFEST.MF

    2. n4js/builds/org.eclipse.n4js.jre.linux.gtk.x86_64/META-INF/p2.inf

    3. n4js/builds/org.eclipse.n4js.jre.macosx.cocoa.x86_64/META-INF/MANIFEST.MF

    4. n4js/builds/org.eclipse.n4js.jre.macosx.cocoa.x86_64/META-INF/p2.inf

    5. n4js/builds/org.eclipse.n4js.jre.win32.win32.x86_64/META-INF/MANIFEST.MF

    6. n4js/builds/org.eclipse.n4js.jre.win32.win32.x86_64/META-INF/p2.inf

  4. Update the openjdk docker image used as base image in the "FROM" line at the top of all docker files:

    1. n4js-n4/jenkins/docker-build/Dockerfile

Quick Links