Continuous integration with Gitlab-CI

A software repository hosted in the MSO4SC Gitlab instance can setup a Continuous Integration/Delivery workflow by means of Gitlab-CI.

This document demonstrates the usage of several useful tools for the continuous in MSO4SC scope. This tools are packed in a lightweight Docker container and published in DockerHub (mso4sc/ci:latest).

We encourage you to use this image as base environment if you need the same features. You can see an example of how to setup CI/CD for MSO4SC in this repository. In this repository we are using this container (mso4sc/ci:latest).

In this example we sequentially perform several dependant steps in order to deploy a new software release:

  • build: build and deliver a new Singularity image

  • test: test in HPC using serverless Cloudify CLI and the HPC plugin

  • deploy: [Currently EMPTY]

  • undeploy: remove the Singularity image from SRegistry if any previous step failed.

Workflow:

(1) build >> (2) test >> (3) deploy >> (4) undeploy
Take this into account
  • If any step fail, the workflow breaks and next steps are cancelled.

  • Undeploy step only occurs if any previous step failed.

Next sections describe some features of the involved tools.

1. Gitlab-CI

Continuous integration service powered by gitlab.

To enable the continuous integration of your project you have to place a .gitlab-ci.yml file in the root path of your repository.

.gitlab-ci.yml is a YAML file to define the workflow to be performed during the continuous integration process.

More info about the Gitlab-CI YAML sintax and semantics in this link.

You can learn more about Gitlab-CI with this quick-start guide.

2. Sregistry

Transfer and storage web service for Singularity images. SRegistry is also a comand line tool to interact with the web service.

Images hosted in SRegistry are reference in a docker-like style COLLECTION/IMAGE:TAG (e.g. mso4sc/ci:latest).

To be able to push to the SRegistry web service from the command line you must use your user credentials.

User credentials must be stored in plain text file. The default path for this file is $HOME/.sregistry.

If you want to use a custom path you must set the right path into SREGISTRY_CLIENT_SECRETS environment variable.

To get your secret token you must be loged in the web service and visit your token page.

SRegistry push sintax:

$ export SREGISTRY_CLIENT_SECRETS=SREGISTRY_TOKEN_FILE
$ sregistry push --name COLLECTION/IMAGE --tag TAG SINGULARITY_IMAGE
  • SREGISTRY_TOKEN_FILE: path to a plain text file containing your JSON token

  • SREGISTRY_CLIENT_SECRETS: environment variable pointing to SREGISTRY_TOKEN_FILE

  • COLLECTION: collection name

  • IMAGE: image name

  • TAG: reference tag

  • SINGULARITY_IMAGE: path to a singularity image

SRegistry command line basic example:

$ sregistry push --name mso4sc/example --tag latest example.simg

Here you can find more info about sRegistry Client.

3. How to do it

  1. Create a secret variable containing your JSON token

    1. Go to your project settings

    2. Click on CI/CD

    3. Expand Secret variables section

    4. Create a new variable

    5. Copy the token (e.g { "registry": { "token": "…​", "username": "…​", "base": "https://sregistry.srv.cesga.es" } })

  2. Create a new .gitlab-ci.yml with the following workflow

    1. Use mso4sc/ci:latest as a base image.

    2. Write your JSON token to a file

    3. Export SREGISTRY_CLIENT_SECRETS pointin to your JSON token file

    4. Build your singularity image

    5. Push the resulting singularity image to SRegistry

    6. Delete the file containing the JSON token

  3. Commit your changes

Then the CI process will start automatically and publish the new image to SRegistry. You need to be assigned as owner of the mso4sc collection to be able to push.

By default new collections are private in SRegistry. If you make public the collection the image will be available to be pulled with Singularity. Note that `mso4sc`collection is public.

Type this to pull a public image with Singularity:

$ singularity pull shub://sregistry.srv.cesga.es/mso4sc/example:latest

4. Cloudify CLI (cfy)

Cloudify CLI (in local execution mode) allows you to execute your blueprints without connecting to the remote MSOOrchestrator instance.

This use case is really usefull for HPC automated testing to ensure the correct functioning of your blueprints before using the MSOOrchestrator.

The invocation pathway of cfy command follows the usual Cloudify flow (with an extra initial validation step) as described below:

  1. Blueprint validation

  2. Application registration

  3. Job environment bootstrap

  4. Job execution

  5. Job environment revert

You can see an implementation inxample in the gitlab-ci.yml file, this workflow is coded with the following commands:

$ ID=BLUEPRINT_ID
$ cfy blueprints validate blueprint.yaml
$ cfy init --install-plugins -b $ID -i ../inputs.yaml blueprint.yaml
$ cfy executions start -b $ID install
$ cfy executions start -b $ID run_jobs
$ cfy executions start -b $ID uninstall

You can get more info about cfy in the following link.