Skip to content

GitLab

Last edited by fabian serin
Page history

CI Pipeline for python projects

Continuous integration pipeline for python projects.

How Continuous integration works

Continuous Integration works by pushing small code chunks to your application’s code base hosted in a Git repository, and to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch. CI pipeline is defined in a .gitlab-ci.yml file present in your code base. Please refer to the documentation for details.

Continuous integration pipeline for python projects.

The common part of the pipeline for all kind of project should install the dependecies, launch the tests, gather tests results as artifacts and compute code coverage.

Example of CI pipeline

stages:
  - pytest

build:
  stage: pytest
  image: python:3.7-slim-buster
  script:
    - echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
    - apt update
    - apt install -y --no-install-recommends git
    - echo ---- ---- ---- INSTALLING DEPENDENCIES ---- ---- ----
    - pip3 install -r requirements.txt
    - pip3 install -r requirements_dev.txt
    - echo ---- ---- ---- UNIT TESTS ---- ---- ----
    - pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term ./tests/unit
  coverage: '/^(?i)(TOTAL).*\s+(\d+\%)$/'
  artifacts:
    when: always
    reports:
      junit: unit_tests_report.xml
      cobertura: coverage.xml

The first part defines the list of stages of the pipeline

stages:
  - pytest

Then the stages are defined inside jobs, the name of the job (here 'build' is arbitrary), we can define in which environment runs the stage. We selected python:3.7-slim-buster which is a lightweight ubuntu based docker image with python 3.7 pre-installed

build:
  stage: pytest
  image: python:3.7-slim-buster

Then for the script itself, the first part installs git, this is not mandatory. We did this here because some of our dependencies are retrieved using 'pip install git+https'. The second part is installing the dependencies.

  • requirements.txt contains the minimum to run the python project
  • requirements_dev.txt contains the required libraries to test the project The last part is to call pytest to run the tests, measure the coverage and generate results. Please refer to the pytest documentation for more details
  script:
    - echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
    - apt update
    - apt install -y --no-install-recommends git
    - echo ---- ---- ---- INSTALLING DEPENDENCIES ---- ---- ----
    - pip3 install -r requirements.txt
    - pip3 install -r requirements_dev.txt
    - echo ---- ---- ---- UNIT TESTS ---- ---- ----
    - pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term ./tests/unit

As a result unit_tests_report.xml and coverage.xml files are produced. It is required to specify those files are artifacts that will be used and interpreted by GitLab. junit and cobertura are reserved keyword for the tests results and for the coverage.

  artifacts:
    when: always
    reports:
      junit: unit_tests_report.xml
      cobertura: coverage.xml

Here we also define

  coverage: '/^(?i)(TOTAL).*\s+(\d+\%)$/'

because by default GitLab pattern for finding the coverage result in the output terminal does not match python's output

Continuous integration pipeline for python lib projects.

The goal of this pipeline is to create the package wheel for this library and push it to a python feed (Package Registry). Doing this enable the user to retrieve this python library with a simple pip install lib

Example of CI pipeline

default:
  image: python:3.7-slim-buster

stages:
  - pytest
  - publish

build:
  stage: pytest
  script:
    - pip3 install -r requirements.txt
    - pip3 install -r requirements_tests.txt
    - pytest test --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

# This job only runs on master, and it creates the lib and push it to the feed
deploylib:
  stage: publish
  script:
    - pip3 install -r requirements.txt
    - pip3 install twine
    - python3 setup.py sdist bdist_wheel
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=${CI_REGISTRY_USER} python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
  rules:
    - if: $CI_COMMIT_BRANCH == 'master'

The first part defines the image to use, the stages then the pytest part as in the previous example. The second stage is about generating the python package, for more pythonic details on this please refer to the documentation.

This stage install only the dependencies needed for running the application then the package is generated by the python3 setup.py sdist bdist_wheel command

The command TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=${CI_REGISTRY_USER} python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/* is using some GitLab environment variables, automatically set during the CI pipeline:

  • CI_JOB_TOKEN : password token used to grant access to the package repository
  • CI_REGISTRY_USER : user used to grant access to the package repository
  • CI_API_V4_URL : base URL of the repository
  • CI_PROJECT_ID : id of the current project (in order to store the packages in this project's package repository)
deploylib:
  stage: publish
  script:
    - pip3 install -r requirements.txt
    - pip3 install twine
    - python3 setup.py sdist bdist_wheel
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=${CI_REGISTRY_USER} python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

The last part only executes this stage when commiting on master. We do not want to generate the package for each commit for every branch, only when the merge is done on master

  rules:
    - if: $CI_COMMIT_BRANCH == 'master'

Continuous integration pipeline for python service projects.

The goal of this pipeline is to create the docker image associated to the service and push it to a docker registry (Container Registry) In this case some automatic tests can be run directly on the newly generated docker image, GitLab is providing several tools.

Example of CI pipeline

stages:
  - pytest
  - build
  - test

build:
  stage: pytest
  image: python:3.7-slim-buster
  script:
    - echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
    - apt update
    - apt install -y --no-install-recommends git
    - echo ---- ---- ---- INSTALLING DEPENDENCIES ---- ---- ----
    - pip3 install -r requirements.txt
    - pip3 install -r requirements_opengroup.txt
    - pip3 install -r requirements_dev.txt
    - echo ---- ---- ---- UNIT TESTS ---- ---- ----
    - pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term ./tests/unit
  coverage: '/^(?i)(TOTAL).*\s+(\d+\%)$/'
  artifacts:
    when: always
    reports:
      junit: unit_tests_report.xml
      cobertura: coverage.xml

deploy:
  stage: build
  image: docker:18.09.7-dind
  variables:
      IMAGE_TAG: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA
      DOCKER_DRIVER: overlay2
      DOCKER_TLS_CERTDIR: ""
  services:
    - name: docker:18.09.7-dind
      entrypoint: ["env", "-u", "DOCKER_TLS_CERTDIR"]
      command: ["dockerd-entrypoint.sh"]

  script:
    - echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
    - apk update
    - apk add git
    - echo ---- ---- ---- BUILD IMAGE ---- ---- ----
    - commit_id=$(git log -n 1 --pretty="%h")
    - echo ---- ---- TAG NAME
    - tag_name="_gitlab_$commit_id"
    - echo $tag_name
    - echo ---- ---- DATE
    - current_utc_date=`date --utc`
    - echo $current_utc_date
    - echo ---- ---- COMMIT BRANCH
    - commit_branch=$commit_id
    - echo $commit_branch
    - echo ---- ---- BUILD IMAGE
    - docker build -t $IMAGE_TAG -t=$CI_REGISTRY_IMAGE:latest --rm . -f ./build/Dockerfile --build-arg PIP_WHEEL_DIR=python-packages --build-arg build_date="$current_utc_date" --build-arg build_number=$commit_id --build-arg commit_id=$commit_id --build-arg build_origin="Gitlab" --build-arg commit_branch=$commit_branch
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - echo ---- ---- PUSH IMAGE
    - docker push $IMAGE_TAG
    - docker push $CI_REGISTRY_IMAGE:latest
#Scanning for vulnerabilities
include:
  - template: Container-Scanning.gitlab-ci.yml
  - template: Dependency-Scanning.gitlab-ci.yml
  - template: Security/SAST.gitlab-ci.yml

The first part is about defining the stages, then running the test. The second part will create the docker image and push it to a docker repository Here we use a docker in docker image, docker in docker let us run docker commands within a docker image.

deploy:
  stage: build
  image: docker:18.09.7-dind
  variables:
      IMAGE_TAG: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA
      DOCKER_DRIVER: overlay2
      DOCKER_TLS_CERTDIR: ""
  services:
    - name: docker:18.09.7-dind
      entrypoint: ["env", "-u", "DOCKER_TLS_CERTDIR"]
      command: ["dockerd-entrypoint.sh"]

This docker image is base on alpine linux, this explain why the installation of git is done with apk update and apk add git (and not with apt install git). By the way, git is needed to retrieve the commit_id and use it when creating the docker image

    - echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
    - apk update
    - apk add git

When creating the docker image we use some GitLab variables:

  • $CI_REGISTRY_IMAGE
  • $CI_REGISTRY_USER
  • $CI_REGISTRY_PASSWORD
  • $CI_REGISTRY : docker registry
  • $CI_COMMIT_REF_SLUG
  • $CI_COMMIT_SHA : git commit sha

Those variables are used to log in to docker container registry associated to the current project and push the newly generated docker images.

    - echo ---- ---- BUILD IMAGE
    - docker build -t $IMAGE_TAG -t=$CI_REGISTRY_IMAGE:latest --rm . -f ./build/Dockerfile --build-arg PIP_WHEEL_DIR=python-packages --build-arg build_date="$current_utc_date" --build-arg build_number=$commit_id --build-arg commit_id=$commit_id --build-arg build_origin="Gitlab" --build-arg commit_branch=$commit_branch
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - echo ---- ---- PUSH IMAGE
    - docker push $IMAGE_TAG
    - docker push $CI_REGISTRY_IMAGE:latest

The last part is running some analysis tools provided by GitLab on the docker image, those tests requires the image to have this tag : $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA

include:
  - template: Container-Scanning.gitlab-ci.yml
  - template: Dependency-Scanning.gitlab-ci.yml
  - template: Security/SAST.gitlab-ci.yml

The results of those tests are accessible in the pipeline details