|
|
# Continuous integration pipeline for python projects.
|
|
|
|
|
|
## How Continuous integration works
|
|
|
Continuous Integration works by pushing small code chunks to your application’s code base hosted in a Git repository, and to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch.
|
|
|
CI pipeline is defined in a .gitlab-ci.yml file present in your code base.
|
|
|
Please refer to the [documentation](https://docs.gitlab.com/ee/ci/) for details.
|
|
|
|
|
|
## Continuous integration pipeline for python projects.
|
|
|
The common part of the pipeline for all kind of project should install the dependecies, launch the tests, gather tests results as artifacts and compute code coverage.
|
|
|
|
|
|
Example of CI pipeline
|
|
|
```yaml
|
|
|
stages:
|
|
|
- pytest
|
|
|
|
|
|
build:
|
|
|
stage: pytest
|
|
|
image: python:3.7-slim-buster
|
|
|
script:
|
|
|
- echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
|
|
|
- apt update
|
|
|
- apt install -y --no-install-recommends git
|
|
|
- echo ---- ---- ---- INSTALLING DEPENDENCIES ---- ---- ----
|
|
|
- pip3 install -r requirements.txt
|
|
|
- pip3 install -r requirements_dev.txt
|
|
|
- echo ---- ---- ---- UNIT TESTS ---- ---- ----
|
|
|
- pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term ./tests/unit
|
|
|
coverage: '/^(?i)(TOTAL).*\s+(\d+\%)$/'
|
|
|
artifacts:
|
|
|
when: always
|
|
|
reports:
|
|
|
junit: unit_tests_report.xml
|
|
|
cobertura: coverage.xml
|
|
|
```
|
|
|
|
|
|
The first part defines the list of stages of the pipeline
|
|
|
```yaml
|
|
|
stages:
|
|
|
- pytest
|
|
|
```
|
|
|
Then the stages are defined inside jobs, the name of the job (here 'build' is arbitrary), we can define in which environment runs the stage. We selected python:3.7-slim-buster which is a lightweight ubuntu based docker image with python 3.7 pre-installed
|
|
|
```yaml
|
|
|
build:
|
|
|
stage: pytest
|
|
|
image: python:3.7-slim-buster
|
|
|
```
|
|
|
Then for the script itself, the first part installs git, this is not mandatory. We did this here because some of our dependencies are retrieved using ['pip install git+https'](https://pip.pypa.io/en/stable/reference/pip_install/?highlight=git%20https#git).
|
|
|
The second part is installing the depenencies.
|
|
|
- requirements.txt contains the minimum to run the python project
|
|
|
- requirements.txt contains the required libraries to test the project
|
|
|
The last part is to call pytest to run the tests, measure the coverage and generate results. Please refer to the [pytest documentation](https://docs.pytest.org/en/stable/contents.html) for more details
|
|
|
```yaml
|
|
|
script:
|
|
|
- echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
|
|
|
- apt update
|
|
|
- apt install -y --no-install-recommends git
|
|
|
- echo ---- ---- ---- INSTALLING DEPENDENCIES ---- ---- ----
|
|
|
- pip3 install -r requirements.txt
|
|
|
- pip3 install -r requirements_dev.txt
|
|
|
- echo ---- ---- ---- UNIT TESTS ---- ---- ----
|
|
|
- pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term ./tests/unit
|
|
|
```
|
|
|
As a result unit_tests_report.xml and coverage.xml files are produced. It is required to specify those files are artifacts that will be used and interpreted by GitLab. junit and cobertura are reserved keyword for the tests results and for the coverage.
|
|
|
```yaml
|
|
|
artifacts:
|
|
|
when: always
|
|
|
reports:
|
|
|
junit: unit_tests_report.xml
|
|
|
cobertura: coverage.xml
|
|
|
```
|
|
|
Here we also define
|
|
|
```yaml
|
|
|
coverage: '/^(?i)(TOTAL).*\s+(\d+\%)$/'
|
|
|
```
|
|
|
because by default [GitLab pattern](https://docs.gitlab.com/ee/ci/pipelines/settings.html#test-coverage-parsing) for finding the coverage result in the output terminal does not match python's output
|
|
|
|
|
|
|
|
|
## Continuous integration pipeline for python lib projects.
|
|
|
The goal of this pipeline is to create the package wheel for this library and push it to a python feed (Package Registry).
|
|
|
Doing this enable the user to retrieve this python library with a simple
|
|
|
`pip install lib`
|
|
|
|
|
|
-[]TODO complete this part based on DDMS libraries
|
|
|
|
|
|
## Continuous integration pipeline for python service projects.
|
|
|
The goal of this pipeline is to create the docker image associated to the service and push it to a docker registry (Container Registry)
|
|
|
In this case some automatic tests can be run directly on the newly generated docker image, [GitLab is providing several tools](https://docs.gitlab.com/ee/user/application_security/).
|
|
|
|
|
|
-[]TODO complete this part based on DDMS Service
|
|
|
|
|
|
## Full example
|
|
|
```yaml
|
|
|
stages:
|
|
|
- pytest
|
|
|
- build
|
|
|
- test
|
|
|
|
|
|
build:
|
|
|
stage: pytest
|
|
|
image: python:3.7-slim-buster
|
|
|
script:
|
|
|
- echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
|
|
|
- apt update
|
|
|
- apt install -y --no-install-recommends git
|
|
|
- echo ---- ---- ---- INSTALLING DEPENDENCIES ---- ---- ----
|
|
|
- pip3 install -r requirements.txt
|
|
|
- pip3 install -r requirements_opengroup.txt
|
|
|
- pip3 install -r requirements_dev.txt
|
|
|
- echo ---- ---- ---- UNIT TESTS ---- ---- ----
|
|
|
- pytest --junit-xml=unit_tests_report.xml --cov=app --cov-report=xml:coverage.xml --cov-report=term ./tests/unit
|
|
|
coverage: '/^(?i)(TOTAL).*\s+(\d+\%)$/'
|
|
|
artifacts:
|
|
|
when: always
|
|
|
reports:
|
|
|
junit: unit_tests_report.xml
|
|
|
cobertura: coverage.xml
|
|
|
|
|
|
deploy:
|
|
|
stage: build
|
|
|
image: docker:18.09.7-dind
|
|
|
variables:
|
|
|
IMAGE_TAG: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA
|
|
|
DOCKER_DRIVER: overlay2
|
|
|
DOCKER_TLS_CERTDIR: ""
|
|
|
services:
|
|
|
- name: docker:18.09.7-dind
|
|
|
entrypoint: ["env", "-u", "DOCKER_TLS_CERTDIR"]
|
|
|
command: ["dockerd-entrypoint.sh"]
|
|
|
|
|
|
script:
|
|
|
- echo ---- ---- ---- SYSTEM DEPENDENCIES ---- ---- ----
|
|
|
- apk update
|
|
|
- apk add git
|
|
|
- echo ---- ---- ---- BUILD IMAGE ---- ---- ----
|
|
|
- commit_id=$(git log -n 1 --pretty="%h")
|
|
|
- echo ---- ---- TAG NAME
|
|
|
- tag_name="_gitlab_$commit_id"
|
|
|
- echo $tag_name
|
|
|
- echo ---- ---- DATE
|
|
|
- current_utc_date=`date --utc`
|
|
|
- echo $current_utc_date
|
|
|
- echo ---- ---- COMMIT BRANCH
|
|
|
- commit_branch=$commit_id
|
|
|
- echo $commit_branch
|
|
|
- echo ---- ---- BUILD IMAGE
|
|
|
- docker build -t $IMAGE_TAG -t=$CI_REGISTRY_IMAGE:latest --rm . -f ./build/Dockerfile --build-arg PIP_WHEEL_DIR=python-packages --build-arg build_date="$current_utc_date" --build-arg build_number=$commit_id --build-arg commit_id=$commit_id --build-arg build_origin="Gitlab" --build-arg commit_branch=$commit_branch
|
|
|
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
|
|
|
- echo ---- ---- PUSH IMAGE
|
|
|
- docker push $IMAGE_TAG
|
|
|
- docker push $CI_REGISTRY_IMAGE:latest
|
|
|
#Scanning for vulnerabilities
|
|
|
include:
|
|
|
- template: Container-Scanning.gitlab-ci.yml
|
|
|
- template: Dependency-Scanning.gitlab-ci.yml
|
|
|
- template: Security/SAST.gitlab-ci.yml
|
|
|
``` |
|
|
\ No newline at end of file |