Commit d2069b6e authored by Pedro Eduardo Trujillo's avatar Pedro Eduardo Trujillo
Browse files

Merge branch 'dev' into 'master' 

Define variable VERSION para jobs de building

See merge request redmic-project/gitlab-ci-templates!83
parents f046bd14 c75c9dc9

README.md

+107 −0
Original line number Diff line number Diff line 
# GitLab CI templates



Collection of CI/CD templates used and extended by REDMIC platform projects



## Backward compatibility



Until [`v1.6.0`](https://gitlab.com/redmic-project/gitlab-ci-templates/-/releases/v1.6.0), templates were defined at project root. Now they are grouped into subdirectories, but there is a compatibility template in place for each one.



These compatibility templates only import current version of templates, adding some warnings about using deprecated version of templates.



Legacy projects using these deprecated templates should continue working ok, but is recommended to migrate template imports to updated versions.



This compatibility layer will be removed when `v2.0.0` is released. If you need to use old templates after this breaking change is done, fix your templates include to a previous version.



## Templates description



You can use templates available at this project into your own projects, importing them at your `.gitlab-ci.yml` file.



Each template contains one or more *GitLab CI Jobs*, which you can use *as-is* or customize, overriding values or extending your own jobs from them.



Note that some templates have a base definition version, prefixed with an underscore (`_`). You can import them if this definition is more convenient to your project.



## Stages



*GitLab CI Jobs* from these templates run at a specific **stage** of your project *GitLab CI Pipelines*. Check description of each template for more info.



You must reference these stages (ordered as you need) at your project's `.gitlab-ci.yml`, or jobs will not run. Jobs stage value can be overwritten too, is you wish.



Here is a list of all stages used by template jobs by default, with suggested order:



1. `test`

1. `build` (`build-parent`, `build-lib` and `build-service` for functional-unit).

1. `deploy` (when referred to package deployment)

1. `pre-package`

1. `package`

1. `post-package`

1. `deploy` (when referred to service deployment)

1. `deploy-external-service`

1. `maintenance`



## Templates usage



Templates are included into `.gitlab-ci.yml` like this:



```yaml

include:

  - project: 'redmic-project/gitlab-ci-templates'

    ref: master

    file: '/scanning/dependency-scanning.yml'



stages:

  - test



...

```



## Templates available



Templates are located at different directories, attending it purpose.



### Files at root level



All templates at root level (except `deprecation-warning.yml`) are deprecated, but linked to current version of templates. Check [Backward compatibility](#backward-compatibility) for further details.



### Building



#### Maven



Build an application with Maven, using [`redmic-project/docker/maven`](https://gitlab.com/redmic-project/docker/maven).



* **functional-unit.yml**: Build a functional-unit project. Include 3 stages: `build-parent`, `build-lib` and `build-service`.

* **library.yml**: Build a library project. Run at `build` stage.

* **microservice.yml**: Build a microservice project. Run at `build` stage.



### Deployment external service



Deploy ancillary services (predefined externally) to your main service.



* **backup-files.yml**: Deploy a files backup service. Defined at [`redmic-project/maintenance/backup-files`](https://gitlab.com/redmic-project/maintenance/backup-files). Run at `deploy-external-service` stage.

* **backup-postgresql.yml**: Deploy a PostgreSQL database backup service. Defined at [`redmic-project/postgres/backup-postgresql`](https://gitlab.com/redmic-project/postgres/backup-postgresql). Run at `deploy-external-service` stage.



### Deployment package



* **npm-deployment.yml**: Upload an already built NPM package to *GitLab packages* of your project. Run at `deploy` stage. Inherit from [GitLab CI templates](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates).



### Deployment service



Deploy a service (based in Docker) in a remote environment, using [`redmic-project/docker/docker-deploy`](https://gitlab.com/redmic-project/docker/docker-deploy).



* **custom-image.yml**: Adapt jobs from `deployment-service/docker-deploy.yml` to use a custom Docker image, defined at your project and uploaded to *GitLab Docker registry* of your project.



* **docker-deploy.yml**: Deploy one or more services defined at your project (using compose files and related contents) to a remote environment of your choice. Run at `deploy` stage. Support multiple environments (like `dev` and `pro`).



* **maintenance.yml**: Relaunch already deployed services, in order to do maintenance tasks. You must trigger these jobs using *GitLab pipeline schedules*. Run at `maintenance` stage. Support multiple environments (like `dev` and `pro`).



#### Functional unit



* **docker-deploy.yml**: Same as `deployment-service/docker-deploy.yml`, but preconfigured to deploy services defined into a functional-unit, using always custom images (like `deployment-service/custom-image.yml` does). Run at `deploy` stage.



### Packaging Docker



Only for projects with Docker image definitions. Include Docker image building, tagging and pushing, and also Dockerfile linting and Docker image scanning.



* **docker-build.yml**: Build a Docker image defined at your project, using [`pedroetb-projects/docker-build`](https://gitlab.com/pedroetb-projects/docker-build). Imports jobs from `packaging-docker/dockerfile-linting.yml` and `scanning/container-scanning.yml` too. Include 3 stages: `pre-package`, `package` and `post-package`.

* **dockerfile-linting.yml**: Run syntax checks over Dockerfile at your project. Run at `pre-package` stage. Already included at `packaging-docker/docker-build.yml`.



### Scanning



* **container-scanning.yml**: Run vulnerability checks over your Docker image. Run at `post-package` stage. Already included at `packaging-docker/docker-build.yml`. Inherit from [GitLab CI templates](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates).

* **dependency-scanning.yml**: Lookup dependencies at your code, finding vulnerabilities and licenses compliance status. Run at `test` stage. Inherit from [GitLab CI templates](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates).

_deployment-functional-unit.yml

+4 −55
Original line number Diff line number Diff line 
include:

  - local: '/_deployment.yml'



variables:

  PROJECT_PARENT_NAME: functional-unit



.deploy:

  variables:

    STACK: ${PROJECT_PARENT_NAME}

    SERVICES_TO_CHECK: ${PROJECT_PARENT_NAME}_${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}



.deploy-microservice-development:

  extends: .deploy-development

  variables:

    COMPOSE_FILE: docker-compose.${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}.tmpl.yml:docker-compose.${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}.dev.yml

  environment:

    name: dev/${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}

    url: https://${DEV_PUBLIC_HOSTNAME}/api/${CI_PROJECT_NAME}/${FUNCTIONAL_UNIT_MICROSERVICE_NAME}



.deploy-microservice-production:

  extends: .deploy-production

  variables:

    COMPOSE_FILE: docker-compose.${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}.tmpl.yml:docker-compose.${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}.prod.yml

  environment:

    name: pro/${CI_PROJECT_NAME}-${FUNCTIONAL_UNIT_MICROSERVICE_NAME}

    url: https://${PRO_PUBLIC_HOSTNAME}/api/${CI_PROJECT_NAME}/${FUNCTIONAL_UNIT_MICROSERVICE_NAME}



.deploy-commands:

  variables: &deploy-commands-variables

    FUNCTIONAL_UNIT_MICROSERVICE_NAME: commands

    STATUS_CHECK_DELAY: 300

# Only for backward compatibility, migrate ASAP your project includes to templates included here!



.deploy-view:

  variables: &deploy-view-variables

    FUNCTIONAL_UNIT_MICROSERVICE_NAME: view

    STATUS_CHECK_DELAY: 180



.deploy-commands-development:

  extends: .deploy-microservice-development

  variables:

    <<: *deploy-commands-variables



.deploy-commands-production:

  extends: .deploy-microservice-production

  variables:

    <<: *deploy-commands-variables



.deploy-view-development:

  extends: .deploy-microservice-development

  variables:

    <<: *deploy-view-variables



.deploy-view-production:

  extends: .deploy-microservice-production

  variables:

    <<: *deploy-view-variables
 
include:

  - local: '/deployment-service/functional-unit/_docker-deploy.yml'

  - local: '/deprecation-warning.yml'

_deployment.yml

+4 −71
Original line number Diff line number Diff line 
.deploy:

  stage: deploy

  dependencies: []

  image: ${DOCKER_DEPLOY_IMAGE_NAME}:${DOCKER_DEPLOY_IMAGE_TAG}

  variables:

    DOCKER_DEPLOY_IMAGE_NAME: redmic/docker-docker-deploy

    DOCKER_DEPLOY_IMAGE_TAG: latest

    STACK: ${CI_PROJECT_NAME}

    DEPLOY_DIR_NAME: deploy

    REGISTRY_URL: ${CI_REGISTRY}

    REGISTRY_USER: ${CI_DEPLOY_USER}

    REGISTRY_PASS: ${CI_DEPLOY_PASSWORD}

  before_script:

    - >

      if [ -z ${COMPOSE_FILE} ];

      then

        firstNewComposeFile=$(echo "${NEW_COMPOSE_FILE}" | cut -d ':' -f 1);

        newSingleComposeFile=compose.yaml;

        firstOldComposeFile=$(echo "${OLD_COMPOSE_FILE}" | cut -d ':' -f 1);

        oldSingleComposeFile=docker-compose.yml;

        if [ -f ${DEPLOY_DIR_NAME}/${firstNewComposeFile} ];

        then

          echo "Detected '${firstNewComposeFile}' compose file, nice!";

          composeFile="${NEW_COMPOSE_FILE}";

        elif [ -f ${DEPLOY_DIR_NAME}/${newSingleComposeFile} ];

        then

          echo "Detected '${newSingleComposeFile}' compose file, nice!";

          composeFile="${newSingleComposeFile}";

        elif [ -f ${DEPLOY_DIR_NAME}/${firstOldComposeFile} ];

        then

          echo "Detected '${firstOldComposeFile}' compose file (old naming)";

          composeFile="${OLD_COMPOSE_FILE}";

        elif [ -f ${DEPLOY_DIR_NAME}/${oldSingleComposeFile} ];

        then

          echo "Detected '${oldSingleComposeFile}' compose file (old naming)";

          composeFile="${oldSingleComposeFile}";

        else

          echo "No compose file specified or detected, using docker-deploy default for COMPOSE_FILE";

        fi;

        if [ ! -z ${composeFile} ];

        then

          echo "Automatically set COMPOSE_FILE to '${composeFile}'";

          export COMPOSE_FILE="${composeFile}";

        fi;

      fi;

  script: deploy

# Only for backward compatibility, migrate ASAP your project includes to templates included here!



.deploy-development:

  extends: .deploy

  variables:

    SSH_REMOTE: ${DEV_SSH_REMOTE}

    SSH_PORT: ${DEV_SSH_PORT}

    DEPLOY_KEY: ${DEV_DEPLOY_KEY}

    NEW_COMPOSE_FILE: compose.tmpl.yaml:compose.dev.yaml

    OLD_COMPOSE_FILE: docker-compose.tmpl.yml:docker-compose.dev.yml

    PUBLIC_HOSTNAME: ${DEV_PUBLIC_HOSTNAME}

    DD_PUBLIC_HOSTNAME: ${DEV_PUBLIC_HOSTNAME}

  environment:

    name: dev



.deploy-production:

  extends: .deploy

  variables:

    SSH_REMOTE: ${PRO_SSH_REMOTE}

    SSH_PORT: ${PRO_SSH_PORT}

    DEPLOY_KEY: ${PRO_DEPLOY_KEY}

    NEW_COMPOSE_FILE: compose.tmpl.yaml:compose.prod.yaml

    OLD_COMPOSE_FILE: docker-compose.tmpl.yml:docker-compose.prod.yml

    PUBLIC_HOSTNAME: ${PRO_PUBLIC_HOSTNAME}

    DD_PUBLIC_HOSTNAME: ${PRO_PUBLIC_HOSTNAME}

  environment:

    name: pro
 
include:

  - local: '/deployment-service/_docker-deploy.yml'

  - local: '/deprecation-warning.yml'

_docker-build.yml

+4 −81
Original line number Diff line number Diff line 
.docker-operations:

  extends: .docker-env

  image: ${DOCKER_BUILD_IMAGE_NAME}:${DOCKER_BUILD_IMAGE_TAG}

  variables:

    DOCKER_BUILD_IMAGE_NAME: pedroetb/docker-build

    DOCKER_BUILD_IMAGE_TAG: latest

    PACKAGED_IMAGE_NAME: ${CI_REGISTRY_IMAGE}/${CI_COMMIT_REF_SLUG}

    PACKAGED_IMAGE_TAG: ${CI_COMMIT_SHA}

    REGISTRY_URL: ${CI_REGISTRY}

    REGISTRY_USER: gitlab-ci-token

    REGISTRY_PASS: ${CI_JOB_TOKEN}

# Only for backward compatibility, migrate ASAP your project includes to templates included here!



.docker-build:

  extends: .docker-operations

  stage: package

  variables:

    NEW_COMPOSE_FILE_NAME: compose.tmpl.yaml:compose.dev.yaml

    OLD_COMPOSE_FILE_NAME: docker-compose.tmpl.yml:docker-compose.dev.yml

  before_script:

    - >

      for directory in ${COMPOSE_PROJECT_DIRECTORY:-build deploy .};

      do

        if [ -z ${COMPOSE_FILE_NAME} ];

        then

          firstNewComposeFile=$(echo "${NEW_COMPOSE_FILE_NAME}" | cut -d ':' -f 1);

          newSingleComposeFile=compose.yaml;

          firstOldComposeFile=$(echo "${OLD_COMPOSE_FILE_NAME}" | cut -d ':' -f 1);

          oldSingleComposeFile=docker-compose.yml;

          if [ -f ${directory}/${firstNewComposeFile} ];

          then

            echo "Detected '${directory}/${firstNewComposeFile}' compose file, nice!";

            composeFileName="${NEW_COMPOSE_FILE_NAME}";

          elif [ -f ${directory}/${newSingleComposeFile} ];

          then

            echo "Detected '${directory}/${newSingleComposeFile}' compose file, nice!";

            composeFileName="${newSingleComposeFile}";

          elif [ -f ${directory}/${firstOldComposeFile} ];

          then

            echo "Detected '${directory}/${firstOldComposeFile}' compose file (old naming)";

            composeFileName="${OLD_COMPOSE_FILE_NAME}";

          elif [ -f ${directory}/${oldSingleComposeFile} ];

          then

            echo "Detected '${directory}/${oldSingleComposeFile}' compose file (old naming)";

            composeFileName="${oldSingleComposeFile}";

          else

            echo "No compose file detected at '${directory}/'";

          fi;

          if [ ! -z ${composeFileName} ];

          then

            echo "Automatically set COMPOSE_FILE_NAME to '${composeFileName}' and COMPOSE_PROJECT_DIRECTORY to '${directory}'";

            export COMPOSE_FILE_NAME="${composeFileName}";

            export COMPOSE_PROJECT_DIRECTORY="${directory}";

            break;

          fi;

        fi;

      done;

      if [ -z ${COMPOSE_FILE_NAME} ];

      then

        echo "No compose file specified or detected, using docker-build default for COMPOSE_FILE_NAME";

      fi;

  script: build



.docker-tag:

  extends: .docker-operations

  stage: post-package

  dependencies: []

  variables:

    NEW_IMAGE_TAG: ${CI_COMMIT_TAG}



.docker-tag-gitlab:

  extends: .docker-tag

  script: tag ${PACKAGED_IMAGE_NAME}:${PACKAGED_IMAGE_TAG} ${CI_REGISTRY_IMAGE}:${NEW_IMAGE_TAG}



.docker-tag-dockerhub:

  extends: .docker-tag

  variables:

    SOURCE_IMAGE_NAME: ${CI_PROJECT_PATH}

    ROOT_NAME: ${DOCKER_HUB_ROOT}

    TARGET_REGISTRY_URL: docker.io

    TARGET_REGISTRY_USER: ${DOCKER_HUB_USER}

    TARGET_REGISTRY_PASS: ${DOCKER_HUB_PASS}

  script: tag ${PACKAGED_IMAGE_NAME}:${PACKAGED_IMAGE_TAG} $(flatten):${NEW_IMAGE_TAG}
 
include:

  - local: '/packaging-docker/_docker-build.yml'

  - local: '/deprecation-warning.yml'

_maintenance.yml

+4 −35
Original line number Diff line number Diff line 
.scheduled-run:

  stage: maintenance

  dependencies: []

  needs: []

  image: ${DOCKER_DEPLOY_IMAGE_NAME}:${DOCKER_DEPLOY_IMAGE_TAG}

  variables:

    DOCKER_DEPLOY_IMAGE_NAME: redmic/docker-docker-deploy

    DOCKER_DEPLOY_IMAGE_TAG: latest

    STACK: ${CI_PROJECT_NAME}

    REGISTRY_URL: ${CI_REGISTRY}

    REGISTRY_USER: ${CI_DEPLOY_USER}

    REGISTRY_PASS: ${CI_DEPLOY_PASSWORD}

    SERVICE: ${CI_PROJECT_NAME}_${CI_PROJECT_NAME}

  script: relaunch

# Only for backward compatibility, migrate ASAP your project includes to templates included here!



.scheduled-run-development:

  extends: .scheduled-run

  variables:

    SSH_REMOTE: ${DEV_SSH_REMOTE}

    SSH_PORT: ${DEV_SSH_PORT}

    DEPLOY_KEY: ${DEV_DEPLOY_KEY}

  rules:

    - if: $DEVELOPMENT_MAINTENANCE_DISABLED

      when: never

    - if: $CI_PIPELINE_SOURCE == "schedule"



.scheduled-run-production:

  extends: .scheduled-run

  variables:

    SSH_REMOTE: ${PRO_SSH_REMOTE}

    SSH_PORT: ${PRO_SSH_PORT}

    DEPLOY_KEY: ${PRO_DEPLOY_KEY}

  rules:

    - if: $PRODUCTION_MAINTENANCE_DISABLED

      when: never

    - if: $CI_PIPELINE_SOURCE == "schedule"
 
include:

  - local: '/deployment-service/_maintenance.yml'

  - local: '/deprecation-warning.yml'