Jenkins

Overview

The CI/CD pipeline provides the integrated tool-kit for building and deploying SPC platform components continuously across multiple environments/reguibs for developing the SPC platform based on the consistent user experience. A Jenkins job is provided for each CI/CD action which could be triggered manually or automatically.

This section contains whole step of CI/CD pipeline including: * Defining service and components as a unit of CI/CD artifact * Building component binaries / docker images / rpm packages based on user-defined Jenkinsfile * Integrating testing for unit test / linting / integrtion test * Issuing release version of service/components based on github tag * Promoting service/component across environments * Propagating service/component across regions * Deploying service to DEV env automatically or another environment by manual

Terminologies

• environment: DEV / STG / PPD / PRD
• region: eu-lab-central-1, …
• promotion: Preparing an artifact to the next environment when it gets ready to be deployed.
• propagation: Replicating an artifact from the master region to another region when the artifact promotion occurs.
• service: It is a unit of target of the CI/CD action for build/promotion/deployment based on the same version information during the pipeline execution. It could be defined as a single component or as multiple components. A set of Jenkins jobs for multiple environment/region for each step are defined for a single service. And it is related to a single deploy execution by a single ansible command based on a single deployment job based on single ansible command.
• component: It is a unit of the artifact as the result of build action. Currently, it supports multiple format of artifact including binary, docker image and rpm package. If the multiple components are defined in a single service, you need to understand that those components are promoted or deployed together based on the same version information by the single jenkins job during the pipeline execution.

Architecture

- Jenkins master - One master node for SPC globally - Working as a farm on Ceres v1 in SPC-KR-STG - Saves settings such as nodes, jobs, credentials and users - Backup to Github (exclude history and workspace) - Restore when the farm is launched - OAuth: Github, Keycloak

• Jenkins slave
  • One slave node per each region (lab-eu, lab-kr)
  • Executes build tasks, test tasks and deploy tasks
  • Workspace: /opt/jenkins-spc-ci/…
  • Connect from master via ssh

Environments and Regions

• All build jobs execute on eu-lab-central-1 region and the results propagate other region

  ENV	eu-lab-central-1	KR	US	SG
  dev	O	TBD	TBD	TBD
  stg	O	TBD	TBD	TBD
  ppd	O	TBD	TBD	TBD
  prd	O	TBD	TBD	TBD

CICD steps by environments

Details of CICD steps

Build

• Build a binary, RPM or dockerize
• Build closure
• Trigger Jenkins builds by pushing to Github
  • Branches
  • Pull Requests
  • Tags

Test

• Unit test
• Test closure
• Convert go test output to junit xml (optional)
  • file path: $(WORKSPACE)/test.xml
  • sample usage: $(GOTEST_CMD) 2>&1 | go-junit-report > test.xml

Promotion

• Promote bundles to the next environment
  • DEV -> STG -> PRD
  • DEV -> STG -> PPD -> PRD
• Only for semantic version (v0.0.1)

Integration Test

• TBD

Deploy

• Automatic deployment for DEV

  • Configure a specific target branch in your Jenkinsfile groovy def ctx = ci([ branch: 'master', ...

  • Trigger Jenkins builds by pushing to Github

    • Specific Branche for deployment
      • build > unit test > upload to artifactory > deploy to dev
    • Branches & Pull Requests
      • build > unit test > upload to artifactory
    • Tags
      • build > unit test > upload to artifactory > deploy to dev > promote to stg
      • tags with semantic version (v0.0.1)

  • NOTE: If the changes of branch and tag occur at the same time and the two builds are triggered, the final deployment target cannot be guaranteed.

• Manual deployment for any env

  • Select a specific version of the artifact
  • Deploy via ansible

Versioning

• snapshot
  • ${BUILD_TIMESTAMP}-${GIT_COMMIT}-${BRANCH_NAME}
  • examples
    • 20200605T025303Z-dacbb59-master
    • 20200602T030628Z-064ddfc-next_sp2
    • 20200603T051917Z-17f72b1-v0.1.11
• latest
  • latest-${BRANCH_NAME}
  • examples
    • latest-master
    • latest-next_sp2
    • latest-v0.1.11
• release
  • ${TAG_VERSION}
  • examples
    • v0.1.11
      

How to use

Multipipeline

• Enhances Pipeline plugin to handle branches better by automatically grouping builds from different branches.
• Automatically creates a new Jenkins job whenever a new branch is pushed to a source code repository.
• Other plugins can define various branch types, e.g. a Git branch, a Subversion branch, a GitHub Pull Request etc.
• https://www.jenkins.io/doc/book/pipeline/multibranch/

Jenkinsfile

• Every module have a Jenkinsfile in the root path of the repository

• Jenkinsfile has a single method ci() with several arguments

• Arguments

  • shared library
    • Import globally shared library ‘spc-ci’ using a version specifier, such as branch, tag, etc
    • ex) spc-ci@$BRANCH_NAME groovy @Library('spc-ci@v2') _
  • service : service name, One service can have multiple components

  • branch : target branch name for deployment

        service: 'spc-ci-example',
        branch: 'master',
    

  1. component

     • name : component name (unique)
     • type : type of the component. It can be one of the binary, docker, and rpm
     • source : a name of target source with real path

         components: [
             [
                 name: "spc-ci-example",
                 type: "docker",
                 source: "spc-ci-example:latest",
             ]
             [
                 name: "spc-ci-example-2",
                 type: "binary",
                 source: "bin/spc-ci-example"
             ]
             [
                 name: "spc-ci-example-3",
                 type: "rpm",
                 source: "bin/spc-ci-example*.rpm"
             ]
         ]
     

  2. build closure

     • The code between thr curly braces is executed in the build job

         build_closure: {
             echo 'spc-ci-example build'
             sh '''
                 make clean
                 make docker-build
             '''
         }
     

  3. test closure

     • The code between thr curly braces is executed in the test job

         test_closure: {
             echo 'spc-ci-example test'
             sh '''make testci'''
         },
     

  4. deploy

     • ansible : options for the deploy job

       ansible: [
           git_branch: 'master',
           playbook: 'platform.yml',
           tags: 'spc-ci-example',
           extras: ''
       ]
       

     • ceres : options for the deploy job

             ceres: [
                 target: 'stg-kr',
                 farm: 'exFarm',
                 roles: [
                     [
                         name: 'exRole',
                         containers: [
                             [
                                 name: 'exContainer',
                                 component: 'exComponent',
                                 type: 'docker'
                             ]
                         ]
                     ]
                 ]
             ]
     

Slack Notification

• Jenkins notification integration with Slack Notification plugin
• For more information
  • https://plugins.jenkins.io/slack/
  • https://api.slack.com/messaging/composing/layouts#attachments

1. Sending messages using the pipeline step to Slack

   • Configure pipeline in your jenkins job groovy slackSend (channel: '#channel-name', color: "good", message: "success to deploy", notifyCommitters: true, teamDomain: 'slack-team-domain', tokenCredentialId: 'slack-token')

   • Example : next/spc-ci-example/DEV deploy on eu-lab-central-1

     @Library('spc-ci@v2') _
     
     def ctx = [:]
     try {
         // deploy
         ctx = ci.deploy([
             service: 'spc-ci-example',
             environment: 'dev',
             region: 'eu-lab-central-1',
             ansible: [
                 git_branch: 'v2',
                 playbook: 'platform.yml',
                 tags: 'spc-ci',
                 extras: '--check'
             ]
         ])
     } catch(Exception e) {
     } finally {
         def color = "danger"
         if (ctx.result == "SUCCESS") {
             color = "good"
         }
         // slack message basic formatting
         slackSend (channel: '#spc-cicd-logs', color: "${color}", message: "${ctx.service} deployment ${ctx.result}\n*Job Name*: ${JOB_NAME}\n*Service*: ${ctx.service}\n*Environment*: ${ctx.environment}\n*Version*: ${ctx.version}\n*Branch*: ${ctx.branch}", notifyCommitters: true, teamDomain: 'rd-ams55', tokenCredentialId: 'slack-token-rd-ams55')
     }
     

2. Send messages using the multipipeline step to Slack

   • Add the following code in your Jenkinsfile groovy slackSend (channel: '#channel-name', color: "danger", message: "fail to deploy", notifyCommitters: true, teamDomain: 'slack-team-domain', tokenCredentialId: 'slack-token')

   • Example : https://github.com/cloud-pi/spc-ci-example/blob/master/Jenkinsfile

     @Library('spc-ci@v2') _
     
     def ctx = ci([
         service: 'spc-ci-example',
         branch: 'ttt',
         ... (skip)
         ]
     ])
     
     def color = "danger"
     if (ctx.result == "SUCCESS") {
         color = "good"
     }
     // slack message basic formatting
     slackSend (channel: '#spc-cicd-logs', color: "${color}", message: "${ctx.service} deployment ${ctx.result}\n*Job Name*: ${JOB_NAME}\n*Service*: ${ctx.service}\n*Environment*: ${ctx.environment}\n*Version*: ${ctx.version}\n*Branch*: ${ctx.branch}", notifyCommitters: true, teamDomain: 'rd-ams55', tokenCredentialId: 'slack-token-rd-ams55')
     
     

Sample

@Library('spc-ci@v2') _

def ctx = ci([
    service: 'spc-ci-example',
    branch: 'master',

    components: [
        [
            name: "spc-ci-example",
            type: "docker",
            source: "spc-ci-example:latest",
        ]
    ],

    build_closure: {
        echo 'spc-ci-example build'
        sh '''
            make clean
            make docker-build
        '''
    },

    test_closure: {
        echo 'spc-ci-example test'
        sh '''make testci'''
    },

    ansible: [
        git_branch: 'master',
        playbook: 'platform.yml',
        tags: 'spc-ci-example',
        extras: ''
    ]
])

if (ctx.deployStep.deploy) {
    build job: "test-qa-lifecycle/ceres-v2-test-suites/test-client-awscli-iam-dev", wait: true
}

• find more information https://github.com/cloud-pi/spc-ci/tree/v2
  • pipeline for manual jobs
  • artifactory repo path

Jenkins Job Layout

• Layout

  • Build
    • Multiple pipeline job
    • Scan target repository filterd by branch name
    • If a commit occurs, a build job is triggered
  • Promote
    • ${ENV} promote to ${next_ENV}
    • ENV : DEV / STG / PPD / PRD
  • Deploy
    • ${ENV} deploy on ${Region}
    • ENV : DEV / STG / PPD / PRD
    • Region : ue-lab-central-1 / us-east-1 / ap-northeast-1 / ap-southeast-1

• View

  • DEPLOY ${region}
    • DEPLOY ap-northeast-1
    • DEPLOY eu-lab-central-1
    • DEPLOY us-east-1
  • PROMOTE

Scenarios

1. Continuously build and unit test for the branches/PRs
   • pipeline flow: build > unit test
   • If a new branch/PR is created, the branch/PR build job will be created and built automatically.
   • You can define the test code on test closure in your Jenkinsfile and the unit test will be performed after build.
   • You can integrate the test report with Jenkins for unit test and lint results.
   • You can see the branch build job in Branches tab in - Build folder.
   • You can see the PR build job in Pull Request tab in - Build folder.
   • If you need to deploy it on DEV for testing you can do that by the manual job.
2. Continuously build and deploy specific branch to DEV environment
   • pipeline flow: build > unit test > deploy to DEV
   • Commit changes to a branch
     • case #1) deploying with MASTER branch
       1. commit code to your FEATURE branch
       2. create, review, and approve a pull request
       3. merge FEATURE branch into MASTER
     • case #2) deploying with DEPLOY branch
       1. commit code to your FEATURE branch
       2. create, review, and approve a pull request
       3. merge FEATURE branch into MASTER
       4. merge MASTER into DEPLOY for release
   • You need to define test closure and build closure for CI/CD steps.
   • You need to define components in the service.
   • You need to see the build job in Branches tab in - Build folder.
   • After building, it is deployed to the DEV environment automatically.
3. TBD) manual build for specific commit/branch
4. Issuing release version
   • pipeline flow: build > unit test > deploy to DEV > promote to STG
   • You can issue the release veresion of your service by creating tag with the semantic version. ex) v0.0.1
   • It is built and deployed on DEV automatically.
   • You can find the build job in Tags tab in - Build folder.
   • After the DEV deployment, the artifact is promted to STG.
   • The promoted artifact will be propagated to the other regions.
5. Deploy to STG by manual for specific version of artifact
   • pipeline flow: deploy to STG
   • You can deploy artifact to STG by manual.
   • You have to choose the version of artfacts to deploy.
6. Promote/Deploy to PPD
   • pipeline flow: promote to PPD / deploy to PPD
   • You can promote artifact to PPD by manual.
   • You have to choose the version of artifact to promote.
   • The promotion is happening only in eu-lab-central-1 region.
   • The promoted artifact will be propagated to the other regions automatically.
   • You can deploy artifact to PPD by manual(after the propagation if required).
   • You have to choose the version of artifact to deploy.

Backup

- Gist on ceres v1 - init.sh (restore from github backup repository) - plugins.txt - Github backup repository - Backup Jenkins configurations, jobs, nodes, users and credentials - Repository: https://github.com/cloud-pi/spc-ci-jobs - Backup Job: https://ci.spcconsole.com/job/ci-backup/job/push/