PipeCD – Customizing application's deployment pipeline

Docs-V0.56.x: Adding a wait stage

Mon, 01 Jan 0001 00:00:00 +0000

In addition to waiting for approvals from someone, the deployment pipeline can be configured to wait an amount of time before continuing. This can be done by adding the WAIT stage into the pipeline. This stage has one configurable field, duration, to configure how long it should wait.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
      - name: WAIT
        with:
          duration: 5m
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN

Deployment with a WAIT stage

Docs-V0.56.x: Adding a manual approval stage

Mon, 01 Jan 0001 00:00:00 +0000

While deploying an application to production environments, some teams require manual approvals before continuing. The manual approval stage enables you to control when the deployment is allowed to continue by requiring a specific person or team to approve. This stage is named WAIT_APPROVAL and you can add it to your pipeline before some stages that should be approved before they can be executed.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
      - name: WAIT_APPROVAL
        with:
          timeout: 6h
          approvers:
            - user-abc
      - name: K8S_PRIMARY_ROLLOUT

As above example, the deployment requires an approval from user-abc before K8S_PRIMARY_ROLLOUT stage can be executed.

The value of user ID in the approvers list depends on your SSO configuration, it must be GitHub’s user ID if your SSO was configured to use GitHub provider, it must be Gmail account if your SSO was configured to use Google provider.

In case the approvers field was not configured, anyone in the project who has Editor or Admin role can approve the deployment pipeline.

Also, it will end with failure when the time specified in timeout has elapsed. Default is 6h.

Deployment with a WAIT_APPROVAL stage

Docs-V0.56.x: Adding an automated deployment analysis stage

Mon, 01 Jan 0001 00:00:00 +0000

Automated Deployment Analysis (ADA) evaluates the impact of the deployment you are in the middle of by analyzing the metrics data, log entries, and the responses of the configured HTTP requests. The analysis of the newly deployed application is often carried out in a manual, ad-hoc or statistically incorrect manner. ADA automates that and helps to build a robust deployment process. ADA is available as a stage in the pipeline specified in the application configuration file.

ADA does the analysis by periodically performing queries against the Analysis Provider and evaluating the results to know the impact of the deployment. Then based on these evaluating results, the deployment can be rolled back immediately to minimize any negative impacts.

The canonical use case for this stage is to determine if your canary deployment should proceed.

Automatic rollback based on the analysis result

Prerequisites

Before enabling ADA inside the pipeline, all required Analysis Providers must be configured in the Piped Configuration according to this guide.

Analysis by metrics

Strategies

You can choose one of the four strategies to fit your use case.

THRESHOLD: A simple method to compare against a statically defined threshold (same as the typical analysis method up to v0.18.0).
PREVIOUS: A method to compare metrics with the last successful deployment.
CANARY_BASELINE: A method to compare the metrics between the Canary and Baseline variants.
CANARY_PRIMARY(not recommended): A method to compare the metrics between the Canary and Primary variants.

THRESHOLD is the simplest strategy, so it’s for you if you attempt to evaluate this feature.

THRESHOLD only checks if the query result falls within the statically specified range, whereas others evaluate by checking the deviation of two time-series data. Therefore, those configuration fields are slightly different from each other. The next section covers how to configure the ADA stage for each strategy.

Configuration

Here is an example for the THRESHOLD strategy.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - strategy: THRESHOLD
              provider: my-prometheus
              interval: 5m
              expected:
                max: 0.01
              query: |
                sum (rate(http_requests_total{status=~"5.*"}[5m]))
                /
                sum (rate(http_requests_total[5m]))

In the provider field, put the name of the provider in Piped configuration prepared in the Prerequisites section.

The ANALYSIS stage will continue to run for the period specified in the duration field. In the meantime, Piped sends the given query to the Analysis Provider at each specified interval.

For each query, it checks if the result is within the expected range. If it’s not expected, this ANALYSIS stage will fail (typically the rollback stage will be started). You can change the acceptable number of failures by setting the failureLimit field.

The other strategies are basically the same, but there are slight differences. Let’s take a look at them.

PREVIOUS strategy

In the PREVIOUS strategy, Piped queries the analysis provider with the time range when the deployment was previously successful, and compares that metrics with the current metrics.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - strategy: PREVIOUS
              provider: my-prometheus
              deviation: HIGH
              interval: 5m
              query: |
                sum (rate(http_requests_total{status=~"5.*"}[5m]))
                /
                sum (rate(http_requests_total[5m]))

In the THRESHOLD strategy, we used expected to evaluate the deployment, but here we use deviation instead. The stage fails on deviation in the specified direction. In the above example, it fails if the current metrics is higher than the previous.

CANARY strategy

With baseline:

In the CANARY_BASELINE strategy, Piped checks if there is a significant difference between the metrics of the two running variants, Canary and Baseline.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - strategy: CANARY_BASELINE
              provider: my-prometheus
              deviation: HIGH
              interval: 5m
              query: |
                sum (rate(http_requests_total{job="foo-{{ .Variant.Name }}", status=~"5.*"}[5m]))
                /
                sum (rate(http_requests_total{job="foo-{{ .Variant.Name }}"}[5m]))

Like PREVIOUS, you specify the conditions for failure with deviation.

It generates different queries for Canary and Baseline to compare the metrics. You can use the Variant args to template the queries. Analysis Template uses the Go templating engine which only replaces values. This allows variant-specific data to be embedded in the query.

The available built-in args currently are:

Property	Type	Description
Variant.Name	string	“canary”, “baseline”, or “primary” will be populated

Also, you can define the custom args using baselineArgs and canaryArgs, and refer them like {{ .VariantCustom.Args.job }}.

          metrics:
            - strategy: CANARY_BASELINE
              provider: my-prometheus
              deviation: HIGH
              baselineArgs:
                job: bar
              canaryArgs:
                job: baz
              interval: 5m
              query: cpu_usage{job="{{ .VariantCustomArgs.job }}", status=~"5.*"}

With primary (not recommended):

If for some reason you cannot provide the Baseline variant, you can also compare Canary and Primary. However, we recommend that you compare it with Baseline that is a variant launched at the same time as Canary as much as possible.

Comparison algorithm

The metric comparison algorithm in PipeCD uses a nonparametric statistical test called Mann-Whitney U test to check for a significant difference between two metrics collection (like Canary and Baseline, or the previous deployment and the current metrics).

Example pipelines

Analyze the canary variant using the THRESHOLD strategy:

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
        with:
          replicas: 20%
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - provider: my-prometheus
              interval: 10m
              expected:
                max: 0.1
              query: rate(cpu_usage_total{app="foo"}[10m])
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN

Analyze the primary variant using the PREVIOUS strategy:

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: K8S_PRIMARY_ROLLOUT
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - strategy: PREVIOUS
              provider: my-prometheus
              interval: 5m
              deviation: HIGH
              query: rate(cpu_usage_total{app="foo"}[5m])

Analyze the canary variant using the CANARY_BASELINE strategy:

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
        with:
          replicas: 20%
      - name: K8S_BASELINE_ROLLOUT
        with:
          replicas: 20%
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - strategy: CANARY_BASELINE
              provider: my-prometheus
              interval: 10m
              deviation: HIGH
              query: rate(cpu_usage_total{app="foo", variant="{{ .Variant.Name }}"}[10m])
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN
      - name: K8S_BASELINE_CLEAN

The full list of configurable ANALYSIS stage fields are here.

See more the example.

Analysis by logs

TBA

Analysis by http

TBA

[Optional] Analysis Template

Analysis Templating is a feature that allows you to define some shared analysis configurations to be used by multiple applications. These templates must be placed at the .pipe directory at the root of the Git repository. Any application in that Git repository can use to the defined template by specifying the name of the template in the application configuration file.

apiVersion: pipecd.dev/v1beta1
kind: AnalysisTemplate
spec:
  metrics:
    http_error_rate:
      interval: 30m
      provider: my-prometheus
      expected:
        max: 0
      query: |
        sum without(status) (rate(http_requests_total{status=~"5.*", job="{{ .App.Name }}"}[1m]))
        /
        sum without(status) (rate(http_requests_total{job="{{ .App.Name }}"}[1m]))

Once the AnalysisTemplate is defined, you can reference from the application configuration using the template field.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  pipeline:
    stages:
      - name: ANALYSIS
        with:
          duration: 30m
          metrics:
            - template:
                name: http_error_rate

Analysis Template uses the Go templating engine which only replaces values. This allows deployment-specific data to be embedded in the analysis template.

The available built-in args are:

Property	Type	Description
App.Name	string	Application Name.
K8s.Namespace	string	The Kubernetes namespace where manifests will be applied.

Also, custom args is supported. Custom args placeholders can be defined as {{ .AppCustomArgs.<name> }}.

Of course, it can be used in conjunction with Variant args.

See here for more examples. And the full list of configurable AnalysisTemplate fields are here.

Docs-V0.56.x: Custom Sync

Mon, 01 Jan 0001 00:00:00 +0000

CUSTOM_SYNC is one stage in the pipeline and you can define scripts to deploy run in this stage.

Note: This feature is marked as a deprecated feature and will be removed later.

How to configure Custom Sync

Add a CUSTOM_SYNC to your pipeline and write commands to deploy your infrastructure. The commands run in the directory where this application configuration file exists.

apiVersion: pipecd.dev/v1beta1
kind: LambdaApp
spec:
  name: sam-simple
  labels:
    env: example
    team: abc
  planner:
    # Must add this configuration to force use CUSTOM_SYNC stage.
    alwaysUsePipeline: true
  pipeline:
    stages:
      - name: CUSTOM_SYNC
        with:
          envs:
            AWS_PROFILE: "sample"
          run: |
            cd sam-app
            sam build
            echo y | sam deploy --profile $AWS_PROFILE

Note:

You can use CUSTOM_SYNC with any current supporting application kind, but keep alwaysUsePipeline true to not run the application kind’s default QUICK_SYNC.
Only one CUSTOM_SYNC stage should be used in an application pipeline.
The commands run with the environment variable PATH that refers ~/.piped/tools at first.

The public piped image available in PipeCD main repo (ref: Dockerfile) is based on alpine and only has a few UNIX command available (ref: piped-base Dockerfile). If you want to use your commands (sam in the above example), you can:

Prepare your own environment container image then add piped binary to it.
Build your own container image based on ghcr.io/pipe-cd/piped image.
Manually update your running piped container (not recommended).

Auto Rollback

When autoRollback is enabled, the deployment will be rolled back in the same way as Rolling Back.

When the rolling back process is triggered in the pipeline including CUSTOM_SYNC, CUSTOM_SYNC_ROLLBACK stage will be added to the deployment pipeline. CUSTOM_SYNC_ROLLBACK is different from ROLLBACK that applications set defaultly, it runs the same commands as CUSTOM_SYNC in the runnning commit to reverts all the applied changes.

Docs-V0.56.x: Script Run stage

Mon, 01 Jan 0001 00:00:00 +0000

SCRIPT_RUN stage is one stage in the pipeline and you can execute any commands.

Note: This feature is at the alpha status. Currently you can use it on all application kinds, but the rollback feature is only for the application kind of KubernetesApp.

How to configure SCRIPT_RUN stage

Add a SCRIPT_RUN to your pipeline and write commands.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: canary-with-script-run
  labels:
    env: example
    team: product
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
        with:
          replicas: 10%
      - name: WAIT
        with:
          duration: 10s
      - name: SCRIPT_RUN
        with:
          env:
            MSG: "execute script1"
          run: |
            echo $MSG
            sleep 10
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN
      - name: SCRIPT_RUN
        with:
          env:
            MSG: "execute script2"
          run: |
            echo $MSG
            sleep 10

You can define the command as run. Also, if you want to use some values as variables, you can define them as env.

The commands run in the directory where this application configuration file exists.

Execute the script file

If your script is so long, you can separate the script as a file. You can put the file with the app.pipecd.yaml in the same dir and then you can execute the script like this.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: script-run
  pipeline:
    stages:
      - name: SCRIPT_RUN
        with:
          run: |
            sh script.sh

.
├── app.pipecd.yaml
└── script.sh

Builtin commands

Currently, you can use the commands which are installed in the environment for the piped.

For example, If you are using the container platform and the offcial piped container image, you can use the command below.

git
ssh
jq
curl
commands installed by piped in $PIPED_TOOL_DIR (check at runtime)
built-in commands installed in the base image

The public piped image available in PipeCD main repo (ref: Dockerfile) is based on alpine and only has a few UNIX commands available (ref: piped-base Dockerfile).

If you want to use your commands, you can realize it with either step below.

Prepare your own environment container image then add piped binary to it.
Build your own container image based on ghcr.io/pipe-cd/piped image.

Default environment values

You can use the envrionment values related to the deployment.

Name	Description	Example
SR_DEPLOYMENT_ID	The deployment id	877625fc-196a-40f9-b6a9-99decd5494a0
SR_APPLICATION_ID	The application id	8d7609e0-9ff6-4dc7-a5ac-39660768606a
SR_APPLICATION_NAME	The application name	example
SR_TRIGGERED_AT	The timestamp when the deployment is triggered	1719571113
SR_TRIGGERED_COMMIT_HASH	The commit hash that triggered the deployment	2bf969a3dad043aaf8ae6419943255e49377da0d
SR_TRIGGERED_COMMANDER	The ID of user who triggered the deployment via UI. This is APIKey’s ID if it was triggered via `pipectl sync`. This is empty if it was triggered by your piped.	userid
SR_REPOSITORY_URL	The repository url configured in the piped config	git@github.com:org/repo.git, https://github.com/org/repo
SR_SUMMARY	The summary of the deployment	Sync with the specified pipeline because piped received a command from user via web console or pipectl
SR_CONTEXT_RAW	The json encoded string of above values	{“deploymentID”:“877625fc-196a-40f9-b6a9-99decd5494a0”,“applicationID”:“8d7609e0-9ff6-4dc7-a5ac-39660768606a”,“applicationName”:“example”,“triggeredAt”:1719571113,“triggeredCommitHash”:“2bf969a3dad043aaf8ae6419943255e49377da0d”,“repositoryURL”:“git@github.com:org/repo.git”,“labels”:{“env”:“example”,“team”:“product”}}
SR_LABELS_XXX	The label attached to the deployment. The env name depends on the label name. For example, if a deployment has the labels `env:prd` and `team:server`, `SR_LABELS_ENV` and `SR_LABELS_TEAM` are registered.	prd, server
SR_IS_ROLLBACK	This is `true` if the deployment is rollbacking. Otherwise, this is `false`.	false

Use `SR_CONTEXT_RAW` with jq

You can use jq command to refer to the values from SR_CONTEXT_RAW.

      - name: SCRIPT_RUN
        with:
          run: |
            echo "Get deploymentID from SR_CONTEXT_RAW"
            echo $SR_CONTEXT_RAW | jq -r '.deploymentID'
            sleep 10
          onRollback: |
            echo "rollback script-run"

Rollback

Note: Currently, this feature is only for the application kind of KubernetesApp.

You can define the command as onRollback to execute when to rollback similar to run. Execute the command to rollback SCRIPT_RUN to the point where the deployment was canceled or failed.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  name: canary-with-script-run
  labels:
    env: example
    team: product
  pipeline:
    stages:
      - name: K8S_CANARY_ROLLOUT
        with:
          replicas: 10%
      - name: WAIT
        with:
          duration: 10s
      - name: SCRIPT_RUN
        with:
          env:
            MSG: "execute script1"
            R_MSG: "rollback script1"
          run: |
            echo $MSG
            sleep 10
          onRollback: |
            echo $R_MSG
            sleep 10
      - name: K8S_PRIMARY_ROLLOUT
      - name: K8S_CANARY_CLEAN

The command defined as onRollback is executed as SCRIPT_RUN_ROLLBACK stage after each ROLLBACK stage.

When there are multiple SCRIPT_RUN stages, they are executed in the same order as SCRIPT_RUN on the pipeline. Also, only for the executed SCRIPT_RUNs are rollbacked.

For example, consider when deployment proceeds in the following order from 1 to 7.

1. K8S_CANARY_ROLLOUT
2. WAIT
3. SCRIPT_RUN
4. K8S_PRIMARY_ROLLOUT
5. SCRIPT_RUN
6. K8S_CANARY_CLEAN
7. SCRIPT_RUN

Then

If 3 is canceled or fails while running, only SCRIPT_RUN of 3 will be rollbacked.
If 4 is canceled or fails while running, only SCRIPT_RUN of 3 will be rollbacked.
If 6 is canceled or fails while running, only SCRIPT_RUNs 3 and 5 will be rollbacked. The order of executing is 3 -> 5.

PipeCD – Customizing application's deployment pipeline

Docs-V0.56.x: Adding a wait stage

Docs-V0.56.x: Adding a manual approval stage

Docs-V0.56.x: Adding an automated deployment analysis stage

Prerequisites

Analysis by metrics

Strategies

Configuration

PREVIOUS strategy

CANARY strategy

Comparison algorithm

Example pipelines

Analysis by logs

Analysis by http

[Optional] Analysis Template

Docs-V0.56.x: Custom Sync

How to configure Custom Sync

Auto Rollback

Docs-V0.56.x: Script Run stage

How to configure SCRIPT_RUN stage

Execute the script file

Builtin commands

Default environment values

Use SR_CONTEXT_RAW with jq

Rollback

Use `SR_CONTEXT_RAW` with jq