PipeCD – Defining application's configuration

Docs-V0.56.x: Configuring Kubernetes application

Mon, 01 Jan 0001 00:00:00 +0000

Based on the application configuration and the pull request changes, PipeCD plans how to execute the deployment: doing quick sync or doing progressive sync with the specified pipeline.

Note:

You can generate an application config file easily and interactively by pipectl init.

Quick sync

Quick sync is a fast way to sync application to the state specified in the target Git commit without any progressive strategy. It just applies all the defined manifests to sync the application. The quick sync will be planned in one of the following cases:

no pipeline was specified in the application configuration file
pipeline was specified but the PR did not make any changes on workload (e.g. Deployment’s pod template) or config (e.g. ConfigMap, Secret)

For example, the application configuration as below is missing the pipeline field. This means any pull request touches the application will trigger a quick sync deployment.

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  input:
    helmChart:
      repository: pipecd
      name: helloworld
      version: v0.3.0

In another case, even when the pipeline was specified, a PR that just changes the Deployment’s replicas number for scaling will also trigger a quick sync deployment.

Sync with the specified pipeline

The pipeline field in the application configuration is used to customize the way to do deployment by specifying and configuring the execution stages. You may want to configure those stages to enable a progressive deployment with a strategy like canary, blue-green, a manual approval, an analysis stage.

To enable customization, PipeCD defines three variants for each Kubernetes application: primary (aka stable), baseline and canary.

primary runs the current version of code and configuration.
baseline runs the same version of code and configuration as the primary variant. (Creating a brand-new baseline workload ensures that the metrics produced are free of any effects caused by long-running processes.)
canary runs the proposed change of code or configuration.

Depending on the configured pipeline, any variants can exist and receive the traffic during the deployment process but once the deployment is completed, only the primary variant should be remained.

These are the provided stages for Kubernetes application you can use to build your pipeline:

K8S_PRIMARY_ROLLOUT
- update the primary resources to the state defined in the target commit
K8S_CANARY_ROLLOUT
- generate canary resources based on the definition of the primary resource in the target commit and apply them
K8S_CANARY_CLEAN
- remove all canary resources
K8S_BASELINE_ROLLOUT
- generate baseline resources based on the definition of the primary resource in the target commit and apply them
K8S_BASELINE_CLEAN
- remove all baseline resources
K8S_TRAFFIC_ROUTING
- split traffic between variants

and other common stages:

WAIT
WAIT_APPROVAL
ANALYSIS

See the description of each stage at Customize application deployment.

Manifest Templating

In addition to plain-YAML, PipeCD also supports Helm and Kustomize for templating application manifests.

A helm chart can be loaded from:

the same git repository with the application directory, we call as a local chart

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  input:
    helmChart:
      path: ../../local/helm-charts/helloworld

a different git repository, we call as a remote git chart

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  input:
    helmChart:
      gitRemote: git@github.com:pipe-cd/manifests.git
      ref: v0.5.0
      path: manifests/helloworld

a Helm chart repository, we call as a remote chart

apiVersion: pipecd.dev/v1beta1
kind: KubernetesApp
spec:
  input:
    helmChart:
      repository: pipecd
      name: helloworld
      version: v0.5.0

A kustomize base can be loaded from:

the same git repository with the application directory, we call as a local base
a different git repository, we call as a remote base

See Examples for more specific.

Reference

See Configuration Reference for the full configuration.

Docs-V0.56.x: Configuring Terraform application

Mon, 01 Jan 0001 00:00:00 +0000

Quick Sync

By default, when the pipeline was not specified, PipeCD triggers a quick sync deployment for the merged pull request. Quick sync for a Terraform deployment does terraform plan and if there are any changes detected it applies those changes automatically.

Sync with the specified pipeline

The pipeline field in the application configuration is used to customize the way to do the deployment. You can add a manual approval before doing terraform apply or add an analysis stage after applying the changes to determine the impact of those changes.

These are the provided stages for Terraform application you can use to build your pipeline:

TERRAFORM_PLAN
- do the terraform plan and show the changes will be applied
TERRAFORM_APPLY
- apply all the infrastructure changes

and other common stages:

WAIT
WAIT_APPROVAL
ANALYSIS

See the description of each stage at Customize application deployment.

Module location

Terraform module can be loaded from:

the same git repository with the application directory, we call as a local module
a different git repository, we call as a remote module

Reference

See Configuration Reference for the full configuration.

Docs-V0.56.x: Configuring Cloud Run application

Mon, 01 Jan 0001 00:00:00 +0000

Note: Cloud Run job has not been supported yet.

Deploying a Cloud Run application requires a service.yaml file placed inside the application directory. That file contains the service specification used by Cloud Run as follows:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: SERVICE_NAME
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/maxScale: '5'
    spec:
      containerConcurrency: 80
      containers:
      - args:
        - server
        image: gcr.io/pipecd/helloworld:v0.5
        ports:
        - containerPort: 9085
        resources:
          limits:
            cpu: 1000m
            memory: 128Mi

Quick sync

By default, when the pipeline was not specified, PipeCD triggers a quick sync deployment for the merged pull request. Quick sync for a Cloud Run deployment will roll out the new version and switch all traffic to it.

Sync with the specified pipeline

The pipeline field in the application configuration is used to customize the way to do the deployment. You can add a manual approval before routing traffic to the new version or add an analysis stage to do some smoke tests against the new version before allowing them to receive the real traffic.

These are the provided stages for Cloud Run application you can use to build your pipeline:

CLOUDRUN_PROMOTE
- promote the new version to receive an amount of traffic

and other common stages:

WAIT
WAIT_APPROVAL
ANALYSIS

See the description of each stage at Customize application deployment.

Here is an example that rolls out the new version gradually:

apiVersion: pipecd.dev/v1beta1
kind: CloudRunApp
spec:
  pipeline:
    stages:
      # Promote new version to receive 10% of traffic.
      - name: CLOUDRUN_PROMOTE
        with:
          percent: 10
      - name: WAIT
        with:
          duration: 10m
      # Promote new version to receive 50% of traffic.
      - name: CLOUDRUN_PROMOTE
        with:
          percent: 50
      - name: WAIT
        with:
          duration: 10m
      # Promote new version to receive all traffic.
      - name: CLOUDRUN_PROMOTE
        with:
          percent: 100

Reference

See Configuration Reference for the full configuration.

Docs-V0.56.x: Configuring Lambda application

Mon, 01 Jan 0001 00:00:00 +0000

Deploying a Lambda application requires a function.yaml file placed inside the application directory. That file contains values to be used to deploy Lambda function on your AWS cluster. Currently, Piped supports deploying all types of Lambda deployment packages:

container images (called container image as Lambda function)
.zip file archives (which stored in AWS S3)

Besides, Piped also supports deploying your Lambda function directly from the function source code which is stored in a remote git repository.

Deploy container image as Lambda function

A sample function.yaml file for container image as Lambda function used deployment as follows:

apiVersion: pipecd.dev/v1beta1
kind: LambdaFunction
spec:
  name: SimpleFunction
  image: ecr.ap-northeast-1.amazonaws.com/lambda-test:v0.0.1
  role: arn:aws:iam::76xxxxxxx:role/lambda-role
  # The amount of memory available to the Lambda application 
  # at runtime. The value can be any multiple of 1 MB.
  memory: 512
  # Timeout of the Lambda application, the value must
  # in between 1 to 900 seconds.
  timeout: 30
  tags:
    app: simple
  environments:
    FOO: bar
  # ephemeralStorage is optional value. If you define a ephemeral storage to lambda, you can
  # use this field. The value must be in between 512 to 10240 MB.
  ephemeralStorage:
    size: 512
  # vpcConfig is optional value. If you define a vpc configuration to lambda, you can
  # use this field.
  vpcConfig:
    securityGroupIds:
      - sg-01234
      - sg-56789
    subnetIds:
      - subnet-01234
      - subnet-56789

Except the tags and the environments field, all others are required fields for the deployment to run.

The role value represents the service role (for your Lambda function to run), not for Piped agent to deploy your Lambda application. To be able to pull container images from AWS ECR, besides policies to run as usual, you need to add Lambda.ElasticContainerRegistry read permission to your Lambda function service role.

The environments field represents environment variables that can be accessed by your Lambda application at runtime. In case of no value set for this field, all environment variables for the deploying Lambda application will be revoked, so make sure you set all currently required environment variables of your running Lambda application on function.yaml if you migrate your app to PipeCD deployment.

Deploy .zip file archives as Lambda function

It’s recommended to use container image as Lambda function due to its simplicity, but as mentioned above, below is a sample function.yaml file for Lambda which uses zip packing source code stored in AWS S3.

apiVersion: pipecd.dev/v1beta1
kind: LambdaFunction
spec:
  name: SimpleZipPackingS3Function
  role: arn:aws:iam::76xxxxxxx:role/lambda-role
  # --- 3 next lines allow Piped to determine your Lambda function code stored in AWS S3.
  s3Bucket: pipecd-sample-lambda
  s3Key: pipecd-sample-src
  s3ObjectVersion: 1pTK9_v0Kd7I8Sk4n6abzCL
  # ---
  handler: app.lambdaHandler
  runtime: nodejs14.x
  memory: 512
  timeout: 30
  environments:
    FOO: bar
  tags:
    app: simple-zip-s3

Value for the runtime field should be listed in AWS Lambda runtimes official docs. All other fields setting are remained as in the case of using container image as Lambda function pattern.

Deploy source code directly as Lambda function

In case you don’t have a separated CI pipeline that provides artifacts (such as container image, built zip files) as its outputs and want to set up a simple pipeline to deploy the Lambda function directly from your source code, this deployment package is for you.

apiVersion: pipecd.dev/v1beta1
kind: LambdaFunction
spec:
  name: SimpleCanaryZipFunction
  role: arn:aws:iam::76xxxxxxx:role/lambda-role
  # source configuration use to determine the source code of your Lambda function.
  source:
    # git remote address where the source code is placing.
    git: git@github.com:username/lambda-function-code.git
    # the commit SHA or tag for remote git. Use branch name means you will always use
    # the latest code of that branch as Lambda function code which is NOT recommended.
    ref: dede7cdea5bbd3fdbcc4674bfcd2b2f9e0579603
    # relative path from the repository root directory to the function code directory.
    path: hello-world
  handler: app.lambdaHandler
  runtime: nodejs14.x
  memory: 128
  timeout: 5
  tags:
    app: canary-zip

All other fields setting are remained as in the case of using .zip archives as Lambda function pattern.

Quick sync

By default, when the pipeline was not specified, PipeCD triggers a quick sync deployment for the merged pull request. Quick sync for a Lambda deployment will roll out the new version and switch all traffic to it.

Sync with the specified pipeline

These are the provided stages for Lambda application you can use to build your pipeline:

LAMBDA_CANARY_ROLLOUT
- deploy workloads of the new version, but it is still receiving no traffic.
LAMBDA_PROMOTE
- promote the new version to receive an amount of traffic.

and other common stages:

WAIT
WAIT_APPROVAL
ANALYSIS

See the description of each stage at Customize application deployment.

Here is an example that rolls out the new version gradually:

apiVersion: pipecd.dev/v1beta1
kind: LambdaApp
spec:
  pipeline:
    stages:
      # Deploy workloads of the new version.
      # But this is still receiving no traffic.
      - name: LAMBDA_CANARY_ROLLOUT
      # Promote new version to receive 10% of traffic.
      - name: LAMBDA_PROMOTE
        with:
          percent: 10
      - name: WAIT
        with:
          duration: 10m
      # Promote new version to receive 50% of traffic.
      - name: LAMBDA_PROMOTE
        with:
          percent: 50
      - name: WAIT
        with:
          duration: 10m
      # Promote new version to receive all traffic.
      - name: LAMBDA_PROMOTE
        with:
          percent: 100

Reference

See Configuration Reference for the full configuration.

Docs-V0.56.x: Configuring ECS application

Mon, 01 Jan 0001 00:00:00 +0000

There are two main ways to deploy an Amazon ECS application.

Your application is a one-time or periodic batch job.
- it’s a standalone task.
- you need to prepare TaskDefinition
Your application is deployed to run continuously or behind a load balancer.
- you need to prepare TaskDefinition and Service

To deploy an Amazon ECS application, the TaskDefinition configuration file must be located in the application directory. This file contains all configuration for ECS TaskDefinition object and will be used by Piped agent while deploying your application/service to the ECS cluster.

To deploy your application to run continuously or to place it behind a load balancer, You need to create ECS Service. The Service configuration file also must be located in the application directory. This file contains all configurations for ECS Service object.

If you’re not familiar with ECS, you can get examples for those files from here.

Note:

You can generate an application config file easily and interactively by pipectl init.

Quick sync

By default, when the pipeline was not specified, PipeCD triggers a quick sync deployment for the merged pull request. Quick sync for an ECS deployment will roll out the new version and switch all traffic to it immediately.

In case of standalone task, only Quick sync is supported.

Here is an example for Quick sync.

apiVersion: pipecd.dev/v1beta1
kind: ECSApp
spec:
  name: simple
  labels:
    env: example
    team: xyz
  input:
    # Path to Service configuration file in Yaml/JSON format.
    serviceDefinitionFile: servicedef.yaml
    # Path to TaskDefinition configuration file in Yaml/JSON format.
    # Default is `taskdef.json`
    taskDefinitionFile: taskdef.yaml
    targetGroups:
      primary:
        targetGroupArn: arn:aws:elasticloadbalancing:ap-northeast-1:XXXX:targetgroup/ecs-lb/YYYY
        containerName: web
        containerPort: 80

apiVersion: pipecd.dev/v1beta1
kind: ECSApp
spec:
  name: standalonetask-fargate
  labels:
    env: example
    team: xyz
  input:
    # Path to TaskDefinition configuration file in Yaml/JSON format.
    # Default is `taskdef.json`
    taskDefinitionFile: taskdef.yaml
    clusterArn: arn:aws:ecs:ap-northeast-1:XXXX:cluster/test-cluster
    awsvpcConfiguration:
      assignPublicIp: ENABLED
      subnets:
        - subnet-YYYY
        - subnet-YYYY
      securityGroups:
          - sg-YYYY

Sync with the specified pipeline

These are the provided stages for ECS application you can use to build your pipeline:

ECS_CANARY_ROLLOUT
- deploy workloads of the new version as CANARY variant, but it is still receiving no traffic.
ECS_PRIMARY_ROLLOUT
- deploy workloads of the new version as PRIMARY variant, but it is still receiving no traffic.
ECS_TRAFFIC_ROUTING
- routing traffic to the specified variants.
ECS_CANARY_CLEAN
- destroy all workloads of CANARY variant.

and other common stages:

WAIT
WAIT_APPROVAL
ANALYSIS

See the description of each stage at Customize application deployment.

Here is an example that rolls out the new version gradually:

apiVersion: pipecd.dev/v1beta1
kind: ECSApp
spec:
  input:
    # Path to Service configuration file in Yaml/JSON format.
    serviceDefinitionFile: servicedef.yaml
    # Path to TaskDefinition configuration file in Yaml/JSON format.
    # Default is `taskdef.json`
    taskDefinitionFile: taskdef.yaml
    targetGroups:
      primary:
        targetGroupArn: arn:aws:elasticloadbalancing:ap-northeast-1:XXXX:targetgroup/ecs-canary-blue/YYYY
        containerName: web
        containerPort: 80
      canary:
        targetGroupArn: arn:aws:elasticloadbalancing:ap-northeast-1:XXXX:targetgroup/ecs-canary-green/YYYY
        containerName: web
        containerPort: 80
  pipeline:
    stages:
      # Deploy the workloads of CANARY variant, the number of workload
      # for CANARY variant is equal to 30% of PRIMARY's workload.
      # But this is still receiving no traffic.
      - name: ECS_CANARY_ROLLOUT
        with:
          scale: 30
      # Change the traffic routing state where
      # the CANARY workloads will receive the specified percentage of traffic.
      # This is known as multi-phase canary strategy.
      - name: ECS_TRAFFIC_ROUTING
        with:
          canary: 20
      # Optional: We can also add an ANALYSIS stage to verify the new version.
      # If this stage finds any not good metrics of the new version,
      # a rollback process to the previous version will be executed.
      - name: ANALYSIS
      # Update the workload of PRIMARY variant to the new version.
      - name: ECS_PRIMARY_ROLLOUT
      # Change the traffic routing state where
      # the PRIMARY workloads will receive 100% of the traffic.
      - name: ECS_TRAFFIC_ROUTING
        with:
          primary: 100
      # Destroy all workloads of CANARY variant.
      - name: ECS_CANARY_CLEAN

NOTE

When you use an ELB for deployments, all listener rules that have the same target groups as configured in app.pipecd.yaml will be controlled.
- That means you need to link target groups to your listener rules before deployments.
- For more information and diagrams, see Issue#4733 [ECS] Modify ELB listener rules other than defaults without adding config.
When you use Service Connect, you cannot use Canary or Blue/Green deployment yet because Service Connect does not support the external deployment yet.
When you use AutoScaling for a service, you can disable reconciling desiredCount by following steps.
1. Create a service without defining desiredCount in the service definition file. See Restrictions of Service Definition.
2. Configure AutoScaling by yourself.

Reference

See Configuration Reference for the full configuration.