PipeCD – LFX Mentorship

Blog: Building the Kubernetes Multi-Cluster Plugin for PipeCD — LFX Mentorship

Fri, 10 Apr 2026 00:00:00 +0000

If you had told me last year that I would be working with Kubernetes and all things clusters, deployments and service meshes, I would have brushed it off. I am truly grateful for the journey thus far.

Earlier last month, I got accepted as an LFX Mentee for Term 1 of this calendar year. For me it is such a big deal, given my background, and how much effort has been put in behind the scenes to get to this stage.

I’m currently a mentee in the LFX Mentorship program working on PipeCD, an open-source GitOps continuous delivery platform. For the past four weeks, I’ve been building out the kubernetes_multicluster plugin specifically implementing the deployment pipeline stages that handle canary, primary and baseline deployments across multiple clusters.

What is PipeCD and what is this plugin?

PipeCD is an open-source GitOps CD platform that manages deployments across different infrastructure targets like Kubernetes, ECS, Terraform, Lambda and more. Each target type has a plugin that knows how to deploy to it.

The kubernetes_multicluster plugin is for teams running the same application across multiple Kubernetes clusters say US, EU and Asia and needing all of them to stay in sync through a single pipeline. Rolling out a new version across clusters one at a time, manually, with no coordination, is error-prone and slow. The plugin lets you define one pipeline that runs across every cluster at the same time, with canary and baseline checks before anything hits production.

Progressive Delivery and Why These Stages Exist

Before a new version reaches all users, it goes through stages. A canary sends a small slice of traffic to the new version first. A baseline runs the current version at the same scale so you have a fair comparison. Primary is the actual promotion. Clean stages remove the temporary resources when you’re done.

This pattern is called progressive delivery, because you roll out gradually, check things look good, then commit. If something looks wrong at the canary stage, you stop there. Nothing has touched production yet.

The kubernetes_multicluster plugin runs all of this across every cluster at the same time. One pipeline, every cluster, same stages.

A full pipeline looks like this:

stages:
  - name: K8S_CANARY_ROLLOUT
  - name: K8S_BASELINE_ROLLOUT
  - name: K8S_TRAFFIC_ROUTING
  - name: K8S_PRIMARY_ROLLOUT
  - name: K8S_CANARY_CLEAN
  - name: K8S_BASELINE_CLEAN

Each of these is a stage I built. The sections below go through what each one does.

What I Built

K8S_CANARY_ROLLOUT

The canary stage deploys the new version of your app as a small slice alongside the existing production deployment. If your app normally runs 3 pods, canary might spin up 1 pod (or 20%) of the new version enough to catch problems without affecting most users.

It loads manifests from Git, creates copies of all workloads with a -canary suffix, scales them down to the configured replica count, adds a pipecd.dev/variant=canary label, and applies them to every target cluster in parallel. The original deployment is never touched this stage only ever adds resources.

K8S_CANARY_CLEAN

Once the canary window is over, whether you promoted or rolled back, the canary pods are just sitting in every cluster doing nothing. K8S_CANARY_CLEAN removes them.

It finds all resources with the label pipecd.dev/variant=canary for the application and deletes them in order: Services first, then Deployments, then everything else. The order matters as you don’t want to remove the Deployment while the Service is still sending traffic to it.

One thing worth noting: the query is scoped strictly to canary-labelled resources. Even if something goes wrong in the deletion logic, it cannot touch primary resources.

K8S_PRIMARY_ROLLOUT

After the canary looks good, you promote the new version to primary, the workload actually serving all your users. This stage takes the manifests from Git, adds the pipecd.dev/variant=primary label, and applies them across all clusters in parallel.

It also has a prune option: after applying, it checks what’s currently running in the cluster against what was just applied, and deletes anything that’s no longer in Git. Useful when you remove a resource from your manifests and want the cluster to reflect that.

K8S_BASELINE_ROLLOUT

This one took me a while to understand and it is the stage I find most interesting to explain as well.

When you’re running a canary, the natural thing is to compare it against primary. The issue is that’s not a fair comparison primary is handling far more traffic than canary, under different conditions.

Baseline gives you a fairer comparison. You take the current version (not the new one) and run it at the same scale as canary. Now your cluster has:

simple             2/2   ← production, current version
simple-canary      1/1   ← new version, being tested
simple-baseline    1/1   ← current version at canary scale

You compare canary vs baseline, same number of pods, same traffic conditions. If canary is worse, it’s obvious.

The key difference from every other rollout stage is one line of code. Canary and primary load manifests from the new Git commit (TargetDeploymentSource). Baseline loads from what’s currently running (RunningDeploymentSource):

// canary.go — new version
manifests, err := p.loadManifests(ctx, ..., &input.Request.TargetDeploymentSource, ...)

// baseline.go — current version
manifests, err := p.loadManifests(ctx, ..., &input.Request.RunningDeploymentSource, ...)

K8S_BASELINE_CLEAN

Once the analysis is done, baseline resources get cleaned up the same way as canary find everything labelled pipecd.dev/variant=baseline and delete it in order. No configuration needed. It doesn’t matter whether createService: true was set during rollout, it finds whatever is there and removes it.

K8S_TRAFFIC_ROUTING

Canary and baseline pods exist in the cluster but get no traffic until this stage runs. Without it, you’re analysing pods that nobody is actually hitting. This stage is what sends real user traffic to them.

Two methods are supported:

PodSelector (no service mesh needed): changes the Kubernetes Service selector to point at one variant. All-or-nothing 100% to canary or 100% back to primary.

Istio: updates VirtualService route weights to split traffic across all three variants at once for example, primary 80%, canary 10%, baseline 10%. Also supports editableRoutes to limit which named routes the stage is allowed to modify.

One small thing I added on top of the traffic routing stage: per-route logging. When the stage runs, it now logs each route it processes whether it was skipped (because it’s not in editableRoutes) or updated with new weights. Before this, the log just said “Successfully updated traffic routing” with no detail. Now you can see exactly which routes changed and to what percentages, which is useful when debugging a misconfigured VirtualService.

Something I Found Interesting

The thing that surprised me was how errgroup handles running across multiple clusters without much extra code.

Every stage needs to run against N clusters, not one. A simple for-loop would run them one at a time slow, and if cluster 2 fails you don’t find out until cluster 1 is already done.

errgroup runs all clusters at the same time and returns the first error:

eg, ctx := errgroup.WithContext(ctx)
for _, tc := range targetClusters {
    tc := tc
    eg.Go(func() error {
        return canaryRollout(ctx, tc.deployTarget, ...)
    })
}
return eg.Wait()

All clusters run in parallel. If any one fails, the stage fails immediately. The same pattern is used across every stage, so adding a new stage is mostly just writing the per-cluster logic the concurrency part is already solved.

What’s Next

The next piece is DetermineStrategy, that is the logic that decides what kind of deployment to trigger based on what changed in Git. After that, livestate drift detection so PipeCD can flag when a cluster has drifted from what Git says it should be.

To get involved, check out the PipeCD project and come join us on Slack.

Blog: My First 30 days as an LFX Mentee with PipeCD

Wed, 08 Apr 2026 00:00:00 +0000

A month ago, I started my journey as an LFX Mentee with PipeCD.

Coming from a non-technical background, the cloud native ecosystem is relatively new to me; I’ve been outside looking in. Right now, I’m working to establish a social media presence for PipeCD, create content covering v1 features, plugin development, and walkthrough videos that make the project easier to adopt.

To effectively do that, my technical knowledge needs to be sharpened. So I’m learning Linux basics and Kubernetes to enable me to understand how PipeCD v1 works, the plugin architecture, and migration from v0 to v1.

This is my first contact with cloud native technology. I aim to document my journey, including what I’m working on, learning, and everything in between

Getting started with LFX and PipeCD

The LFX Mentorship program provides opportunities to contribute to open source projects while learning from experienced maintainers.

Through this program, I joined PipeCD as a mentee for the Community building, Technical content, and Social media growth project.

PipeCD is an open-source continuous delivery solution built around GitOps to enable engineers deploy multiple application kinds across multi-cloud environments.

With the release of PipeCD v1, the project is evolving to be more flexible and extensible through its plugin-based architecture. This makes it easier for teams to integrate PipeCD into their existing workflows and environments, rather than having to change everything to adopt it.

At a high level, PipeCD connects to your application and deployment configuration, then manages how changes are rolled out; gradually, all at once, or in controlled stages. It provides teams with visibility into deployments, along with the ability to monitor, verify, and roll back changes as needed.

If you’re a DevOps or platform engineer seeking more control and flexibility in production, consider PipeCD. It’s designed for safer, more controlled continuous delivery, allowing you to ship fast with confidence.

The [documentation] (https://pipecd.dev/docs-v1.0.x/) is a great place to start and get a picture of how it all fits together.

What I’ve been up to:

My focus in the past month has been getting fundamental knowledge on:

Linux and Kubernetes
Computer networking (The OSI model)
Encryption and decryption
Cloud compute
IaaS, SaaS, PaaS, FaaS, and more

I’m intentionally building this foundation because understanding these concepts is key to fully grasping how PipeCD works.

The purpose is not only to learn, but to communicate this clearly to developers and create more detailed videos.

Beyond learning, I actively engage with the community, welcoming new members, reviewing issues, and PRs on GitHub. Essentially, creating a feedback loop between contributors and maintainers

I also create content, post regularly on social media platforms, and host community meetings.

PipeCD community is diverse and inclusive. The Maintainers and contributors are kind, helpful, and supportive. Early on, even before I got selected as a mentee, I got comfortable:

following conversations on Slack
Understanding community needs and
familiarizing myself with the project structure

It was less about immediate contribution and more about learning how things work.

Reflections and next step:

The past month has been a really interesting experience; quite challenging, but very insightful.

A perfect blend of what I’m good at (community work) and what I want to learn (cloud native technologies). In the coming days, I will set up PipeCD locally on my machine and explore the technicalities.

There’s still a lot I’m figuring out, work to do, and I’m up for a challenge.

Progress isn’t yet about big milestones but small, consistent steps. building systems and structures to help me better navigate this space.

As I grow into my role as a PipeCD community manager, concepts are gradually becoming clearer, and I’m learning how everything connects. My goal is to ultimately become a maintainer long-term.

PipeCD welcomes contributors from all around the world, irrespective of your background. Technical or non-technical, there’s room to learn, work, and make an impact.

To get involved, check out the PipeCD project and come join us on [Slack] (https://app.slack.com/client/T08PSQ7BQ/C01B27F9T0X)

Kindly stay connected by following PipeCD on [LinkedIn] (https://www.linkedin.com/company/pipecd/) and subscribing to our YouTube Channel.

See you in my next drop.

Gloriah.