You are currently viewing Tailoring Third-Party Helm Charts: Source-Level Customizations and Effortless Updates

Tailoring Third-Party Helm Charts: Source-Level Customizations and Effortless Updates

Version 1.4

Deploying third-party Helm charts typically constrains the user to only make such customizations to the chart that the original author of the chart explicitly allowed users to make. In this article I am proposing a method of tailoring Helm charts that allows the user to freely modify the source code of the Helm chart, offering complete freedom to the user, while also providing the option to effortlessly update the Helm chart and re-apply the customizations on top of the latest version of the chart.

State of the Art

The advantages of combining Helm and GitOps (especially using Argo CD) are described in Hannah’s blog post. For a more in-depth description of the capabilities of Argo CD with regards to Helm, you can read the official Argo CD documentation.

Trevor describes three patterns for deploying Helm charts with Argo CD:

  1. Argo application pointing at a chart in a Helm repo.
  2. Argo application pointing at a chart in a Git repo containing binary Helm packages.
  3. Argo application pointing at a Kustomization file which renders a Helm chart.

The former two have the disadvantage that the user can only make such customizations that the original author of the chart explicitly allowed users to make. In contrast, the latter one allows the user to make any source-level customizations to the chart, but comes with other disadvantages: complexity when attempting to troubleshoot the chart locally and the requirement to patch the Helm chart only after it was rendered, not prior to that.

The new proposed approach attempts to combine the advantages of the patterns described above. 

Helm with git-upstream

This pattern allows the users to tailor the source code of the chart and store the customizations as git commits. Whenever a new version of the Helm chart is released, the user can update the chart and reapply the patches with the help of the git-upstream tool.

The git-upstream tool allows the user to store the patches in a form of patch queue. The patch queue is like a pair of glasses that the user looks through at the upstream Helm chart – the chart can be updated and the patches preserved.

The git-upstream tool solves a problem with vanilla git – the lack of ability to easily track an upstream project. Git allows to store custom patches on a branch, but when new changes are introduced to the upstream project, it only allows the user to merge those changes to the custom branch or to rebase the custom branch on top of the latest upstream changes and both of these methods have disadvantages. With merging, the user lacks visibility into the custom changes, as they get lost in the history of the branch. On the other hand, with rebasing, any user who previously cloned and checked out the custom branch will experience an error when running git pull. The git-upstream tool offers a third way: the rebase & weld operation, which has the advantages of both merge and rebase, without the disadvantages. For a complete description of the rebase & weld method, check out my previous blog post on the concepts behind git-upstream.

Let’s proceed with installing git-upstream.

Installation

To get to the latest version of git-upstream, I recommend installing from source. As of this writing, git-upstream requires Python 3.9, not newer, so install it on your workstation using the method appropriate for your operating system (if you are not sure how, google for “install old version of Python on <your system>”). Then run the following commandse

git clone https://opendev.org/x/git-upstream.git
cd git-upstream
python3.9 -m pip install -r requirements.txt
python3.9 -m pip install .

Repo Creation

GitHub has built-in buttons to fork a repository and to sync with upstream, but we’re not going to use them, because GitHub uses an ordinary merge or rebase, not the rebase & weld strategy. In addition, GitHub doesn’t allow to make a fork private or internal, even if you’re a user of GitHub Enterprise. So we’re going to use the command line to fork the repo.

I recommend that you use the same name for the repository as the original, but put it under a different organization. For example:
https://github.com/kubernetes/dashboard
becomes
https://github.com/akorzynski/dashboard.

Create the repository using the standard GUI, for example:

Don’t create a README file or a license for this repository. All the files will be cloned from the original open-source repo. We want a completely empty repository for our purposes, as in the screenshot below:

GitHub create new repository

Define the Environment

Define the following environment variables to correspond to your situation. With those variables defined, you will be able to copy/paste the commands from the upcoming sections, without having to modify them.

# source organization
UPSTREAM_ORG=kubernetes 

# destination organization
MY_ORG=akorzynski

# name of repository
REPO=dashboard

# path in the repo
CHART_PATH=charts/helm-chart/kubernetes-dashboard

Copy Repository Contents

Run the following commands:

# Clone the repository
git clone "https://github.com/$MY_ORG/$REPO"
cd "$REPO"

# Add the upstream repository as a new remote, 
# so that you can fetch the data from it
git remote add upstream "https://github.com/$UPSTREAM_ORG/$REPO"

# Fetch the data
git fetch --all

# Retrieve the head branch name
HEAD_BRANCH="$(git remote show upstream | \
               grep '^ *HEAD branch:' | \
               grep --only-matching '[^ ]*$' )"
              
# Push the head branch to your internal repository
git push origin --set-upstream \
  "upstream/$HEAD_BRANCH:refs/heads/$HEAD_BRANCH"

# Push the data to your internal repository, 
# prefixing branches with 'upstream/'
# (command based on git-upstream's documentation)
git for-each-ref refs/remotes/upstream --format "%(refname:short)" | \
  sed -e 's@\(upstream/\(.*\)\)$@\1:refs/heads/upstream/\2@' | \
  xargs git push --tags origin

# Checkout the head branch
git checkout --track "origin/$HEAD_BRANCH"

Download Helm Dependencies

Before we deploy the Helm chart, we need to download the dependencies. We’re going to store them in git as source code, which will allow us to have good visibility into the changes to the source code of the dependencies.

The Helm dependency update will be just another git commit on top of the Helm chart, but with a special Change-Id string added to the bottom of the commit message. The Change-Id will allow us to track that commit, so that we can replace it with a new one the next time we update the dependencies.

The Change-Id string looks like the following: Change-Id: I01078eafe753956fbcddc3895c61f18c922f804a. The Change-Id is a SHA prefixed with the letter I, in order to distinguish it from the Git SHA. The concept of the Change-Id was first introduced by the Gerrit code review tool, but using Gerrit is not required in this pattern, because the git-upstream tool implements support for Change-Ids.

Run the following commands to update Helm dependencies:

# Define the Change-Id
CHANGE_ID="I01078eafe753956fbcddc3895c61f18c922f804a"

# Go to the directory containing the chart
cd "$CHART_PATH"

# Remove all dependencies
git rm -rf charts/*

# Download the dependencies as *.tgz archives
helm dependency update

# Uncompress the archives
find charts -name '*.tgz' -exec tar -C charts -zxvf '{}' ';'

# Remove the archives
rm -f charts/*.tgz

# Add the updated dependencies to git index.
# We use --force, because the original maintainer
# may have included these files in .gitignore.
git add --force charts Chart.lock

# Commit the dependencies, adding a Change-Id string to the commit message.
git commit -m "helm dependency update" -m "Change-Id: $CHANGE_ID"

Commit your change

Now you can develop your customization to the Helm chart using your favorite editor. Then commit:

git commit -a -m "My change"

Deploy

At this point we reach a snag in the existing GitOps ecosystem: Helm tools typically expect to deploy charts from binary tarballs, not the source code of the chart in text form. Although there is a plug-in called helm-git, it expects the user to store binary tarballs in git, as the plug-in doesn’t read Helm source code. The native Helm support in Argo CD can also only read binary tarballs, stored either in a Helm repository or in Git. We need something else – the support for deploying a chart directly from its source code stored in Git.

The ideal experience for us would be to have Helm deploy a chart using only the following information: 

  • the URL to the Git repository,
  • the Git ref (tag, commit SHA or branch),
  • the path to the source code of the chart within the repository.

At some point in the future it would be worthwhile to extend both Helm and Argo CD to allow for specifying the location of the chart using the information above. Until then, we have two workarounds:

  1. pre-render the chart using helm template, save the rendered Kubernetes manifests in another Git repository and point Argo CD to those rendered manifests
  2. publish the Helm chart in binary form to a Helm repository and point Argo CD to that repository

I have implemented the former for a client and it requires significant custom scripting, which is outside the scope of this article. The latter option is easier to implement and there are plenty of online resources explaining how to publish a Helm chart. The instructions are dependent on the kind of Helm repository that is available to you, so I won’t cover those details in this article. 

Import Upstream

What happens when the upstream Helm chart changes and you want to update it? Here is where git-upstream comes into play.

First, find the branch you want to import (it might also be a tag, but I won’t cover this case here). Usually it’s upstream/$HEAD_BRANCH. Then run the following commands:

# Remove the upstream remote, so that it doesn't interfere
# with the commands below by creating ambiguity 
git remote remove upstream

# Define a variable with the upstream branch to import
UPSTREAM_BRANCH="upstream/$HEAD_BRANCH"

# Checkout the branch
git checkout "$UPSTREAM_BRANCH"

Now re-run the commands listed in the Download Helm Dependencies section. Then run the commands below:

# Checkout your head branch
git checkout "$HEAD_BRANCH"

# Create an import branch without the last "welding merge"
git upstream import --no-merge --force --import-branch "import/$UPSTREAM_BRANCH" "$UPSTREAM_BRANCH"

# Switch to the import branch
git checkout "import/$UPSTREAM_BRANCH"

# Optional: you can use the interactive rebase
# to alter the patch queue as required
git rebase --interactive "import/$UPSTREAM_BRANCH-base"

# Return to head branch
git checkout "$HEAD_BRANCH"

# Create the final "welding merge"
git upstream import --finish --import-branch "import/$UPSTREAM_BRANCH" "$UPSTREAM_BRANCH"

Conclusion

We are done! You’ve now been through the steps to tailor a third-party Helm chart. I have a vision that Helm charts could be routinely distributed in source using the method above, rather than in binary form. Let me know what you think in the comments section.

Leave a Reply