A team implementing GitOps at an organization needs to make a decision on the directory structure of the GitOps repository, effectively defining the user interface for the whole organization. However, there’s currently no established standard for such a directory structure, which means that all engineers using the GitOps repository need to spend time understanding the structure before they can begin making changes to it. In order to save the time and money for organizations implementing GitOps, I’m proposing a flexible, user-friendly and universal directory structure of a GitOps repository, which is useful to anyone implementing GitOps at an organization.
State of the Art
GitOps engineers can be inspired by a number of authors who have proposed GitOps directory structures in documentation or the blogosphere.
Kostis Kapelonis
Kostis makes a case that engineers shouldn’t use branches to store information about different GitOps environments – they should use directories instead. The main argument is that promoting a configuration change from one environment to another is never as simple as a merge. Read his blog post for his full argument.
In a separate article Kostis proposes a specific directory structure:
gitops-Kostis_Kapelonis ├── base ├── envs │ ├── integration-gpu │ ├── integration-non-gpu │ ├── load-gpu │ ├── load-non-gpu │ ├── prod-asia │ ├── prod-eu │ ├── prod-us │ ├── qa │ ├── staging-asia │ ├── staging-eu │ └── staging-us └── variants ├── asia ├── eu ├── non-prod ├── prod └── us
The Kostis’ structure has three top-level directories: base
, envs
and variants
.
The base
contains the configuration shared between environments. The envs
directory contains the environment-specific configuration information, such as the image tag, which can be promoted between environments. Finally, variants
contains mixins, i.e. groupings that don’t follow a hierarchy.
Oded Ben Ozer
Oded proposes a structure for Infrastructure-as-Code (IaC) repositories, but his ideas are also relevant to application configuration repositories:
gitops-Oded_Ben_Ozer ├── clusters │ ├── dev │ │ └── us-east4 │ │ └── c2 │ │ └── componentA │ ├── lab │ │ └── europe-west4 │ │ └── c1 │ │ └── componentA │ ├── prod │ │ ├── europe-west3 │ │ │ └── c2 │ │ │ └── componentA │ │ ├── europe-west4 │ │ │ └── c2 │ │ │ └── componentA │ │ ├── us-central1 │ │ │ └── c2 │ │ │ └── componentA │ │ ├── us-east4 │ │ │ └── c2 │ │ │ └── componentA │ │ └── us-west1 │ │ └── c2 │ │ └── componentA │ └── staging │ ├── europe-west4 │ │ └── c1 │ │ └── componentA │ └── us-central1 │ ├── c1 │ │ └── componentA │ └── c2 │ └── componentA └── workspace └── componentA
The directory structure contains two top-level directories: clusters
and workspace
.
The name of the workspace
directory is meant to indicate that this is where an engineer makes changes. Special automation will then promote those changes to specific clusters. The workspace
directory contains directories for individual components (which could be applications or other resources).
The clusters
directory contains environments (dev
, prod
, …), which contain regions (us-west1
, europe-west4
, …), which in turn contain individual clusters (c1
, c2
, …). These clusters contain directories for individual components.
Oded has implemented a GitHub bot called Telefonistka, which facilitates promoting releases between environments. Oded’s directory structure is tailored to that project. A more detailed description of the directory structure is available in the README file to Telefonistka.
Christian Hernandez
Christian proposes a directory structure containing both the infrastructure and application configuration:
gitops-Christian_Hernandez ├── bootstrap │ ├── base │ └── overlays │ └── default ├── components │ ├── applicationsets │ └── argocdproj ├── core │ ├── gitops-controller │ └── sample-admin-workload └── apps ├── bgd-blue │ ├── base │ └── overlays │ ├── dev │ ├── prod │ └── stage └── myapp ├── base └── overlays ├── dev ├── prod └── stage
The infrastructure configuration is kept in the first three top-level directories: bootstrap
, components
and core
. The application configuration is kept in the last top-level directory: apps
.
The apps
directory contains a directory per application. Each application directory contains the base
templates and the overlays
for specific environments: dev
, prod
or stage
.
Proposed Directory Structure
The new directory structure combines ideas from the authors above and aims to achieve universality, flexibility and clarity. I’m going to introduce various parts of the structure step by step.
gitops-Aleksander_Korzynski ├── org1 │ ├── app1 │ ├── app2 │ └── app3 ├── org2 └── org3
At the top level there are separate organizations – these can correspond to product teams, or any other kind of organizational units (org1
, org2
, …). Every such unit contains directories that correspond to individual applications/microservices (app1
, app2
, …). The structure under an application directory is simple, but also the most interesting:
app1 ├── unpromotables └── workspaces
There are two top-level directories under an application: unpromotables
and workspaces
. If an engineer is unsure where to make changes, the name workspaces
is meant to attract them, so let’s begin discussing that one.
Workspaces
The workspaces
directory contains configuration data that can be promoted from one environment to another.
workspaces ├── prod ├── qa └── staging
Under workspaces
there are directories containing the configuration for specific environments, such as prod
, staging
, and qa
. Each such directory can contain multiple files, such as:
image-version.yaml
– containing the tag of the application image;base-version.yaml
– containing the git repo URL and git ref, pointing to the location of the base templates for the Kubernetes manifests;settings.yaml
– any other settings.
You can use any number of files and any filenames – the above are just examples. The templating engine is supposed to be configured to take all *.yaml
files from a workspace into account.
The version of the image is kept in a separate file, so that it’s easy to automatically update the file by simply overwriting it, without using a yaml parser or regex string replacement. For example, the image-version.yaml
file may contain the following content:
image: tag: "1.2.3"
Note also that it’s important to pin specific versions in those files, such as 1.2.3
, rather than latest
, for two reasons:
- updating an image or a dependency should be an explicit change to the pinned version (that’s how GitOps works);
- let’s assume that you test a release in the
staging
environment and want to promote toprod
– you want to be sure that any config file that you copy fromstaging
toprod
has explicit version pinning, so that the deployment toprod
pulls the same images and dependencies as the ones tested instaging
.
The same applies to a dependency kept in another git repository – use an immutable git ref, preferably a version tag (such as v1.2.3
) or a commit sha (a tag is more human-readable). Be careful to avoid moving tags, such as v1
, which gets moved whenever a new minor version is released. Avoid pinning to a branch as well.
When you follow the above rules, promotion is as simple as copying the contents of one workspace to another:
git rm -rf staging cp -r qa staging git add staging
Note that in the commands above we first remove the destination directory and then copy the entire contents of the source directory – this ensures that there are no lingering files left in the destination.
There is an exception to the rule of always pinning a specific version of a dependency – if you are experimenting in a poc
cluster and never mean to promote your experiment to higher environments, you could pin your dependencies to latest
, but I discourage it, because you might forget and promote such an ambiguous reference. Instead of using latest
, it is always better to have some automation that will update your version pins whenever a new version becomes available. I recommend that even if you are merely experimenting in a poc
cluster.
Note that an engineer can make changes to the lowest environment, such as qa
, and promote to subsequent environments, but can also make a change directly to a higher-level environment, such as staging
. This follows a common pattern, where an engineer begins development in a lower-level environment and promotes to higher-level environments, but when they find a bug in a higher-level environment, they can commit the fix directly to the higher-level environment and then port the fix to the lower-level environments.
Note also that I don’t recommend storing the entire base template in the GitOps repository. That’s because it would make it difficult to share the base template between different applications. When storing independent copies of the base template in every application directory, you run the risk of significant drift between the different copies of the template.
Instead, we store the source code of the base template in a separate git repository and store a reference to a commit in that repo in base-version.yaml
in the GitOps repo. That has the advantage of easier and more transparent management of updates to the base template. An engineer can create a feature branch in the base template repository and reference the feature branch in the base-version.yaml
file for as long as they are experimenting with the new feature. Following successful tests of the feature, the branch in the base template repo can be merged and receive a tag which then can be referenced in the base-version.yaml
file and promoted between environments. Other applications can have the base-version.yaml
file updated at their own pace.
The above method of managing base templates will be described in detail in another blog post. I will then also describe how to manage a third-party Helm chart in the above manner, including an easy and transparent pulling of updates from upstream and optionally managing a queue of custom patches on top of the chart.
If your organization deploys to different environment classes (regions, clusters, …) sequentially, rather than in parallel, divide the workspace into smaller classes as well. You can achieve that by separating different parts of the name of the workspace using the @
character. The choice of the @
character is deliberate – it ensures that the names of individual classes can contain hyphens or underscores and the @
character still stands out:
workspaces ├── prod@eu-west2 ├── prod@us-east1 ├── qa └── staging
Note that we don’t create subdirectories here – this is to make sure that promotion is still as easy as copying the contents of one directory into another.
You might ask – what about the configuration that isn’t supposed to be promoted from one environment to another, such as the replica count? This is what the unpromotables
directory is for.
Unpromotables
The unpromotables
directory contains the configuration that isn’t supposed to be copied from one environment to another, because it’s strictly tied to a specific environment.
unpromotables ├── prod ├── qa └── staging
Each environment directory under unpromotables
may contain subdirectories for specific environment classes (regions, clusters, …). For example, below is a layer of subdirectories per region:
unpromotables ├── prod │ ├── eu-west2 │ └── us-east1 ├── qa │ └── us-east1 └── staging └── us-east1
Note that we create subdirectories inside unpromotables
, because that allows us to easily maintain files that are common to the entire environment (such as prod
) and keep them separate from files specific to environment classes (such as us-east1
or eu-west2
). This is different from the workspaces
directory, where we have to maintain a completely flat structure of directories in order to support promotions between workspaces.
Some organizations divide environment classes in the further subclasses. For example, they maintain separate web
and mgmt
(management) clusters. Others maintain separate pci
and non-pci
clusters to distinguish applications that process Payment Card Information (PCI) from the ones that don’t – this makes it easier to pass regulatory audits related to processing PCI. However, creating another level of subdirectories may be detrimental to user experience, who would need to traverse a deep directory structure. For that reason, I suggest that instead of creating further subdirectories, you create longer directory names, with individual parts separated with the @
character, for example:
unpromotables ├── prod │ ├── eu-west2@non-pci │ └── us-east1@non-pci ├── qa │ └── us-east1@non-pci └── staging └── us-east1@non-pci
An application is typically deployed to only one namespace, but in the rare case that an application needs to be deployed to multiple namespaces in a given cluster, there can be another suffix:
unpromotables ├── prod │ ├── eu-west2@non-pci@namespace1 │ ├── eu-west2@non-pci@namespace2 │ ├── us-east1@non-pci@namespace1 │ └── us-east1@non-pci@namespace2 ├── qa │ ├── us-east1@non-pci@namespace1 │ └── us-east1@non-pci@namespace2 └── staging ├── us-east1@non-pci@namespace1 └── us-east1@non-pci@namespace2
In case you need to define configuration data that spans an entire class, for instance: all of us-east1
, including prod
, staging
and qa
, you can create a non-hierarchical grouping, in programming commonly known as a mixin. You do this with the use of the ANY
keyword:
unpromotables ├── prod │ ├── eu-west2@mgmt │ └── us-east1@mgmt ├── qa
│ ├── eu-west2@mgmt │ └── us-east1@mgmt ├── staging │ └── us-east1@mgmt
└── ANY
├── ANY@mgmt
├── eu-west2
└── us-east1
In the above structure there is a directory called ANY
with sister directories of prod
, staging
and qa
, where we define configuration data that spans multiple environments. Inside ANY
there is an ANY@mgmt
directory that includes configuration for all mgmt
clusters. There is also a self-explanatory us-east1
directory, which could also be written as us-east1@ANY
, but we omit the trailing @ANY
, because it can be inferred.
Note that making a change to one of the ANY
mixins may result in a change to multiple environments at once, including prod
. This means that the change to prod
may have not been tested. This is why in such a case I recommend one of the following two patterns:
- Whenever an engineer makes a pull request, and before the pull request is merged, the change is deployed to an ephemeral environment, where it’s available for testing (implementing such ephemeral environments is outside the scope of this article); or
- The change should be first committed under
unpromotables/qa
, tested, committed underunpromotables/staging
, tested and in the last commit simultaneously implemented underunpromotables/ANY
and removed fromunpromotables/qa
andunpromotables/staging
(the last commit is effectively a deployment toprod
and a no-op forqa
andstaging
).
Complete Structure
Finally, the full directory structure of an application deployed to the management cluster family would look as follows:
app1 ├── unpromotables │ ├── prod │ │ ├── eu-west2@mgmt │ │ └── us-east1@mgmt │ ├── qa │ │ ├── eu-west2@mgmt │ │ └── us-east1@mgmt │ ├─── staging │ │ └── us-east1@mgmt
│ └── ANY
│ ├── eu-west2
│ └── us-east1 └── workspaces ├── prod@eu-west2 ├── prod@us-east1 ├── qa └── staging
Note that in the above example the qa
environment has two regions deployed in parallel, because there is a single workspace for the entire environment. However, the prod
environment has two regions deployed sequentially, because there is a separate workspace for each region.
Note that the list of clusters is intentionally listed under an application, rather than the reverse – applications listed under clusters. This is intentional: an engineer typically works in the context of a given application, so it will only be confusing for the engineer to be presented with a list of all clusters within an organization and require the engineer to first find the clusters relevant to a given application. It’s easier to simply find a directory for a given application and then only be presented with the list of clusters that are relevant to it.
Below is the complete directory structure, including multiple organizations, multiple apps and a sample application deployed to the pci
cluster family:
gitops-Aleksander_Korzynski ├── org1 │ ├── app1 │ │ ├── unpromotables │ │ │ ├── prod │ │ │ │ ├── eu-west2@pci │ │ │ │ └── us-east1@pci │ │ │ ├── qa
│ │ │ │ ├── eu-west2@pci │ │ │ │ └── us-east1@pci │ │ │ ├── staging │ │ │ │ └── us-east1@pci
│ │ │ └── ANY
│ │ │ ├── eu-west2
│ │ │ └── us-east1 │ │ └── workspaces │ │ ├── prod@eu-west2
│ │ ├── prod@us-east1 │ │ ├── qa │ │ └── staging │ ├── app2 │ └── app3 ├── org2 └── org3
That’s it! The structure above is supposed to be simple, but still flexible and universal. If you have feedback, please write it in the comments, as I’m looking forward to your opinion to make the proposed structure even better.
You might ask – how to implement the structure above using Kustomize or Helm? I’m going to cover that in subsequent articles.
Very good overview of all ideas from the community. I love the split between “unpromotables” the rest of settings.
Thanks!
I was also very intrigued by the “unpromotables” idea, in my current implementation I just placed these env-specific files with the “promotable” part, forcing the CI to copy non-relevant files to target folders(a dev.yaml to prod/ folder) making the whole repo less intuitive.
The downside of the unpromotables concept is that it splits the configuration to two seperate folders as “global” parameter files must be “promotables” to insure gradual rollout of changes, nevertheless, it’s very interesting concept
Thanks!