You are currently viewing How to Structure Directories in a GitOps Repository for the Best User-Friendliness and Flexibility

How to Structure Directories in a GitOps Repository for the Best User-Friendliness and Flexibility

Version 1.1

A team implementing GitOps at an organization needs to make a decision on the directory structure of the GitOps repository, effectively defining the user interface for the whole organization. However, there’s currently no established standard for such a directory structure, which means that all engineers using the GitOps repository need to spend time understanding the structure before they can begin making changes to it. In order to save the time and money for organizations implementing GitOps, I’m proposing a flexible, user-friendly and universal directory structure of a GitOps repository, which is useful to anyone implementing GitOps at an organization.

State of the Art

GitOps engineers can be inspired by a number of authors who have proposed GitOps directory structures in documentation or the blogosphere.

Kostis Kapelonis

Kostis makes a case that engineers shouldn’t use branches to store information about different GitOps environments – they should use directories instead. The main argument is that promoting a configuration change from one environment to another is never as simple as a merge. Read his blog post for his full argument. 

In a separate article Kostis proposes a specific directory structure:

gitops-Kostis_Kapelonis
├── base
├── envs
│   ├── integration-gpu
│   ├── integration-non-gpu
│   ├── load-gpu
│   ├── load-non-gpu
│   ├── prod-asia
│   ├── prod-eu
│   ├── prod-us
│   ├── qa
│   ├── staging-asia
│   ├── staging-eu
│   └── staging-us
└── variants
    ├── asia
    ├── eu
    ├── non-prod
    ├── prod
    └── us

The Kostis’ structure has three top-level directories: base, envs and variants

The base contains the configuration shared between environments. The envs directory contains the environment-specific configuration information, such as the image tag, which can be promoted between environments. Finally, variants contains mixins, i.e. groupings that don’t follow a hierarchy.

Oded Ben Ozer

Oded proposes a structure for Infrastructure-as-Code (IaC) repositories, but his ideas are also relevant to application configuration repositories:

gitops-Oded_Ben_Ozer
├── clusters
│   ├── dev
│   │   └── us-east4
│   │       └── c2
│   │           └── componentA
│   ├── lab
│   │   └── europe-west4
│   │       └── c1
│   │           └── componentA
│   ├── prod
│   │   ├── europe-west3
│   │   │   └── c2
│   │   │       └── componentA
│   │   ├── europe-west4
│   │   │   └── c2
│   │   │       └── componentA
│   │   ├── us-central1
│   │   │   └── c2
│   │   │       └── componentA
│   │   ├── us-east4
│   │   │   └── c2
│   │   │       └── componentA
│   │   └── us-west1
│   │       └── c2
│   │           └── componentA
│   └── staging
│       ├── europe-west4
│       │   └── c1
│       │       └── componentA
│       └── us-central1
│           ├── c1
│           │   └── componentA
│           └── c2
│               └── componentA
└── workspace
    └── componentA

The directory structure contains two top-level directories: clusters and workspace.

The name of the workspace directory is meant to indicate that this is where an engineer makes changes. Special automation will then promote those changes to specific clusters. The workspace directory contains directories for individual components (which could be applications or other resources).

The clusters directory contains environments (dev, prod, …), which contain regions (us-west1, europe-west4, …), which in turn contain individual clusters (c1, c2, …). These clusters contain directories for individual components.

Oded has implemented a GitHub bot called Telefonistka, which facilitates promoting releases between environments. Oded’s directory structure is tailored to that project. A more detailed description of the directory structure is available in the README file to Telefonistka.

Christian Hernandez

Christian proposes a directory structure containing both the infrastructure and application configuration:

gitops-Christian_Hernandez
├── bootstrap
│   ├── base
│   └── overlays
│       └── default
├── components
│   ├── applicationsets
│   └── argocdproj
├── core
│   ├── gitops-controller
│   └── sample-admin-workload
└── apps
    ├── bgd-blue
    │   ├── base
    │   └── overlays
    │       ├── dev
    │       ├── prod
    │       └── stage
    └── myapp
        ├── base
        └── overlays
            ├── dev
            ├── prod
            └── stage

The infrastructure configuration is kept in the first three top-level directories: bootstrap, components and core. The application configuration is kept in the last top-level directory: apps

The apps directory contains a directory per application. Each application directory contains the base templates and the overlays for specific environments: dev, prod or stage

Proposed Directory Structure

The new directory structure combines ideas from the authors above and aims to achieve universality, flexibility and clarity. I’m going to introduce various parts of the structure step by step. 

gitops-Aleksander_Korzynski
├── org1
│   ├── app1
│   ├── app2
│   └── app3

├── org2
└── org3

At the top level there are separate organizations – these can correspond to product teams, or any other kind of organizational units (org1, org2, …). Every such unit contains directories that correspond to individual applications/microservices (app1, app2, …). The structure under an application directory is simple, but also the most interesting:

app1
├── unpromotables
└── workspaces

There are two top-level directories under an application: unpromotables and workspaces. If an engineer is unsure where to make changes, the name workspaces is meant to attract them, so let’s begin discussing that one.

Workspaces

The workspaces directory contains configuration data that can be promoted from one environment to another. 

workspaces
├── prod
├── qa
└── staging

Under workspaces there are directories containing the configuration for specific environments, such as prod, staging, and qa. Each such directory can contain multiple files, such as:

  • image-version.yaml – containing the tag of the application image;
  • base-version.yaml – containing the git repo URL and git ref, pointing to the location of the base templates for the Kubernetes manifests;
  • settings.yaml – any other settings.

You can use any number of files and any filenames – the above are just examples. The templating engine is supposed to be configured to take all *.yaml files from a workspace into account. 

The version of the image is kept in a separate file, so that it’s easy to automatically update the file by simply overwriting it, without using a yaml parser or regex string replacement. For example, the image-version.yaml file may contain the following content:

image:
  tag: "1.2.3"

Note also that it’s important to pin specific versions in those files, such as 1.2.3, rather than latest, for two reasons:

  1. updating an image or a dependency should be an explicit change to the pinned version (that’s how GitOps works);
  2. let’s assume that you test a release in the staging environment and want to promote to prod – you want to be sure that any config file that you copy from staging to prod has explicit version pinning, so that the deployment to prod pulls the same images and dependencies as the ones tested in staging

The same applies to a dependency kept in another git repository – use an immutable git ref, preferably a version tag (such as v1.2.3) or a commit sha (a tag is more human-readable). Be careful to avoid moving tags, such as v1, which gets moved whenever a new minor version is released. Avoid pinning to a branch as well.

When you follow the above rules, promotion is as simple as copying the contents of one workspace to another:

git rm -rf staging
cp -r qa staging
git add staging

Note that in the commands above we first remove the destination directory and then copy the entire contents of the source directory – this ensures that there are no lingering files left in the destination. 

There is an exception to the rule of always pinning a specific version of a dependency – if you are experimenting in a poc cluster and never mean to promote your experiment to higher environments, you could pin your dependencies to latest, but I discourage it, because you might forget and promote such an ambiguous reference. Instead of using latest, it is always better to have some automation that will update your version pins whenever a new version becomes available. I recommend that even if you are merely experimenting in a poc cluster. 

Note that an engineer can make changes to the lowest environment, such as qa, and promote to subsequent environments, but can also make a change directly to a higher-level environment, such as staging. This follows a common pattern, where an engineer begins development in a lower-level environment and promotes to higher-level environments, but when they find a bug in a higher-level environment, they can commit the fix directly to the higher-level environment and then port the fix to the lower-level environments. 

Note also that I don’t recommend storing the entire base template in the GitOps repository. That’s because it would make it difficult to share the base template between different applications. When storing independent copies of the base template in every application directory, you run the risk of significant drift between the different copies of the template.

Instead, we store the source code of the base template in a separate git repository and store a reference to a commit in that repo in base-version.yaml in the GitOps repo. That has the advantage of easier and more transparent management of updates to the base template. An engineer can create a feature branch in the base template repository and reference the feature branch in the base-version.yaml file for as long as they are experimenting with the new feature. Following successful tests of the feature, the branch in the base template repo can be merged and receive a tag which then can be referenced in the base-version.yaml file and promoted between environments. Other applications can have the base-version.yaml file updated at their own pace.

The above method of managing base templates will be described in detail in another blog post. I will then also describe how to manage a third-party Helm chart in the above manner, including an easy and transparent pulling of updates from upstream and optionally managing a queue of custom patches on top of the chart.

If your organization deploys to different environment classes (regions, clusters, …) sequentially, rather than in parallel, divide the workspace into smaller classes as well. You can achieve that by separating different parts of the name of the workspace using the @ character. The choice of the @ character is deliberate – it ensures that the names of individual classes can contain hyphens or underscores and the @ character still stands out: 

workspaces
├── prod@eu-west2
├── prod@us-east1
├── qa
└── staging

Note that we don’t create subdirectories here – this is to make sure that promotion is still as easy as copying the contents of one directory into another.

You might ask – what about the configuration that isn’t supposed to be promoted from one environment to another, such as the replica count? This is what the unpromotables directory is for.

Unpromotables

The unpromotables directory contains the configuration that isn’t supposed to be copied from one environment to another, because it’s strictly tied to a specific environment.

unpromotables
├── prod
├── qa
└── staging

Each environment directory under unpromotables may contain subdirectories for specific environment classes (regions, clusters, …). For example, below is a layer of subdirectories per region:

unpromotables
├── prod
│   ├── eu-west2
│   └── us-east1
├── qa
│   └── us-east1
└── staging
    └── us-east1

Note that we create subdirectories inside unpromotables, because that allows us to easily maintain files that are common to the entire environment (such as prod) and keep them separate from files specific to environment classes (such as us-east1 or eu-west2). This is different from the workspaces directory, where we have to maintain a completely flat structure of directories in order to support promotions between workspaces.

Some organizations divide environment classes in the further subclasses. For example, they maintain separate web and mgmt (management) clusters. Others maintain separate pci and non-pci clusters to distinguish applications that process Payment Card Information (PCI) from the ones that don’t – this makes it easier to pass regulatory audits related to processing PCI. However, creating another level of subdirectories may be detrimental to user experience, who would need to traverse a deep directory structure. For that reason, I suggest that instead of creating further subdirectories, you create longer directory names, with individual parts separated with the @ character, for example:

unpromotables
├── prod
│   ├── eu-west2@non-pci
│   └── us-east1@non-pci
├── qa
│   └── us-east1@non-pci
└── staging
    └── us-east1@non-pci

An application is typically deployed to only one namespace, but in the rare case that an application needs to be deployed to multiple namespaces in a given cluster, there can be another suffix:

unpromotables
├── prod
│   ├── eu-west2@non-pci@namespace1
│   ├── eu-west2@non-pci@namespace2
│   ├── us-east1@non-pci@namespace1
│   └── us-east1@non-pci@namespace2
├── qa
│   ├── us-east1@non-pci@namespace1
│   └── us-east1@non-pci@namespace2
└── staging
    ├── us-east1@non-pci@namespace1
    └── us-east1@non-pci@namespace2

In case you need to define configuration data that spans an entire class, for instance: all of us-east1, including prod, staging and qa, you can create a non-hierarchical grouping, in programming commonly known as a mixin. You do this with the use of the ANY keyword: 

unpromotables
├── prod
│   ├── eu-west2@mgmt
│   └── us-east1@mgmt
├── qa
│   ├── eu-west2@mgmt │   └── us-east1@mgmt ├── staging │ └── us-east1@mgmt
└── ANY
├── ANY@mgmt
├── eu-west2
└── us-east1

In the above structure there is a directory called ANY with sister directories of prod, staging and qa, where we define configuration data that spans multiple environments. Inside ANY there is an ANY@mgmt directory that includes configuration for all mgmt clusters. There is also a self-explanatory us-east1 directory, which could also be written as us-east1@ANY, but we omit the trailing @ANY, because it can be inferred. 

Note that making a change to one of the ANY mixins may result in a change to multiple environments at once, including prod. This means that the change to prod may have not been tested. This is why in such a case I recommend one of the following two patterns:

  1. Whenever an engineer makes a pull request, and before the pull request is merged, the change is deployed to an ephemeral environment, where it’s available for testing (implementing such ephemeral environments is outside the scope of this article); or
  2. The change should be first committed under unpromotables/qa, tested, committed under unpromotables/staging, tested and in the last commit simultaneously implemented under unpromotables/ANY and removed from unpromotables/qa and unpromotables/staging (the last commit is effectively a deployment to prod and a no-op for qa and staging).

Complete Structure

Finally, the full directory structure of an application deployed to the management cluster family would look as follows:

app1
├── unpromotables
│   ├── prod
│   │   ├── eu-west2@mgmt
│   │   └── us-east1@mgmt
│   ├── qa
│   │   ├── eu-west2@mgmt
│   │   └── us-east1@mgmt
│   ├─── staging
│   │   └── us-east1@mgmt
│ └── ANY
│   ├── eu-west2
│   └── us-east1 └── workspaces ├── prod@eu-west2 ├── prod@us-east1 ├── qa └── staging

Note that in the above example the qa environment has two regions deployed in parallel, because there is a single workspace for the entire environment. However, the prod environment has two regions deployed sequentially, because there is a separate workspace for each region.

Note that the list of clusters is intentionally listed under an application, rather than the reverse – applications listed under clusters. This is intentional: an engineer typically works in the context of a given application, so it will only be confusing for the engineer to be presented with a list of all clusters within an organization and require the engineer to first find the clusters relevant to a given application. It’s easier to simply find a directory for a given application and then only be presented with the list of clusters that are relevant to it.

Below is the complete directory structure, including multiple organizations, multiple apps and a sample application deployed to the pci cluster family:

gitops-Aleksander_Korzynski
├── org1
│   ├── app1
│   │   ├── unpromotables
│   │   │   ├── prod
│   │   │   │   ├── eu-west2@pci
│   │   │   │   └── us-east1@pci
│   │   │   ├── qa
│ │ │ │ ├── eu-west2@pci │ │ │ │ └── us-east1@pci │ │ │ ├── staging │ │ │ │ └── us-east1@pci
│ │ │ └── ANY
│ │ │ ├── eu-west2
│ │ │ └── us-east1 │ │ └── workspaces │ │ ├── prod@eu-west2
│ │ ├── prod@us-east1 │ │ ├── qa │ │ └── staging │ ├── app2 │ └── app3 ├── org2 └── org3

That’s it! The structure above is supposed to be simple, but still flexible and universal. If you have feedback, please write it in the comments, as I’m looking forward to your opinion to make the proposed structure even better.

You might ask – how to implement the structure above using Kustomize or Helm? I’m going to cover that in subsequent articles.

This Post Has 4 Comments

  1. Kostis

    Very good overview of all ideas from the community. I love the split between “unpromotables” the rest of settings.

  2. Oded Ben Ozer

    I was also very intrigued by the “unpromotables” idea, in my current implementation I just placed these env-specific files with the “promotable” part, forcing the CI to copy non-relevant files to target folders(a dev.yaml to prod/ folder) making the whole repo less intuitive.
    The downside of the unpromotables concept is that it splits the configuration to two seperate folders as “global” parameter files must be “promotables” to insure gradual rollout of changes, nevertheless, it’s very interesting concept

Leave a Reply