Install Grafana-operator using Flux and Kustomize

How to install grafana-operator using Flux and Kustomize

By Edvin ‘NissesSenap’ Norling | Wednesday, March 29, 2023

As a part of grafana-operator v5.0.0-rc1 we introduce Kustomize as a way of installing the operator.

To showcase this new feature, I thought why not use GitOps?

GitOps is a rather well-used term nowadays, but you can summarize it in 4 steps.

Declarative
Versioned and immutable
Pulled automatically
Continuously reconciled

To find out more about GitOps look at the CNCF GitOps working groups documentation.

In this case I have decided to use Flux to deploy grafana-operator through GitOps, but there are other options. For example ArgoCD which is also a graduated CNCF project just like Flux.

This blog’s focus is to showcase how you can use Kustomize to install the grafana-operator, I will take many shortcuts to keep this blog simple, read the official Flux documentation for best practices.

What we will do?

We will install the grafana-operator using Flux and to manage our grafana instance and dashboard.

Assuming that you follow the instructions of this blog, your Flux fleet-infra repository will look something like this.

├── clusters
│   └── my-cluster
│       ├── flux-system
│       │   ├── gotk-components.yaml
│       │   ├── gotk-sync.yaml
│       │   └── kustomization.yaml
│       ├── grafana
│       │   ├── dashboard.yaml
│       │   ├── grafana.yaml
│       │   └── kustomization.yaml
│       ├── grafana-operator.yaml
│       └── grafana.yaml

You can find all the files available to copy in the grafana-operator repository, or you can just copy paste them from the blog.

Prerequisite

In this example, I will use Kind as my cluster because I think it’s simple, but the walkthrough should work with any Kubernetes solution. I will use GitHub for my GitOps repository but Flux supports multiple source control management (SCM) providers.

So what will you need?

Kind
Flux cli
github repository
github token

Setup cluster

Create a Kind cluster with ingress support.

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

When the cluster is up, install ingress-nginx.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
# Wait for ingress-nginx to become ready
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=90s

Bootstrap Flux

Read the official documentation on how to bootstrap your Flux setup, as I mentioned earlier, I will use github.

Before bootstrapping your cluster, you need to create a github PAT(Personal Access Token) with enough access.

I have been unable to find exactly what the minimal access needed for the token, so I just created a PAT with all the access. You should not do the same.

export GITHUB_TOKEN=github_pat_soooLongAndSecretPAT
flux bootstrap github --owner=<github-username> --repository=fleet-infra --private=false --personal=true --path=clusters/my-cluster

This will create a repository called fleet-infra in your GitHub account, install Flux to your cluster and setup an autosync to GitHub. You should now be able to view Flux running in the fleet-infra namespace.

Install operator using Flux and Kustomize

We are using Flux to package our Kustomize files through OCI, and they are built and released just as our helm solution.

There are two ways of installing the operator, either with namespace access or cluster access, take a look at our documentation for more information. Add the following file to your Flux repo under clusters/my-cluster/grafana-operator.yaml.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: grafana-operator
  namespace: flux-system
spec:
  interval: 10m
  url: oci://ghcr.io/grafana/kustomize/grafana-operator
  ref:
    tag: v5.0.0-rc3
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: grafana-operator
  namespace: flux-system
spec:
  interval: 10m
  targetNamespace: grafana
  images:
    - name: ghcr.io/grafana/grafana-operator
      newTag: v5.0.0-rc3
  prune: true
  sourceRef:
    kind: OCIRepository
    name: grafana-operator
  path: ./overlays/namespace_scoped

Depending on if you want to install cluster scoped or namespace scoped, you need to change the path.

Install operator using Kustomize

If you want to install the grafana-operator without using GitOps, you can also download the generated artifact and install it manually. For example, you can run the following Flux command to download the artifact and unpack it. Then you can run a normal kubectl apply command.

flux pull artifact oci://ghcr.io/grafana/kustomize/grafana-operator:v5.7.0 -output ./grafana-opreator

But of course we recommend that you manage your grafana-operator installation through your GitOps solution, no matter if it’s Flux or some other solution.

Install Grafana

Okay great, we got grafana-operator installed, but don’t we actually want to install a Grafana instance as well?

Let’s setup a Grafana instance and some dashboard trough code.

Since we already setup our kind cluster with ingress-nginx I will use one of our basic HTTP examples.

Let’s not make this more complicated than it has to be, you should have your secrets in some KMS/vault/sealed-secrets, SOAP or similar solution, but definitely not checked in to your GitOps repository non-encrypted. But this is an example, so for this time let’s create our admin and password secrets manual.

cat <<EOF | kubectl apply -f -
kind: Secret
apiVersion: v1
metadata:
  name: credentials
  namespace: grafana
stringData:
  GF_SECURITY_ADMIN_PASSWORD: secret
  GF_SECURITY_ADMIN_USER: root
type: Opaque
EOF

Now create our Grafana instance together with a very basic dashboard.

The Grafana instance should be stored in clusters/my-cluster/grafana/grafana.yaml

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  namespace: grafana
  labels:
    dashboards: "grafana"
spec:
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: grafana
              env:
                - name: GF_SECURITY_ADMIN_USER
                  valueFrom:
                    secretKeyRef:
                      key: GF_SECURITY_ADMIN_USER
                      name: credentials
                - name: GF_SECURITY_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: GF_SECURITY_ADMIN_PASSWORD
                      name: credentials
  ingress:
    spec:
      ingressClassName: nginx
      rules:
        - host: grafana.127.0.0.1.nip.io
          http:
            paths:
              - backend:
                  service:
                    name: grafana-service
                    port:
                      number: 3000
                path: /
                pathType: Prefix

The Grafana dashboard should be stored in clusters/my-cluster/grafana/dashboard.yaml

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-sample
  namespace: grafana
spec:
  resyncPeriod: 30s
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  json: >
    {
      "id": null,
      "title": "Simple Dashboard",
      "tags": [],
      "style": "dark",
      "timezone": "browser",
      "editable": true,
      "hideControls": false,
      "graphTooltip": 1,
      "panels": [],
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {
        "time_options": [],
        "refresh_intervals": []
      },
      "templating": {
        "list": []
      },
      "annotations": {
        "list": []
      },
      "refresh": "5s",
      "schemaVersion": 17,
      "version": 0,
      "links": []
    }

We also need a Kustomization file to clusters/my-cluster/grafana/kustomization.yaml to help Flux to find the files.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - grafana.yaml
  - dashboard.yaml

And finally create the Flux kustomization file (yes, the naming is a bit confusing). Lets store it in clusters/my-cluster/grafana.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: grafana
  namespace: flux-system
spec:
  force: false
  dependsOn:
    - name: grafana-operator # Depends on the grafana-operator being synced
  healthChecks: # Check that grafana-deployment comes up
    - apiVersion: apps/v1
      kind: Deployment
      name: grafana-deployment
      namespace: grafana
  interval: 10m0s
  targetNamespace: grafana
  path: ./clusters/my-cluster/grafana
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

After pushing the changes to your Flux repository, the changes should be applied to your cluster.

If you don’t want to wait for the automatic reconcile, you can run which will trigger a reconcile of the git repo.

flux reconcile source git flux-system -n flux-system

You should now be able to go to the Grafana URL that you defined, in my case http://grafana.127.0.0.1.nip.io

And you should see the simple dashboard among your dashboards.

Conclusion

We have used very basic setup of Flux to install grafana-operator to our cluster. When that was done, we installed a basic Grafana instance and a dashboard giving us dashboards as code.

The dashboard wasn’t the prettiest, but it’s an easy example. Hopefully someone will make a more in depth blog on how to use grafana-operator to bootstrap entire clusters monitoring in the future.