Blog - Rico Berger

Kubernetes Logging with ClickHouse and OpenTelemetry

Rico Berger — Mon, 03 Nov 2025 17:00:00 +0000

In today's blog post, we will deploy ClickHouse and the OpenTelemetry Collector in a Kubernetes cluster to collect the cluster's logs. Finally, we will deploy Grafana to explore the logs stored in ClickHouse.

ClickHouse

In the first step, we need to create a new ClickHouse cluster. In this blog post, we will use the ClickHouse Operator for this purpose. The operator can be deployed using the following commands:

kubectl create namespace clickhouse-operator
kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/clickhouse/clickhouse-operator.yaml

NAME                                   READY   STATUS    RESTARTS   AGE
clickhouse-operator-6bcc776b68-77xf6   2/2     Running   0          14s

Once the ClickHouse operator is running, we can create a ClickHouse cluster. Since we want a cluster with three shards, we also need to deploy ClickHouse Keeper. To deploy ClickHouse Keeper and create the ClickHouse cluster afterward, we can use the following commands:

kubectl create namespace otel
kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/clickhouse/clickhouse-keeper.yaml
kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/clickhouse/clickhouse.yaml

NAME                               READY   STATUS    RESTARTS   AGE
chi-clickhouse-otel-0-0-0          1/1     Running   0          18s
chi-clickhouse-otel-1-0-0          1/1     Running   0          19s
chi-clickhouse-otel-2-0-0          1/1     Running   0          19s
chk-clickhouse-keeper-otel-0-0-0   1/1     Running   0          70s
chk-clickhouse-keeper-otel-0-1-0   1/1     Running   0          63s
chk-clickhouse-keeper-otel-0-2-0   1/1     Running   0          56s

OpenTelemetry Collector

In the next step, we need to deploy the OpenTelemetry Collector. The OpenTelemetry project provides pre-built distributions of the collector. However, since it is recommended to build a custom Docker image, we will choose this option instead of using one of the pre-built distributions.

We will use the following Dockerfile and builder manifest to build our custom Docker image registry.homelab.ricoberger.dev/otel-collector:v0.138.0 using the following command:

docker buildx build --platform=linux/arm64,linux/amd64 -f ./Dockerfile -t registry.homelab.ricoberger.dev/otel-collector:v0.138.0 --push .

receivers:
  # Configure the OTLP receiver, so that the OpenTelementry Collector can
  # receive logs via gRPC or HTTP using OTLP format.
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
      http:
        endpoint: ${env:MY_POD_IP}:4318
  # Configure the Filelog receiver, which is responsible for collecting the logs
  # from all Pods. Because we are using the Filelog receiver we also have to
  # deploy the OpenTelemetry Collector as DaemonSet in our Kubernetes cluster.
  filelog:
    include: [/var/log/pods/*/*/*.log]
    start_at: beginning
    storage: file_storage
    include_file_path: true
    include_file_name: false
    # The following operator will perform some simple tasks on the collected log
    # lines:
    # - The container operator parses logs in docker, cri-o and containerd
    #   formats and is responsible for parsing the timestamp and attributes.
    # - The json_parser operator parses the string-type field selected by
    #   parse_from as JSON. This means it will parse the body of all log lines
    #   as attributes and sets the severity for each log line from the parsed
    #   result.
    # - The trace_parser operator sets the trace on an entry by parsing a value
    #   from the body.
    operators:
      - id: container_parser
        type: container
      - id: json_parser
        type: json_parser
        on_error: send_quiet
        severity:
          parse_from: attributes.level
      - id: trace_parser
        type: trace_parser
        trace_id:
          parse_from: attributes.trace_id
        span_id:
          parse_from: attributes.span_id
        trace_flags:
          parse_from: attributes.trace_flags
        on_error: send_quiet

exporters:
  # Configure the ClickHouse exporter. We are mostly using the default
  # configureation, the only exception is that we disable the creation of the
  # schema. This is required because we want to use multiple shards, which is
  # is not supported in the default schema.
  clickhouse:
    endpoint: clickhouse://clickhouse-clickhouse.otel.svc.cluster.local:9000?dial_timeout=10s&compress=lz4
    database: otel
    username: "${env:CLICKHOUSE_USERNAME}"
    password: "${env:CLICKHOUSE_PASSWORD}"
    create_schema: false
    async_insert: true
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

processors:
  # To improve the ingestion performance of our logs into ClickHouse we batch
  # the log lines, before sending them over to ClickHouse.
  batch:
    send_batch_size: 10000
    timeout: 10s
  memory_limiter:
    check_interval: 5s
    limit_mib: 400
    spike_limit_mib: 100
  # The Kubernetes attributes processor will automatically expand the resource
  # attributes of our logs with Kubernetes metadata, such as the Pod name,
  # container name, etc.
  k8sattributes:
    passthrough: false
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: connection
    extract:
      metadata:
        - "k8s.namespace.name"
        - "k8s.deployment.name"
        - "k8s.statefulset.name"
        - "k8s.daemonset.name"
        - "k8s.cronjob.name"
        - "k8s.job.name"
        - "k8s.node.name"
        - "k8s.pod.name"
        - "k8s.pod.uid"
        - "k8s.pod.start_time"
      labels:
        - tag_name: $$1
          key_regex: (.*)
          from: pod
      annotations:
        - tag_name: $$1
          key_regex: (.*)
          from: pod
  # The last processor we are using is the transform processor, to add the
  # service name for each log line, based on the container name added by the
  # Kubernetes attributes processor.
  transform/logs:
    error_mode: silent
    log_statements:
      - context: log
        conditions:
          - IsMap(resource.attributes) and
            resource.attributes["k8s.container.name"] != nil
        statements:
          - set(resource.attributes["service.name"],
            resource.attributes["k8s.container.name"])

# Add extensions, for health checks, zPages and file storage. The file storage
# extension is used within the Filelog receiver, to store the offset for all log
# files, so the receiver can pick up where it left off in the case of a
# collector restart
extensions:
  file_storage:
    directory: /var/lib/otel-collector
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
  zpages:
    endpoint: ${env:MY_POD_IP}:55679

service:
  extensions:
    - file_storage
    - health_check
    - zpages
  pipelines:
    logs:
      receivers:
        - otlp
        - filelog
      processors:
        - batch
        - memory_limiter
        - k8sattributes
        - transform/logs
      exporters:
        - clickhouse

Now that we have defined our OpenTelemetry Collector configuration, we can deploy the OpenTelemetry Collector as a DaemonSet. Since we have disabled schema creation in the ClickHouse exporter, we will also create and run a CronJob to establish the schema in ClickHouse. The following commands can be used to deploy the CronJob and DaemonSet:

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/otel-collector/otel-collector-create-schema.yaml
kubectl create job --from=cronjob/otel-collector-create-schema otel-collector-create-schema-manual

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/otel-collector/otel-collector.yaml

chi-clickhouse-otel-0-0-0                   1/1     Running     0          3h16m
chi-clickhouse-otel-1-0-0                   1/1     Running     0          3h16m
chi-clickhouse-otel-2-0-0                   1/1     Running     0          3h16m
chk-clickhouse-keeper-otel-0-0-0            1/1     Running     0          3h31m
chk-clickhouse-keeper-otel-0-1-0            1/1     Running     0          3h31m
chk-clickhouse-keeper-otel-0-2-0            1/1     Running     0          3h31m
otel-collector-create-schema-manual-twxtq   0/1     Completed   0          3h15m
otel-collector-ggt92                        1/1     Running     0          3h14m
otel-collector-wsj6n                        1/1     Running     0          3h14m

Demo Application

We now have a functioning OpenTelemetry Collector that gathers all the logs from our Kubernetes cluster using the Filelog receiver and stores them in our ClickHouse cluster via the ClickHouse exporter.

In the next step, we will deploy the echoserver, which is configured to send logs to the OpenTelemetry Collector using the OTLP format. You can use the following commands to install the echoserver.

kubectl create namespace echoserver
helm upgrade --install echoserver oci://ghcr.io/ricoberger/charts/echoserver --version 1.1.0 -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/echoserver/values-otlp.yaml --namespace echoserver

NAME                          READY   STATUS    RESTARTS   AGE
echoserver-57d88d558f-hzmvw   1/1     Running   0          14m

We can now use the echoserver to generate logs for testing the filtering of logs later.

kubectl port-forward -n echoserver svc/echoserver 8080

curl -vvv "http://localhost:8080/"
curl -vvv "http://localhost:8080/panic"
curl -vvv "http://localhost:8080/status"
curl -vvv "http://localhost:8080/status?status=400"
curl -vvv "http://localhost:8080/timeout?timeout=10s"
curl -vvv "http://localhost:8080/headersize?size=100"
curl -vvv -X POST -d '{"method": "POST", "url": "http://localhost:8080/", "body": "test", "headers": {"x-test": "test"}}' http://localhost:8080/request
curl -vvv "http://localhost:8080/fibonacci?n=100"

Grafana

In the final section of this blog post, we will deploy Grafana with a pre-configured ClickHouse datasource to view our logs.

kubectl create namespace grafana
helm upgrade --install grafana oci://ghcr.io/grafana/helm-charts/grafana --version 10.1.4 -f https://raw.githubusercontent.com/ricoberger/playground/6b942ebe1df3b121b09042274f6598485d159826/kubernetes/kubernetes-logging-with-clickhouse-and-opentelemetry/grafana/values.yaml --namespace grafana

NAME                       READY   STATUS    RESTARTS   AGE
grafana-55b587fdc8-4cg6t   1/1     Running   0          13m

Afterward, we can access our Grafana instance using the following port-forward command with the username admin and the password admin.

kubectl port-forward -n grafana svc/grafana 3000:80

When we navigate to the Explore section and select the ClickHouse datasource, we can view the logs. For instance, we can filter the logs of our ClickHouse cluster by using ServiceName with the value clickhouse.

We can also filter the logs using the LogAttributes. In the following example, we will view all GET requests for the echoserver.

Wrapping Up

That's it for today's blog post. We have successfully deployed the OpenTelemetry Collector to gather logs in our Kubernetes cluster and store them in ClickHouse. Finally, we have deployed and configured Grafana to explore our stored logs. If you enjoyed the blog post, feel free to follow me on my social media channels.

Fixing the YAML Language Server

Rico Berger — Sun, 14 Sep 2025 12:00:00 +0000

First, I want to apologize for the clickbait title. The YAML Language Server is a great project, and I don't want to disrespect the maintainers. However, while reworking my Neovim configuration, I encountered some issues with the YAML Language Server that I would like to share with you, along with how I resolved them.

Motivation

While reworking my Neovim configuration, I also decided to adjust the YAML Language Server settings to support completion and validation for Kubernetes manifests. I began with the following configuration:

yaml = {
  format = {
    enable = false,
  },
  completion = true,
  hover = true,
  validate = true,
  schemas = {
    kubernetes = {
      "/kubernetes/**/*.yml",
      "/kubernetes/**/*.yaml",
    },
  },
  schemaStore = {
    enable = false,
    url = "",
  },
},

This worked well for all standard Kubernetes manifests. The problems began when I started working with CustomResourceDefinitions (CRDs), as I consistently encountered the following error:

kubernetes/namespaces/default/example-vs.yaml|2 col 7-21 error   1| Value is not accepted. Valid values: "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", "MutatingAdmissionPolicy", "MutatingAdmissionPolicyBinding", "StorageVersion", "DaemonSet", "Deployment", "ReplicaSet", "StatefulSet", "TokenRequest", "TokenReview", "LocalSubjectAccessReview", "SelfSubjectAccessReview", "SelfSubjectRulesReview", "SubjectAccessReview", "HorizontalPodAutoscaler", "Scale", "CronJob", "Job", "CertificateSigningRequest", "ClusterTrustBundle", "Lease", "LeaseCandidate", "LimitRange", "Namespace", "Node", "PersistentVolume", "PersistentVolumeClaim", "Pod", "ReplicationController", "ResourceQuota", "Service", "FlowSchema", "PriorityLevelConfiguration", "Ingress", "IngressClass", "NetworkPolicy", "IPAddress", "ServiceCIDR", "PodDisruptionBudget", "DeviceClass", "ResourceClaim", "ResourceClaimTemplate", "ResourceSlice", "CSIDriver", "CSINode", "VolumeAttachment", "StorageVersionMigration", "CustomResourceDefinition", "APIService".

This was somewhat expected, as the YAML Language Server is unaware of the CustomResourceDefinitions I am using. To resolve this, there are two options:

Add the schema for each CustomResourceDefinition to the YAML Language Server configuration. The issue with this approach is that I would need to define a unique pattern for each file, which is not feasible.

schemas = {
  kubernetes = {
    "*-deploy.yml",
    "*-deploy.yaml",
    ...
  },
  ["https://raw.githubusercontent.com/datreeio/CRDs-catalog/refs/heads/main/networking.istio.io/virtualservice_v1.json"] = {
    "*-vs.yml",
    "*-vs.yaml",
  }
}

Add an annotation to all CustomResourceDefinitions, which was also not a feasible solution for me.

# yaml-language-server: $schema=https://raw.githubusercontent.com/datreeio/CRDs-catalog/refs/heads/main/networking.istio.io/virtualservice_v1.json
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: example
  namespace: default
spec: ...

After conducting some research, I decided to create my own JSON schema for Kubernetes, which includes the Kubernetes JSON schema and the schemas for all CustomResourceDefinitions I am using. The configuration for the YAML Language Server to utilize the new schema is as follows:

schemas = {
  ["https://raw.githubusercontent.com/ricoberger/kubernetes-json-schema/refs/heads/main/schemas/all.json"] = {
    "/kubernetes/**/*.yml",
    "/kubernetes/**/*.yaml",
  }
}

The problem with this approach is that the YAML Language Server has extensive logic related to the default Kubernetes schema, which prevents it from functioning as expected. With the configuration mentioned above, I consistently received the following error:

kubernetes/namespaces/default/example-deploy.yaml|2 col 1-2 error| Matches multiple schemas when only one must validate.

This is a known issue with the YAML Language Server. At that point, I was so frustrated that I decided to fork the YAML Language Server to add an option to overwrite the default Kubernetes schema.

With this small change, everything is now functioning as expected. I receive completion and validation for all Kubernetes manifests, including the CustomResourceDefinitions I am using.

Usage

In the following section, we will explore how to use the forked version of the YAML Language Server and the custom Kubernetes JSON schema with Neovim.

First, we need to clone the repository, check out the branch with the modified version, and build the YAML Language Server.

git clone git@github.com:ricoberger/yaml-language-server.git
cd yaml-language-server
git checkout add-option-to-overwrite-kubernetes-schema

npm install
npm run build

In the next step, we will create a bash script yamlls in our PATH to set the YAMLLS_KUBERNETES_SCHEMA_URL environment variable and to start our custom YAML Language Server.

#!/usr/bin/env bash

export YAMLLS_KUBERNETES_SCHEMA_URL="https://raw.githubusercontent.com/ricoberger/kubernetes-json-schema/refs/heads/main/schemas/all.json"

node /Users/ricoberger/Documents/GitHub/ricoberger/yaml-language-server/out/server/src/server.js $@

Lastly, we configure the YAML Language Server in Neovim (yamlls.lua) to use our yamlls script for starting the server.

return {
  cmd = { "yamlls", "--stdio" },
  filetypes = {
    "yaml",
    "yaml.docker-compose",
    "yaml.gitlab",
    "yaml.helm-values",
  },
  single_file_support = true,
  settings = {
    redhat = {
      telemetry = {
        enabled = false,
      },
    },
    yaml = {
      format = {
        enable = false,
      },
      completion = true,
      hover = true,
      validate = true,
      schemas = {
        kubernetes = {
          "/kubernetes/**/*.yml",
          "/kubernetes/**/*.yaml",
        },
      },
      schemaStore = {
        enable = false,
        url = "",
      },
    },
  },
  on_init = function(client)
    client.server_capabilities.documentFormattingProvider = nil
    client.server_capabilities.documentRangeFormattingProvider = nil
  end,
}

If you are using the Helm Language Server, you can adjust the configuration in helm_ls.lua to also utilize the yamlls script for starting the YAML Language Server.

Schema Generation

In the last section of this blog post, we will examine the ricoberger/kubernetes-json-schema repository and explain how the generation of the JSON schema works.

The JSON schema is generated by the generate.sh script. First, we create a new kind cluster and apply all CustomResourceDefinitions (CRDs) from the crds directory. Next, we start a kubectl proxy to access the Kubernetes API of the kind cluster.

kind create cluster --image=kindest/node:v1.34.0
kubectl apply --server-side -f crds/
kubectl proxy --port=5555 --accept-hosts='^.*'

The openapi2jsonschema.py script is used to fetch the OpenAPI v2 definition from the kind cluster and convert it to JSON schema. The generated JSON schema files are stored in the schemas directory.

openapi2jsonschema.py "schemas" "http://127.0.0.1:5555/openapi/v2"

Wrapping Up

I'm still unsure whether I'm missing something obvious or if the YAML Language Server is simply not designed to handle this use case. The latter suggests that there are several issues related to the handling of CustomResourceDefinitions (824, #841, #962, and #1050). If you have any suggestions or improvements, please let me know. I would be happy to hear your thoughts.

Reworking My Neovim Configuration

Rico Berger — Thu, 11 Sep 2025 17:00:00 +0000

It has been a while since I last shared my Neovim configuration. Over time, I've made several adjustments to enhance my workflow. Recently, I decided to streamline my setup by removing some plugins and replacing them with built-in alternatives. In this post, I'll walk you through the changes I've made and my current Neovim configuration.

Managing Plugins

Until now, I have used lazy.nvim to manage my plugins. While lazy.nvim is an excellent plugin manager, I wanted to simplify my setup by using the built-in plugin manager.

Adding plugins with the built-in plugin manager is straightforward. We can do this by calling vim.pack.add(). For example, to add my favorite color scheme, catppuccin, we can simply include the following lines in our init.lua file:

vim.pack.add({
  { src = "https://github.com/catppuccin/nvim", name = "catppuccin" },
}, { load = true })

require("catppuccin").setup({...})
vim.cmd.colorscheme("catppuccin")

While exploring the built-in plugin manager, I found this Reddit post about implementing lazy loading. For instance, I'm using CopilotChat.nvim for AI assistance. To load this plugin only when needed, we can use the following configuration:

vim.keymap.set("n", "co", "CopilotChatOpen")

vim.api.nvim_create_autocmd("CmdUndefined", {
  group = vim.api.nvim_create_augroup( "lazy-load-copilotchat", { clear = true }),
  pattern = { "CopilotChat*" },
  callback = function()
    vim.pack.add({
      { src = "https://github.com/nvim-lua/plenary.nvim", },
      { src = "https://github.com/CopilotC-Nvim/CopilotChat.nvim", name = "CopilotChat", },
    }, { load = true })

    require("CopilotChat").setup({...})
  end,
  once = true,
})

This will load the plugin only when we use the :CopilotChatOpen command or any other command provided by the plugin.

File Explorer

Since its release, I've been using the snacks.nvim explorer. I primarily use it to switch between files in the same directory and to copy, move, rename, and delete files. I accomplish these tasks with the following keymaps:

vim.keymap.set("n", "ee", function() local dir = vim.fn.expand("%:.:h") if dir == "." or dir == "" then return ":edit " end return ":edit " .. vim.fn.expand("%:.:h") .. "/" end, { expr = true })
vim.keymap.set("n", "ec", function() return ":!cp " .. vim.fn.expand("%:.") .. " " .. vim.fn.expand("%:.") end, { expr = true })
vim.keymap.set("n", "em", function() return ":!mv " .. vim.fn.expand("%:.") .. " " .. vim.fn.expand("%:.") end, { expr = true })
vim.keymap.set("n", "er", function() return ":!rm " .. vim.fn.expand("%:.") end, { expr = true })
vim.keymap.set("n", "ey", function() vim.fn.setreg("+", vim.fn.expand("%:.")) end)

Finding Files

My go-to plugin for opening files has been the snacks.nvim picker. However, after reading this Reddit post, I wanted to explore the built-in capabilities of Neovim for this purpose.

With the Neovim nightly version, we can use the findfunc option to define a custom function for finding files, the wildtrigger() function to initiate wildcard expansion in the command line, and the matchfuzzy() function to enable fuzzy matching.

To use fd for finding files with the :find command and to fuzzy match these files with the provided argument, we can add the following code to our init.lua file:

if vim.fn.executable("fd") == 1 then
  function _G.fd_find_files(cmdarg, _)
    local fnames = vim.fn.systemlist("fd --full-path --hidden --color never --type f --exclude .git")

    if #cmdarg == 0 then
      return fnames
    else
      return vim.fn.matchfuzzy(fnames, cmdarg)
    end
  end

  vim.opt.findfunc = "v:lua.fd_find_files"
end

To automatically enable autocompletion when typing the :find command, we can add the following autocmd:

vim.api.nvim_create_autocmd({ "CmdlineChanged", "CmdlineLeave" }, {
  pattern = { "*" },
  group = vim.api.nvim_create_augroup( "cmdline-autocompletion", { clear = true }),
  callback = function(ev)
    local function should_enable_autocomplete()
      local cmdline_cmd = vim.fn.split(vim.fn.getcmdline(), " ")[1]
      return cmdline_cmd == "find"
    end

    if ev.event == "CmdlineChanged" and should_enable_autocomplete() then
      vim.opt.wildmode = "noselect:lastused,full"
      vim.fn.wildtrigger()
    end

    if ev.event == "CmdlineLeave" then
      vim.opt.wildmode = "full"
    end
  end,
})

Other features I utilized from the snacks.nvim picker include finding recently used files, sending found files to the quickfix list, locating buffers, and identifying marks. I achieved these functionalities with custom functions and keymaps, which can be found in my dotfiles repository.

Searching Through Files

The last feature I used from snacks.nvim was searching through files. To replace this functionality, I decided to use the built-in :grep command along with rg. To set this up, we can add the following configuration to our init.lua file:

if vim.fn.executable("rg") == 1 then
  vim.opt.grepprg = "rg --vimgrep --smart-case --hidden --color=never --glob='!.git'"
  vim.opt.grepformat = "%f:%l:%c:%m"
end

Together with the following keymaps, I replicated the search functionality I had with snacks.nvim and added some useful search commands. These include searching in the directory of the current file, searching in the current file, searching for the word under the cursor, searching for visually selected text, and searching for todo comments like TODO, FIXME, and BUG. The last keymap also replaces the todo-comments.nvim plugin for me.

vim.keymap.set("n", "ss", ":silent grep!")
vim.keymap.set("n", "sc", function() return ":silent grep! --glob='" .. vim.fn.expand("%:.:h") .. "/**' " end, { expr = true })
vim.keymap.set("n", "s/", function() return ":silent grep! --glob='" .. vim.fn.expand("%:.") .. "' " end, { expr = true })
vim.keymap.set("n", "sw", ":silent grep!")
vim.keymap.set("v", "sv", 'y:silent grep!"')
vim.keymap.set("n", "st", ":silent grep! -e='todo:' -e='warn:' -e='info:' -e='xxx:' -e='bug:' -e='fixme:' -e='fixit:' -e='bug:' -e='issue:'")

LSP, Linting and Formatting

With the recently added support for inline completion in Neovim's built-in LSP client, I decided to remove the copilot.lua plugin and use the @github/copilot-language-server instead. Along with the following autocmd, I can now utilize the inline completion feature provided by GitHub Copilot. I also set up a keymap to accept suggestions with .

vim.api.nvim_create_autocmd("LspAttach", {
  callback = function(event)
    local client = vim.lsp.get_client_by_id(event.data.client_id)
    local buffer = event.buf

    if client then
      if client:supports_method( vim.lsp.protocol.Methods.textDocument_inlineCompletion) then
        vim.lsp.inline_completion.enable(true)
        vim.keymap.set("i", "", function()
          if not vim.lsp.inline_completion.get() then
            return ""
          end
        end, {
          expr = true,
          replace_keycodes = true,
        })
      end
    end
  end,
})

I also added the efm-langserver to replace nvim-lint and conform.nvim for linting and formatting. You can find the configuration for efm-langserver in my dotfiles.

Formatting Using Automatic Commands

Continuous Profiling Using Parca

Rico Berger — Sat, 28 Jun 2025 12:00:00 +0000

In today's blog post, we will examine continuous profiling using Parca. We will set up Parca in a Kubernetes cluster, explore its architecture, and collect profiles from an example application. Finally, we will review the Parca UI to analyze the collected profiles. If you are not familiar with continuous profiling, I recommend reading the "What is profiling?" section in the Parca documentation.

Parca has two main components: the Parca Server and the Parca Agent. The Parca Server stores profiling data and enables querying and analysis over time. The Parca Agent is an eBPF-based whole-system profiler.

The diagram below illustrates the architecture of Parca and the Parca Agent.

Parca can source profiles by retrieving them from targets via HTTP or by using the Parca Agent, which pushes them to the Parca Server. The Parca Server then stores these profiles. Different series of profiles are identified by their unique label combinations and can be visualized through the Parca UI, which is provided by the Parca Server, using icicle-graphs.

Installation

In the following we will deploy an example application to a Kubernetes cluster, which exposes HTTP endpoints serving pprof, the Parca Server and the Parca Agent. Let's start by creating a new namespace called parca and deploying the echoserver as our example application.

kubectl create namespace parca
helm upgrade --install echoserver --namespace parca oci://ghcr.io/ricoberger/charts/echoserver --version 1.0.3 --set-json='podLabels={"app": "echoserver"}'

In the next step we deploy the Parca Server, by applying the 1-1-parca-server.yaml file from the ricoberger/playground repository:

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/d278b2c9dd149d2aea7bfda036a764ae51e0e6a1/kubernetes/parca/1-1-parca-server.yaml

This will create a Service, StatefulSet, and ConfigMap for the Parca Server. The ConfigMap contains the configuration for the Parca Server. We define the storage location (the local filesystem¹) that Parca will use to store the profiles and add a scrape configuration for our example application.

object_storage:
  bucket:
    config:
      directory: /data
    type: FILESYSTEM
scrape_configs:
  - job_name: echoserver
    scrape_interval: 45s
    scrape_timeout: 60s
    static_configs:
      - targets:
          - echoserver.parca.svc.cluster.local:8080
    profiling_config:
      pprof_config:
        fgprof:
          enabled: true
          path: /debug/pprof/fgprof

Last but not least, we deploy the Parca Agent using the 1-2-parca-agent.yaml file:

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/d278b2c9dd149d2aea7bfda036a764ae51e0e6a1/kubernetes/parca/1-2-parca-agent.yaml

The Parca Agent is deployed as a DaemonSet and requires a ClusterRole, ClusterRoleBinding, and ServiceAccount to watch all Nodes and Pods within the cluster. The deployed ConfigMap contains the configuration for the Parca Agent, including the relabel configuration, ensuring that the pushed profiles have all the necessary labels for later differentiation.

At this point, our example application, the Parca Server and the Parca Agent, should be up and running:

NAME                              READY   STATUS    RESTARTS   AGE
echoserver-664ccd6944-hjxpb       1/1     Running   0          4m15s
parca-agent-lq8h5                 1/1     Running   0          3m7s
parca-server-0                    1/1     Running   0          3m37s

We can also access the Parca UI via kubectl port-forward -n parca svc/parca-server 7070, where we should see our example application in the targets list: http://localhost:7070/targets.

Parca Operator

Before we continue exploring the Parca UI, I want to share a small side project of mine. The Parca Operator allows us to dynamically populate the scrape_configs of the Parca Server using a ParcaScrapeConfig Custom Resource.

The operator uses the Kubernetes Service Discovery feature of Parca to dynamically add, update, and remove scrape configurations, similar to the Prometheus Operator. This allows us to replace our static configuration for the echoserver, which would not function correctly if we run more than one replica of the service.

Let's first update our configuration for the Parca Server by removing the scrape_configs:

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/d278b2c9dd149d2aea7bfda036a764ae51e0e6a1/kubernetes/parca/2-1-parca-server.yaml

Now we can install the Parca Operator using the corresponding Helm chart and the 3-1-parca-operator.yaml values file. The operator will mount the Parca Server's configuration file and create a new configuration file as a Secret in the parca namespace, named parca-server-generated.

helm upgrade --install parca-operator --namespace parca oci://ghcr.io/ricoberger/charts/parca-operator --version 1.2.0 -f https://raw.githubusercontent.com/ricoberger/playground/d278b2c9dd149d2aea7bfda036a764ae51e0e6a1/kubernetes/parca/3-1-parca-operator.yaml

In the next step, we will update our Parca Server setup to use the Secret instead of the ConfigMap for its configuration. We will also add a ClusterRole, ClusterRoleBinding, and ServiceAccount for the Parca Server, enabling it to list and watch all Pods in the cluster.

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/d278b2c9dd149d2aea7bfda036a764ae51e0e6a1/kubernetes/parca/3-2-parca-server.yaml

NAME                              READY   STATUS    RESTARTS   AGE
echoserver-664ccd6944-hjxpb       1/1     Running   0          8m2s
parca-agent-lq8h5                 1/1     Running   0          6m54s
parca-operator-6789868cdd-7txdf   1/1     Running   0          103s
parca-server-0                    1/1     Running   0          49s

Last but not least, we will create a ParcaScrapeConfig for our example application:

kubectl apply -f https://raw.githubusercontent.com/ricoberger/playground/d278b2c9dd149d2aea7bfda036a764ae51e0e6a1/kubernetes/parca/3-3-echoserver-parcascrapeconfig.yaml

In the ParcaScrapeConfig we select all Pods, which are having the app label set to echoserver. We also set the port which is used to serve the pprof endpoints.

In the ParcaScrapeConfig, we select all Pods with the app label set to echoserver. We also specify the port (http) used to serve the pprof endpoints. The remaining configuration is similar to the static configuration we used previously.

apiVersion: parca.ricoberger.de/v1alpha1
kind: ParcaScrapeConfig
metadata:
  name: echoserver
  namespace: parca
spec:
  selector:
    matchLabels:
      app: echoserver
  scrapeConfig:
    port: http
    interval: 45s
    timeout: 60s
    profilingConfig:
      pprofConfig:
        fgprof:
          enabled: true
          path: /debug/pprof/fgprof

If we access the targets list in the Parca UI now, we should see a new job for the echoserver with a different set of labels than before.

Explore Profiles

In the last section of this blog post, we will explore the profiles ingested into Parca. Let's open the Parca UI by creating a port-forward to the Parca Server using the command kubectl port-forward -n parca svc/parca-server 7070 and then opening http://localhost:7070/ in our browser.

In the first step we explore the profiles captured by the Parca Agent. To do this, we select On-CPU from the Profile Type dropdown menu. For now, we are only interested in the profiles of the echoserver. To filter the profiles, we select the container label and the echoserver value. Lastly, we select the pod label from the Sum By dropdown menu. After that, we should see the profiles captured by the Parca Agent for the echoserver.

To simulate a CPU intensive task, we create a port-forward to the echoserver using the command kubectl port-forward -n parca svc/echoserver 8080. Afterwards we run the following cURL command:

curl -vvv "http://localhost:8080/fibonacci?n=100000000"

Once the request is complete, we can refresh the profiles displayed in the Parca UI by clicking the Search button. We should now observe a spike in the CPU usage of the echoserver. By hovering over one of the data points, we can see how many cores the echoserver used per second and the total CPU usage time. Clicking on the data point will display the captured sample below the graph, allowing us to check where the most time was spent. As expected, most of the time should be attributed to the main.fibonacciHandler function.

When using the Parca Agent, only the On-CPU profiles are available. However, since we added a scrape configuration for the echoserver, the other profiles should also be available. These include:

Fgprof Samples Total: CPU profile samples observed regardless of their current On/Off CPU scheduling status
Fgprof Samples Time Total: CPU profile measured regardless of their current On/Off CPU scheduling status in nanoseconds
Goroutine Created Total: Stack traces that created all current goroutines.
Memory Allocated Objects Total: A sampling of all past memory allocations by objects.
Memory Allocated Bytes Total: A sampling of all past memory allocations in bytes.
Memory In-Use Objects: A sampling of memory allocations of live objects by objects.
Memory In-Use Bytes: A sampling of memory allocations of live objects by bytes.
Process CPU Nanoseconds: CPU profile measured by the process itself in nanoseconds.
Process CPU Samples: CPU profile samples observed by the process itself.

If we are selecting the Process CPU Samples from the Profile Types dropdown, we should see a similar pattern as for the On-CPU profiles.

Lastly, we will select the Memory In-Use Bytes item from the Profile Types dropdown. When we hover over a data point in the graph, we can see the number of bytes used by the echoserver during this time. If we click on the data point, we can view the sample below the graph to analyze where most of our memory was spent.

Parca supports the same storage configuration options as Thanos. This means it can also be used together with an S3 compatible object storage or an Azure Storage Account. This would also be the recommended storage option for a production deployment, because Parca doesn't handle the retention for stored data and it is recommended to configure this via a lifecycle rule in the object storage. ↩︎

Deploy YugabyteDB on a Multi-Zone AKS Cluster

Rico Berger — Sun, 13 Apr 2025 19:00:00 +0000

After getting started with YugabyteDB in the last blog post, I wanted to explore how to set up a zone-aware YugabyteDB cluster. To do this, we will create a multi-zone AKS cluster and use the standard single-zone YugabyteDB Helm Chart to deploy one-third of the nodes in the database cluster across each of the three zones.

With the commerce schema from the last post, the final architecture will appear as shown in the following graphic. We will have three StatefulSets for the YB-Master and three StatefulSets for the YB-TServer. Each StatefulSet will contain one Pod running in the specified AKS zone. The tablets for each table in the commerce schema will be distributed across all Pods in the cluster. All the configuration files we are using can be found in the ricoberger/playground GitHub repository.

Create a AKS Cluster

Create a AKS cluster, if you have not already done so, by running the following command. Note that if you do not specify 3 zones in the zones parameter explicitly then AKS may place the 3 nodes in only 2 zones.

az group create --name yugabyte --location germanywestcentral
az aks create --resource-group yugabyte --name yugabyte --node-count 3 --zones 1 2 3

We create a StorageClass. We need to specify WaitForFirstConsumer mode for the volumeBindingMode so that volumes will be provisioned according to pods zone affinities.

kubectl apply --server-side -f storageclass.yaml

Create a YugabyteDB Cluster

Add the Helm chart repository and make sure that we have the latest updates to the repository by running the following commands.

helm repo add yugabytedb https://charts.yugabyte.com
helm repo update

Before we can install the Helm charts, we have to create the 3 namespaces first.

kubectl create namespace yb-germanywestcentral-1
kubectl create namespace yb-germanywestcentral-2
kubectl create namespace yb-germanywestcentral-3

Now we create the overall YugabyteDB cluster in such a way that one third of the nodes are hosted in each zone.

helm upgrade --install yb-germanywestcentral-1 yugabytedb/yugabyte --version 2.25.1 --namespace yb-germanywestcentral-1 --wait -f values-yb-germanywestcentral-1.yaml
helm upgrade --install yb-germanywestcentral-2 yugabytedb/yugabyte --version 2.25.1 --namespace yb-germanywestcentral-2 --wait -f values-yb-germanywestcentral-2.yaml
helm upgrade --install yb-germanywestcentral-3 yugabytedb/yugabyte --version 2.25.1 --namespace yb-germanywestcentral-3 --wait -f values-yb-germanywestcentral-3.yaml

Check the Cluster Status

We can check the status of the cluster using various commands noted below.

Check the pods

kubectl get pods -A | grep yb-germanywestcentral

yb-germanywestcentral-1                     yb-master-0                                                      3/3     Running            0                7m18s
yb-germanywestcentral-1                     yb-tserver-0                                                     3/3     Running            0                7m18s
yb-germanywestcentral-2                     yb-master-0                                                      3/3     Running            0                5m55s
yb-germanywestcentral-2                     yb-tserver-0                                                     3/3     Running            0                5m55s
yb-germanywestcentral-3                     yb-master-0                                                      3/3     Running            0                4m53s
yb-germanywestcentral-3                     yb-tserver-0                                                     3/3     Running            0                4m53s

Check the services.

kubectl get service -A | grep yb-germanywestcentral

yb-germanywestcentral-1                     yb-masters                                      ClusterIP      None                    7000/TCP,7100/TCP,15433/TCP                                                            8m6s
yb-germanywestcentral-1                     yb-tservers                                     ClusterIP      None                    9000/TCP,12000/TCP,11000/TCP,13000/TCP,9100/TCP,6379/TCP,9042/TCP,5433/TCP,15433/TCP   8m6s
yb-germanywestcentral-2                     yb-masters                                      ClusterIP      None                    7000/TCP,7100/TCP,15433/TCP                                                            6m42s
yb-germanywestcentral-2                     yb-tservers                                     ClusterIP      None                    9000/TCP,12000/TCP,11000/TCP,13000/TCP,9100/TCP,6379/TCP,9042/TCP,5433/TCP,15433/TCP   6m42s
yb-germanywestcentral-3                     yb-masters                                      ClusterIP      None                    7000/TCP,7100/TCP,15433/TCP                                                            5m40s
yb-germanywestcentral-3                     yb-tservers                                     ClusterIP      None                    9000/TCP,12000/TCP,11000/TCP,13000/TCP,9100/TCP,6379/TCP,9042/TCP,5433/TCP,15433/TCP   5m40s

We can also access the YB-Master Admin UI for the cluster at http://localhost:7000. Note that we can use any of the above three services for this purpose as all of them will show the same cluster metadata.

kubectl port-forward --namespace yb-germanywestcentral-1 svc/yb-masters 7000

Configure Zone-Aware Replica Placement

The default replica placement policy treats every yb-tserver as equal irrespective of its placement_* setting. We go to http://localhost:7000/cluster-config to confirm that the default configuration is still in effect.

To make the replica placement zone-aware, so that one replica is placed in each zone, we run the following command:

kubectl exec -it --namespace yb-germanywestcentral-1 yb-master-0 -- bash -c "/home/yugabyte/master/bin/yb-admin --master_addresses yb-master-0.yb-masters.yb-germanywestcentral-1.svc.cluster.local:7100,yb-master-0.yb-masters.yb-germanywestcentral-2.svc.cluster.local:7100,yb-master-0.yb-masters.yb-germanywestcentral-3.svc.cluster.local:7100 modify_placement_info azure.germanywestcentral.germanywestcentral-1,azure.germanywestcentral.germanywestcentral-2,azure.germanywestcentral.germanywestcentral-3 3"

To see the new configuration, we go to http://localhost:7000/cluster-config.

Single Namespace

The following section was added after the blog post was first published on April 14, 2025.

After publishing the first version of the blog post, I wondered if it was possible to create a zone-aware YugabyteDB cluster within a single namespace, allowing PodDisruptionBudgets to apply to the entire cluster. It turns out this is achievable by setting the oldNamingStyle value to false in the Helm chart. You can find the updated values files in the ricoberger/playground repository.

kubectl create namespace yugabytedb

helm upgrade --install yb-germanywestcentral-1 yugabytedb/yugabyte --version 2.25.1 --namespace yugabytedb --wait -f values-single-namespace-yb-germanywestcentral-1.yaml
helm upgrade --install yb-germanywestcentral-2 yugabytedb/yugabyte --version 2.25.1 --namespace yugabytedb --wait -f values-single-namespace-yb-germanywestcentral-2.yaml
helm upgrade --install yb-germanywestcentral-3 yugabytedb/yugabyte --version 2.25.1 --namespace yugabytedb --wait -f values-single-namespace-yb-germanywestcentral-3.yaml

kubectl exec -it --namespace yugabytedb yb-germanywestcentral-1-yb-master-0 -- bash -c "/home/yugabyte/master/bin/yb-admin --master_addresses yb-germanywestcentral-1-yb-master-0.yb-germanywestcentral-1-yb-masters.yugabytedb.svc.cluster.local:7100,yb-germanywestcentral-2-yb-master-0.yb-germanywestcentral-2-yb-masters.yugabytedb.svc.cluster.local:7100,yb-germanywestcentral-3-yb-master-0.yb-germanywestcentral-3-yb-masters.yugabytedb.svc.cluster.local:7100 modify_placement_info azure.germanywestcentral.germanywestcentral-1,azure.germanywestcentral.germanywestcentral-2,azure.germanywestcentral.germanywestcentral-3 3"

Getting Started with YugabyteDB

Rico Berger — Sun, 13 Apr 2025 13:00:00 +0000

Hi and welcome to another database blog post. Last time, we explored Vitess; this time, we will look at Yugabyte. We will set up a simple Yugabyte cluster on Kubernetes. Once the cluster is running, we will create the commerce schema from the "Getting Started with Vitess" blog post and experiment with the cluster. Finally, we will examine how to monitor a Yugabyte cluster using Prometheus. Please note that I am not an expert in YugabyteDB; I simply want to experiment with it in this blog post.

Before we begin setting up YugabyteDB, we need a running Kubernetes cluster. If you want to try this on your local machine, you can use a tool like kind to create a local cluster.

If you are not familar with the architecture and key concepts of YugabyteDB, I recommend reviewing the documentation before proceeding. All the configuration files we are using can be found in the ricoberger/playground GitHub repository.

Installation

We will create our YugabyteDB cluster using the Helm chart. To install the Helm chart, we will create a new namespace called yugabytedb and set up a new cluster using the 002_values.yaml values file. Afterward, we can verify that the cluster is running by executing kubectl get pods.

kubectl apply --server-side -f 001_namespace.yaml

helm repo add yugabytedb https://charts.yugabyte.com
helm repo update

helm upgrade --install yugabytedb yugabytedb/yugabyte --version 2.25.1 --namespace yugabytedb --wait -f 002_values.yaml

NAME           READY   STATUS    RESTARTS   AGE
yb-master-0    3/3     Running   0          2m4s
yb-master-1    3/3     Running   0          2m4s
yb-master-2    3/3     Running   0          2m4s
yb-tserver-0   3/3     Running   0          2m4s
yb-tserver-1   3/3     Running   0          2m4s
yb-tserver-2   3/3     Running   0          2m4s

After a few minutes, it should show that all pods are in the status of running. At the point we can also check the state of the cluster using the yb_servers() database function:

# Password: mypassword
kubectl exec -it -n yugabytedb yb-tserver-0 -- ysqlsh --dbname yugabyte --username yugabyte --password -c "select * from yb_servers();"

                         host                          | port | num_connections | node_type | cloud  |   region    | zone  |                       public_ip                       |               uuid
-------------------------------------------------------+------+-----------------+-----------+--------+-------------+-------+-------------------------------------------------------+----------------------------------
 yb-tserver-0.yb-tservers.yugabytedb.svc.cluster.local | 5433 |               0 | primary   | cloud1 | datacenter1 | rack1 | yb-tserver-0.yb-tservers.yugabytedb.svc.cluster.local | 416a684d83e74d96962b95b2128b9870
 yb-tserver-2.yb-tservers.yugabytedb.svc.cluster.local | 5433 |               0 | primary   | cloud1 | datacenter1 | rack1 | yb-tserver-2.yb-tservers.yugabytedb.svc.cluster.local | 579810f2f1cc46adab7ed05ce00dde64
 yb-tserver-1.yb-tservers.yugabytedb.svc.cluster.local | 5433 |               0 | primary   | cloud1 | datacenter1 | rack1 | yb-tserver-1.yb-tservers.yugabytedb.svc.cluster.local | 7119b812c43448bc8d72e6b5580cdff8
(3 rows)

Create a User, Database and Schema

Now that we have a running YugabyteDB cluster, we can create a user, a database, and the schema for our commerce application. We create a database named commerce and a user named commerceuser. The user gets all priveleges to access the database and schema.

# Password: mypassword
kubectl exec -it -n yugabytedb yb-tserver-0 -- ysqlsh --dbname yugabyte --username yugabyte --password

-- Create database
create database commerce;

-- Create user
create role commerceuser login password 'commercepassword';

-- Grant priveleges to user con database
grant all on database commerce to commerceuser;

-- Connect to database
\c commerce

-- Grant priveleges to user on schema in database
grant all on schema public to commerceuser;

Now we can log in to the commerce database using the newly created user.

# Password: commercepassword
kubectl exec -it -n yugabytedb yb-tserver-0 -- ysqlsh --dbname commerce --username commerceuser --password

Once we connect to the commerce database, we can create the schema and insert data into the tables we create.

-- Create schema
create table if not exists product(
  sku varchar(128),
  description varchar(128),
  price bigint,
  primary key(sku)
);

create table if not exists customer(
  customer_id bigserial,
  email varchar(128),
  primary key(customer_id)
);

create table if not exists corder(
  order_id bigserial,
  customer_id bigint,
  sku varchar(128),
  price bigint,
  primary key(order_id)
);

-- Load data
insert into product(sku, description, price) values('SKU-1001', 'Monitor', 100);
insert into product(sku, description, price) values('SKU-1002', 'Keyboard', 30);

-- Show relations
\d

-- Close database session
\q

At this point, we have our initial schema, and it's time to examine the created pods, tables, and tablets. As shown in the graphic below, we created two StatefulSets during the installation of the YugabyteDB cluster. The yb-master contains three YB-Master pods. We can check which of the three is the current leader using the yb-admin list_all_masters command. In our case, the leader is the yb-master-0 pod.

Additionally, we can see that one tablet was created for each table using the yb_admin list_tablets .

and yb-admin list_tablet_servers commands. For example, the leader of the product table is yb-tserver-0, while the other two pods are followers.

Additionally, the yb-tserver StatefulSet contains three YB-TServer pods. Using the yb_admin list_tablets .

and yb-admin list_tablet_servers commands we can see how our created tables are splitted into tablets and how the tablets are distributed. For example for the product table one tablet was created, which is replicated three times. The leader is running on yb-tserver-2.

Note: For you the architecture could slightly different:

The YB-Master leader could be on another pod

The leader for each tablet could be on another YB-TServer pod

Details

Getting Started with Vitess

Rico Berger — Thu, 10 Apr 2025 13:00:00 +0000

Hi and welcome to another blog post. Today, we will explore Vitess. We will set up a simple Vitess cluster on Kubernetes. Once the cluster is running, we will discuss how to move tables and reshard the cluster. Finally, we will look at how to monitor a Vitess cluster using Prometheus. Please note that I am not an expert in Vitess; I simply want to experiment with it in this blog post.

Before we begin setting up Vitess, we need a running Kubernetes cluster. If you want to try this on your local machine, you can use a tool like kind to create a local cluster. Additionally, we need to install the MySQL Client and vtctldclient locally:

go install vitess.io/vitess/go/cmd/vtctldclient@v0.21.3
brew install mysql@8.4
brew link mysql@8.4

If you are not familiar with the architecture and concepts of Vitess, I recommend reviewing the documentation before proceeding. All the configuration files we are using can be found in the ricoberger/playground GitHub repository.

Installation

We will create our Vitess cluster using the Vitess Operator. To install the operator, we will create a new namespace called vitess, install the CRDs, and set up the operator using the following commands. Afterward, we can verify that the operator is running by executing kubectl get pods.

kubectl apply --server-side -f 001_namespace.yaml
kubectl apply --server-side -f 002_crds.yaml
kubectl apply --server-side -f 003_operator.yaml

NAME                               READY   STATUS    RESTARTS   AGE
vitess-operator-7cc877ccc5-vdndl   1/1     Running   0          21s

Once the operator is running, we can launch our first Vitess cluster. The cluster will use one cell (zone1) that includes all the control plane components (VTAdmin, vtctld, Topology Store) and one keyspace named commerce, which will contain one primary tablet and one replica tablet.

To bring up the cluster we can apply the 101_initial_cluster.yaml manifest. Afterwards we can check the state of the cluster using kubectl get pods. After a few minutes, it should show that all pods are in the status of running.

kubectl apply --server-side -f 101_initial_cluster.yaml

NAME                                                         READY   STATUS    RESTARTS      AGE
example-commerce-x-x-zone1-vtorc-c13ef6ff-86bd96dfb4-kp8w5   1/1     Running   2 (59s ago)   71s
example-etcd-faf13de3-1                                      1/1     Running   0             72s
example-etcd-faf13de3-2                                      1/1     Running   0             72s
example-etcd-faf13de3-3                                      1/1     Running   0             72s
example-vttablet-zone1-2469782763-bfadd780                   3/3     Running   2 (46s ago)   71s
example-vttablet-zone1-2548885007-46a852d0                   3/3     Running   1 (46s ago)   71s
example-zone1-vtadmin-c03d7eae-68d845dbfd-wnlk9              2/2     Running   0             72s
example-zone1-vtctld-1d4dcad0-75f6fb7c6b-78rpv               1/1     Running   1 (51s ago)   72s
example-zone1-vtgate-bc6cde92-57fdc84bb6-cdj75               1/1     Running   2 (45s ago)   72s
vitess-operator-7cc877ccc5-vdndl                             1/1     Running   0             2m29s

For ease-of-use, Vitess provides a script to port-forward from Kubernetes to our local machine. This script also recommends setting up aliases for mysql and vtctldclient. Once the port-forward starts running, the VTAdmin UI will be available at http://localhost:14000/.

alias vtctldclient="vtctldclient --server=localhost:15999"
alias mysql="mysql -h 127.0.0.1 -P 15306 -u user"
./pf.sh &

In the last step of the initial installation we will create our initial scheme, which will deploy a single unsharded keyspace named commerce, with the following tables:

The product table contains the product information for all of the products.
The customer table has a customer_id that has an auto_increment. A typical customer table would have a lot more columns, and sometimes additional detail tables.
The corder table (named so because order is an SQL reserved word) has an order_id auto-increment column. It also has foreign keys into customer(customer_id) and product(sku).

vtctldclient ApplySchema --sql-file="102_create_commerce_schema.sql" commerce
vtctldclient ApplyVSchema --vschema-file="103_vschema_commerce_initial.json" commerce

We should now be able to connect to the VTGate Server in our cluster by running the mysql command.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| commerce           |
| information_schema |
| mysql              |
| sys                |
| performance_schema |
+--------------------+
5 rows in set (0.02 sec)

Move Tables

In the next step we will create a new keyspace named customer and move the customer and corder tables to the newly created keyspace. This is the recommended approach before splitting a single table across multiple servers (sharding).

Let's start by loading some data into our created tables and looking at the data we inserted. Notice that all of our tables are currently in the commerce schema/keyspace here.

mysql < 201_insert_commerce_data.sql
mysql --table < 202_select_commerce_data.sql

Using commerce
Customer
+-------------+--------------------+
| customer_id | email              |
+-------------+--------------------+
|           1 | alice@domain.com   |
|           2 | bob@domain.com     |
|           3 | charlie@domain.com |
|           4 | dan@domain.com     |
|           5 | eve@domain.com     |
+-------------+--------------------+
Product
+----------+-------------+-------+
| sku      | description | price |
+----------+-------------+-------+
| SKU-1001 | Monitor     |   100 |
| SKU-1002 | Keyboard    |    30 |
+----------+-------------+-------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        1 |           1 | SKU-1001 |   100 |
|        2 |           2 | SKU-1002 |    30 |
|        3 |           3 | SKU-1002 |    30 |
|        4 |           4 | SKU-1002 |    30 |
|        5 |           5 | SKU-1002 |    30 |
+----------+-------------+----------+-------+

When we list our tablets using the following command, we can see that we have two tablets running: one primary and one replica.

mysql -e "show vitess_tablets"

+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State   | Alias            | Hostname     | PrimaryTermStartTime |
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
| zone1 | commerce | -     | PRIMARY    | SERVING | zone1-2469782763 | 10.244.5.244 | 2025-04-10T06:08:22Z |
| zone1 | commerce | -     | REPLICA    | SERVING | zone1-2548885007 | 10.244.14.73 |                      |
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+

Now it is time to deploy new tablets for our customer keyspace by applying the 203_customer_tablets.yaml manifest. After some minutes we should see the newly created tablets in a running state. We should also see that a new vtorc instance was created for the customer keyspace.

kubectl apply --server-side -f 203_customer_tablets.yaml

NAME                                                         READY   STATUS    RESTARTS        AGE
example-commerce-x-x-zone1-vtorc-c13ef6ff-86bd96dfb4-kp8w5   1/1     Running   2 (5m52s ago)   6m4s
example-customer-x-x-zone1-vtorc-53d270f6-7754f557c-bb87n    1/1     Running   0               72s
example-etcd-faf13de3-1                                      1/1     Running   0               6m5s
example-etcd-faf13de3-2                                      1/1     Running   0               6m5s
example-etcd-faf13de3-3                                      1/1     Running   0               6m5s
example-vttablet-zone1-1250593518-17c58396                   3/3     Running   0               72s
example-vttablet-zone1-2469782763-bfadd780                   3/3     Running   2 (5m39s ago)   6m4s
example-vttablet-zone1-2548885007-46a852d0                   3/3     Running   1 (5m39s ago)   6m4s
example-vttablet-zone1-3778123133-6f4ed5fc                   3/3     Running   2 (35s ago)     72s
example-zone1-vtadmin-c03d7eae-68d845dbfd-wnlk9              2/2     Running   0               6m5s
example-zone1-vtctld-1d4dcad0-75f6fb7c6b-78rpv               1/1     Running   1 (5m44s ago)   6m5s
example-zone1-vtgate-bc6cde92-57fdc84bb6-cdj75               1/1     Running   2 (5m38s ago)   6m5s
vitess-operator-7cc877ccc5-vdndl                             1/1     Running   0               7m22s

Before we continue we restart the port-forward after launching the pods has completed. Afterwards we can list our tables again. We can see that we have four tablets now. The two existing ones for the commerce keyspace and two new ones for the customer keyspace.

killall kubectl
./pf.sh &

mysql -e "show vitess_tablets"

+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State   | Alias            | Hostname     | PrimaryTermStartTime |
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
| zone1 | commerce | -     | PRIMARY    | SERVING | zone1-2469782763 | 10.244.5.244 | 2025-04-10T06:08:22Z |
| zone1 | commerce | -     | REPLICA    | SERVING | zone1-2548885007 | 10.244.14.73 |                      |
| zone1 | customer | -     | PRIMARY    | SERVING | zone1-1250593518 | 10.244.11.55 | 2025-04-10T06:13:10Z |
| zone1 | customer | -     | REPLICA    | SERVING | zone1-3778123133 | 10.244.8.170 |                      |
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+

In the next step we will create a MoveTables workflow, which copies the tables from the commerce keyspace into the customer keyspace. This operation does not block any database activity.

vtctldclient MoveTables --target-keyspace customer --workflow commerce2customer create --source-keyspace commerce --tables 'customer,corder'

To see what happens under the covers, let's look at the routing rules that the MoveTables operation created. These are instructions used by a VTGate to determine which backend keyspace to send requests to for a given table.

vtctldclient GetRoutingRules

We can monitor the progress of the MoveTables operation using the status action. We can also validate its correctness by performing a logical diff between the source and target to confirm that they are fully synced with the VDiff command.

# Monitoring Progress
vtctldclient MoveTables --target-keyspace customer --workflow commerce2customer status --format=json

{
  "table_copy_state": {},
  "shard_streams": {
    "customer/-": {
      "streams": [
        {
          "id": 1,
          "tablet": {
            "cell": "zone1",
            "uid": 1250593518
          },
          "source_shard": "commerce/-",
          "position": "2a5c9b12-15d2-11f0-a7a3-a2742b031576:1-40",
          "status": "Running",
          "info": "VStream Lag: 0s"
        }
      ]
    }
  },
  "traffic_state": "Reads Not Switched. Writes Not Switched"
}

# Validate Correctness
vtctldclient VDiff --target-keyspace customer --workflow commerce2customer create
vtctldclient VDiff --format=json --target-keyspace customer --workflow commerce2customer show last --verbose

VDiff 1163c387-4284-4214-9896-74b3af6a0cef scheduled on target shards, use show to view progress
{
  "Workflow": "commerce2customer",
  "Keyspace": "customer",
  "State": "started",
  "UUID": "1163c387-4284-4214-9896-74b3af6a0cef",
  "RowsCompared": 0,
  "HasMismatch": false,
  "Shards": "-",
  "StartedAt": "2025-04-10 06:19:56",
  "TableSummary": {
    "corder": {
      "TableName": "corder",
      "State": "started",
      "RowsCompared": 0,
      "MatchingRows": 0,
      "MismatchedRows": 0,
      "ExtraRowsSource": 0,
      "ExtraRowsTarget": 0
    },
    "customer": {
      "TableName": "customer",
      "State": "pending",
      "RowsCompared": 0,
      "MatchingRows": 0,
      "MismatchedRows": 0,
      "ExtraRowsSource": 0,
      "ExtraRowsTarget": 0
    }
  },
  "Reports": {
    "corder": {
      "-": {
        "TableName": "",
        "ProcessedRows": 0,
        "MatchingRows": 0,
        "MismatchedRows": 0,
        "ExtraRowsSource": 0,
        "ExtraRowsTarget": 0
      }
    },
    "customer": {
      "-": {
        "TableName": "",
        "ProcessedRows": 0,
        "MatchingRows": 0,
        "MismatchedRows": 0,
        "ExtraRowsSource": 0,
        "ExtraRowsTarget": 0
      }
    }
  },
  "Progress": {
    "Percentage": 0
  }
}

Once the MoveTables operation is complete, the first step in making the changes live is to switch all query serving traffic from the old commerce keyspace to the customer keyspace for the tables we moved. Queries against the other tables will continue to route to the commerce keyspace.

vtctldclient MoveTables --target-keyspace customer --workflow commerce2customer SwitchTraffic

If we now look at the routing rules after the SwitchTraffic step, we will see that all queries against the customer and corder tables will get routed to the customer keyspace.

vtctldclient GetRoutingRules

The final step is to complete the migration using the Complete action. This will (by default) get rid of the routing rules that were created and DROP the original tables in the source keyspace (commerce). Along with freeing up space on the original tablets, this is an important step to eliminate potential future confusion.

vtctldclient MoveTables --target-keyspace customer --workflow commerce2customer complete

Resharding

In this step, we will divide our customer keyspace into two shards. The final architecture will appear as shown in the graphic below.

Before we can start we have to create a sequence table for our auto-increment columns and we have to decide for sharding keys or Primary Vindexes within a VSchema. More information regarding these two topic can be found in the Vitess documentation¹. To create the sequence tables and VSchema we can run the following commands:

vtctldclient ApplySchema --sql="$(cat 301_create_commerce_seq.sql)" commerce
vtctldclient ApplyVSchema --vschema="$(cat 302_vschema_commerce_seq.json)" commerce
vtctldclient ApplyVSchema --vschema="$(cat 303_vschema_customer_sharded.json)" customer
vtctldclient ApplySchema --sql="$(cat 304_create_customer_sharded.sql)" customer

At this point, you have finalized our sharded VSchema and vetted all the queries to make sure they still work. Now, it’s time to reshard. To do this we will create the target shards by applying the 305_new_shards.yaml manifest. Afterwards some minutes we should see four new tablets and two new vtorc pods (via kubectl get pods) for the created target shards.

kubectl apply --server-side -f 305_new_shards.yaml

# Restart the port-forward afterwards:
killall kubectl
./pf.sh &

NAME                                                          READY   STATUS    RESTARTS      AGE
example-commerce-x-x-zone1-vtorc-c13ef6ff-86bd96dfb4-kp8w5    1/1     Running   2 (19m ago)   19m
example-customer-80-x-zone1-vtorc-836adff9-b67657589-ndpxq    1/1     Running   0             76s
example-customer-x-80-zone1-vtorc-2bf8b95e-86b8b56fbc-q69h4   1/1     Running   0             76s
example-customer-x-x-zone1-vtorc-53d270f6-7754f557c-bb87n     1/1     Running   0             14m
example-etcd-faf13de3-1                                       1/1     Running   0             19m
example-etcd-faf13de3-2                                       1/1     Running   0             19m
example-etcd-faf13de3-3                                       1/1     Running   0             19m
example-vttablet-zone1-0118374573-10d08e80                    3/3     Running   2 (35s ago)   76s
example-vttablet-zone1-0120139806-fed29577                    3/3     Running   0             76s
example-vttablet-zone1-1250593518-17c58396                    3/3     Running   0             14m
example-vttablet-zone1-2289928654-7de47379                    3/3     Running   0             76s
example-vttablet-zone1-2469782763-bfadd780                    3/3     Running   2 (19m ago)   19m
example-vttablet-zone1-2548885007-46a852d0                    3/3     Running   1 (19m ago)   19m
example-vttablet-zone1-3778123133-6f4ed5fc                    3/3     Running   2 (13m ago)   14m
example-vttablet-zone1-4277914223-0f04a9a6                    3/3     Running   0             76s
example-zone1-vtadmin-c03d7eae-68d845dbfd-wnlk9               2/2     Running   0             19m
example-zone1-vtctld-1d4dcad0-75f6fb7c6b-78rpv                1/1     Running   1 (19m ago)   19m
example-zone1-vtgate-bc6cde92-57fdc84bb6-cdj75                1/1     Running   2 (19m ago)   19m
vitess-operator-7cc877ccc5-vdndl                              1/1     Running   0             20m

Now we can start the Reshard operation. It occurs online, and will not block any read or write operations to your database:

vtctldclient Reshard --target-keyspace customer --workflow cust2cust create --source-shards '-' --target-shards '-80,80-'

After the reshard is complete, we can use VDiff to check data integrity and ensure our source and target shards are consistent:

vtctldclient VDiff --target-keyspace customer --workflow cust2cust create
vtctldclient VDiff --format=json --target-keyspace customer --workflow cust2cust show last

VDiff da8b1af8-eaf0-415b-9b63-4d3606798435 scheduled on target shards, use show to view progress
{
  "Workflow": "cust2cust",
  "Keyspace": "customer",
  "State": "started",
  "UUID": "da8b1af8-eaf0-415b-9b63-4d3606798435",
  "RowsCompared": 4,
  "HasMismatch": false,
  "Shards": "-80,80-",
  "StartedAt": "2025-04-10 06:29:57",
  "Progress": {
    "Percentage": 100,
    "ETA": "2025-04-10 06:29:57"
  }
}

After validating for correctness, the next step is to switch all traffic from the source shards to the target shards:

vtctldclient Reshard --target-keyspace customer --workflow cust2cust SwitchTraffic

We should now be able to see the data that has been copied over to the new shards:

mysql --table < 306_select_customer-80_data.sql
mysql --table < 307_select_customer80-_data.sql

Using customer/-80
Customer
+-------------+--------------------+
| customer_id | email              |
+-------------+--------------------+
|           1 | alice@domain.com   |
|           2 | bob@domain.com     |
|           3 | charlie@domain.com |
|           5 | eve@domain.com     |
+-------------+--------------------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        1 |           1 | SKU-1001 |   100 |
|        2 |           2 | SKU-1002 |    30 |
|        3 |           3 | SKU-1002 |    30 |
|        5 |           5 | SKU-1002 |    30 |
+----------+-------------+----------+-------+

Using customer/80-
Customer
+-------------+----------------+
| customer_id | email          |
+-------------+----------------+
|           4 | dan@domain.com |
+-------------+----------------+
COrder
+----------+-------------+----------+-------+
| order_id | customer_id | sku      | price |
+----------+-------------+----------+-------+
|        4 |           4 | SKU-1002 |    30 |
+----------+-------------+----------+-------+

We can now complete the created Reshard workflow and remove the shard that is no longer required:

vtctldclient Reshard --target-keyspace customer --workflow cust2cust complete
kubectl apply --server-side -f 308_down_shard_-.yaml
kubectl delete vitessshards.planetscale.com example-customer-x-x-dc880356

Afterwards the list of running pods should look as follows. As we can see the two tablets for the old shard as well as the vtorc pod were removed.

NAME                                                          READY   STATUS    RESTARTS      AGE
example-commerce-x-x-zone1-vtorc-c13ef6ff-86bd96dfb4-kp8w5    1/1     Running   2 (33m ago)   33m
example-customer-80-x-zone1-vtorc-836adff9-b67657589-ndpxq    1/1     Running   0             15m
example-customer-x-80-zone1-vtorc-2bf8b95e-86b8b56fbc-q69h4   1/1     Running   0             15m
example-etcd-faf13de3-1                                       1/1     Running   0             33m
example-etcd-faf13de3-2                                       1/1     Running   0             33m
example-etcd-faf13de3-3                                       1/1     Running   0             33m
example-vttablet-zone1-0118374573-10d08e80                    3/3     Running   2 (14m ago)   15m
example-vttablet-zone1-0120139806-fed29577                    3/3     Running   0             15m
example-vttablet-zone1-2289928654-7de47379                    3/3     Running   0             15m
example-vttablet-zone1-2469782763-bfadd780                    3/3     Running   2 (33m ago)   33m
example-vttablet-zone1-2548885007-46a852d0                    3/3     Running   1 (33m ago)   33m
example-vttablet-zone1-4277914223-0f04a9a6                    3/3     Running   0             15m
example-zone1-vtadmin-c03d7eae-68d845dbfd-wnlk9               2/2     Running   0             33m
example-zone1-vtctld-1d4dcad0-75f6fb7c6b-78rpv                1/1     Running   1 (33m ago)   33m
example-zone1-vtgate-bc6cde92-57fdc84bb6-cdj75                1/1     Running   2 (33m ago)   33m
vitess-operator-7cc877ccc5-vdndl                              1/1     Running   0             34m

Monitoring

To monitor our Vitess cluster we will use Prometheus and Grafana. We will not go through the setup of Prometheus and Grafana within this post and assume that we already have a running Prometheus and Grafana instance. To monitor our Vitess cluster with Prometheus and Grafana we will create a scrape configuration for Prometheus and import some dashboards² into Grafana.

kubectl apply --server-side -f 401_monitoring.yaml

Once we get some data in the dashboards, we can also generate some load by running the example application. The following command will create customers in our customer table. We can increase the load after some time by restarting the command with the -goroutines flag.

go run . -create-customers
go run . -create-customers -goroutines=20

The following commands will create some orders in our corders table. To create a new order we select a random customer, all products and create a new order for the selected customer and one of the selected products.

go run . -create-orders -goroutines=10
go run . -create-orders -goroutines=100

Now we can also restart some tablets and monitor the behaviour of Vitess via the created dashboards. In teh following we test the following scenarios:

Delete the primary tablet for the commerce keyspace
Delete the replica for shard 80- in the customer keyspace
Delete the primary for shard 80- in the customer keyspace

Last but not least, we can monitor a single tablet by creating a port forward and opening http://localhost:15005 in our browser. The dashboard displays the number of queries per second, the current query and transaction log, real-time queries, and much more.

kubectl port-forward example-vttablet-zone1-4277914223-0f04a9a6 15005:15000

That's it for today's post. I had a lot of fun playing around with Vitess and hopefully gained a better understanding of how it works. I hope you also enjoyed the post, and I'll see you next time.

Docuemntation for Sequences and Vindexes ↩︎
We will use the dashboards from the following Gist ↩︎

Use OpenTelemetry Traces in React Applications

Rico Berger — Sat, 29 Mar 2025 11:00:00 +0000

Some years ago, I watched the KubeCon talk "Tracing is For Everyone: Tracing User Events with GraphQL and OpenTelemetry" by Nina Stawski. Since then, I have wanted to experiment with instrumenting frontend apps using OpenTelemetry. Recently, the topic came up at work, so I finally created a small proof of concept to explore this further.

For this proof of concept, we are creating a simple task management app using React and an Express API. The app allows users to create tasks, delete tasks, and mark tasks as completed. The tasks are stored in memory on the Express server. The source code for the complete app is available in the ricoberger/playground repository. A screenshot of the final app is provided below.

To instrument our React application with OpenTelemetry, we create a TraceProvider.jsx file in our project. In the first step, we import the WebTracerProvider class from @opentelemetry/sdk-trace-web to create a new trace provider that allows us to automatically trace activities in the browser. Within the trace provider, we can add resources to the spans, such as the service name. We also need to set the span processors:

The ConsoleSpanExporter will print all spans to the console. We use it solely for testing purposes. In a production setup, this exporter should not be used.
The OTLPTraceExporter exports traces to the OpenTelemetry collector and requires an endpoint that ends with /v1/traces. In our case, we will use the same address as our frontend and proxy all requests to the endpoint for the OpenTelemetry collector. We will examine this in more detail later.

const provider = new WebTracerProvider({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: "react-app",
  }),
  spanProcessors: [
    new SimpleSpanProcessor(new ConsoleSpanExporter()),
    new SimpleSpanProcessor(
      new OTLPTraceExporter({
        url: "http://localhost:3000/v1/traces",
      }),
    ),
  ],
});

In the next step, we register the trace provider for use with the OpenTelemetry API. We utilize the ZoneContextManager to trace asynchronous operations. Additionally, we set the appropriate propagators to ensure the trace context is passed between services. In our case, we use the B3Propagator and the W3CTraceContextPropagator.

provider.register({
  contextManager: new ZoneContextManager(),
  propagator: new CompositePropagator({
    propagators: [new B3Propagator(), new W3CTraceContextPropagator()],
  }),
});

Afterward, we register the instrumentations for the OpenTelemetry SDK. We use DocumentLoadInstrumentation to automatically create a span for the page load and all loaded assets, and FetchInstrumentation to trace all API requests. We also set the urls for which we want to propagate the trace header and clear the timing resources.

registerInstrumentations({
  instrumentations: [
    new DocumentLoadInstrumentation(),
    new FetchInstrumentation({
      propagateTraceHeaderCorsUrls: ["http://localhost:3000"],
      clearTimingResources: true,
    }),
  ],
});

Lastly, we export a TraceProvider component. This component will wrap our root component to ensure that the tracing setup is properly initialized.

export function TraceProvider(props) {
  return <>{props.children};
}

At this point, we have automatically instrumented all document loads and API requests. To create spans manually, we need to create a new tracer using our provider.

export const webTracer = provider.getTracer("web-tracer");

Now we can use the webTracer to create a new span. We can then execute a function, in this case, an API call, within the context of the span and add events to it.

function toggleTaskCompleted(id) {
  const span = webTracer.startSpan("toggleTaskCompleted");
  context.with(trace.setSpan(context.active(), span), () => {
    fetch("/api/tasks/" + id, {
      method: "PUT",
      headers: {
        "Content-Type": "application/json",
        Accept: "application/json",
      },
    })
      .then((res) => {
        trace.getSpan(context.active()).addEvent("parseJson");
        return res.json();
      })
      .then((data) => {
        trace.getSpan(context.active()).addEvent("updateTask");
        const updatedTasks = tasks.map((task) => {
          if (id === task.id) {
            return data;
          }
          return task;
        });

        trace.getSpan(context.active()).addEvent("setTasks");
        span.end();
        setTasks(updatedTasks);
      });
  });
}

An example trace for when a user marks a task as done in our project can be found below. It includes our manually created span react-app: toggleTaskCompleted and the automatically generated span react-app: HTTP PUT from the FetchInstrumentation. Since we propagate the trace headers, the spans are also connected to those from our Express backend.

As promised, I would like to review the project setup and explain why we are using http://localhost:3000/v1/traces for the OTLPTraceExporter. We are running the project using a Docker Compose file, which starts four containers:

A jaeger container that runs Jaeger. Jaeger stores our traces and provides a user-friendly interface to view them.
A otel-collector container that runs the OpenTelemetry Collector. The collector receives all spans from our Express backend and React frontend, then forwards them to Jaeger. You can find the configuration for the collector here.
A backend container that runs our Express app.
A frontend container that runs NGINX to serve our frontend app. In the NGINX configuration, we define that all requests starting with /api are routed to the backend container, while requests starting with /v1/traces are routed to the otel-collector container.

That's it! If you want to try this on your own, you can check out the example project and start it with docker compose -f docker-compose.yaml up --force-recreate --build. Afterward, you can open http://localhost:3000 in your browser to create, delete, and mark tasks as completed. If you access the Jaeger frontend at http://localhost:16686, you should see some traces from our React frontend and Express backend.

Use a VPN to Access your Homelab

Rico Berger — Mon, 24 Mar 2025 20:00:00 +0000

After my last blog post, a colleague (hi Falk 👋) shared insights about his homelab setup. Since I'm always open to new ideas, today's post will explore how to access our homelab via VPN and how to use Traefik, Cloudflare, and Let's Encrypt to obtain a free certificate for accessing our services via HTTPS.

Setup a VPN

Fortunately, the router we are using in this blog post supports access to the home network via a VPN using WireGuard in the background¹. We can simply activate the function and download the configuration file.

When we examine the downloaded configuration file, we see that we will use the public IP address of our router to connect to the VPN from outside. However, as we are getting a dynamically assigned public IP address, we have to note down the address every day before leaving the house. Since this is very uncomfortable, we can register a dynamic host record (DynDNS). There are many DynDNS providers available; however, we will build our own using Cloudflare.

In the first step we need to create a API token at Cloudflare, which has the permissions to create and update DNS records.

Type	Item	Permission
Zone	DNS	Edit

Now we will create a new DNS record at Cloudflare for our VPN that points to the public IP address of our router.

export CLOUDFLARE_ZONE_ID=
export CLODUFLARE_API_TOKEN=

curl https://api.cloudflare.com/client/v4/zones/${CLOUDFLARE_ZONE_ID}/dns_records \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "content": "",
    "name": "vpn.homelab.ricoberger.dev",
    "proxied": false,
    "ttl": 300,
    "type": "A"
  }'

To update the DNS record whenever our public IP address changes, we can create a small script². This script will retrieve our public IP address and the ID of the DNS record, then update the DNS record by ID with the new IP address.

#!/usr/bin/env bash

ipaddress=$(curl 'https://api.ipify.org?format=json' | jq -r .ip)

dnsrecord=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/${CLOUDFLARE_ZONE_ID}/dns_records?type=A&name=vpn.homelab.ricoberger.dev" \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -H "Content-Type: application/json")


dnsrecordid=$(echo "${dnsrecord}" | jq -r '{"result"}[] | .[0] | .id')
dnsrecordipaddress=$(echo "${dnsrecord}" | jq -r '{"result"}[] | .[0] | .content')

if [ "${ipaddress}" == "${dnsrecordipaddress}" ]; then
  echo "DNS record is already up to date"
  exit 0
fi

curl -X PUT https://api.cloudflare.com/client/v4/zones/${CLOUDFLARE_ZONE_ID}/dns_records/${dnsrecordid} \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "content": "'${ipaddress}'",
    "name": "vpn.homelab.ricoberger.dev",
    "proxied": false,
    "ttl": 300,
    "type": "A"
  }'

Last but not least we create a new cronjob to run the script every five minutes. To add a new cronjob we run crontab -e and add the following entry:

*/5 * * * * update-dns.sh

At this point, we have a DNS record (vpn.homelab.ricoberger.dev) to access our VPN and have ensured that it automatically updates when our router's public IP address changes. In the next step, we will set up Traefik with a Let's Encrypt certificate to access our services via HTTPS.

Setup Traefik

We will deploy Traefik using Docker and Docker Compose. For that we are creating a docker-compose.yaml file with the following content:

services:
  traefik:
    container_name: traefik
    image: traefik:v3.3.4
    restart: always
    security_opt:
      - no-new-privileges:true
    # Use the "proxy" network for the Traefik container
    networks:
      - proxy
    # Map the http and https ports to the host system
    ports:
      - 80:80
      - 443:443
    # Mount the ".env" file which contains our secrets
    env_file: .env
    # - The "docker.sock" file is required to use the Docker provider within
    #   Traefik, see https://doc.traefik.io/traefik/providers/docker/
    # - The "traefik.yaml" file contains the Traefik configuration
    # - The "acme.json" file is used to store the certificate which is issued by
    #   Let's Encrypt
    # - The "config.yaml" file holds the configuration for the File provider,
    #   see https://doc.traefik.io/traefik/providers/file/
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data/traefik.yaml:/traefik.yaml
      - ./data/acme.json:/acme.json
      - ./data/config.yaml:/config.yaml
    # To be able to redirect traffic from Traefik to a service, which is running
    # on the host system we have to add the following extra hosts
    extra_hosts:
      - host.docker.internal:host-gateway
    # Add labels to the Traefik container to configure the entrypoint, the host
    # which can be used to access the Traefik dashboard, the domains for which
    # the certificate should be issued and the basic auth and http to https
    # redirect middleware
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.homelab.ricoberger.dev`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.homelab.ricoberger.dev`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=homelab.ricoberger.dev"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.homelab.ricoberger.dev"
      - "traefik.http.routers.traefik-secure.service=api@internal"

networks:
  proxy:
    external: true

In the docker-compose.yaml file above, we defined our intention to use the proxy Docker network for Traefik. To create the Docker network, we can run the following command:

docker network create proxy

The .env file containing our secrets should include the Cloudflare API token, which we used to create the DNS record for the VPN. It should also have the email address used to issue the certificate and the credentials for the Traefik dashboard. You can create the credentials by running echo $(htpasswd -nB admin) | sed -e s/\\$/\\$\\$/g.

CF_DNS_API_TOKEN=
TRAEFIK_CERTIFICATESRESOLVERS_CLOUDFLARE_ACME_EMAIL=
TRAEFIK_DASHBOARD_CREDENTIALS=

To create the acme.json file, which will store the certificate issued by Let's Encrypt, we will run the following commands:

touch data/acme.json
chmod 600 data/acme.json

In the traefik.yaml file, we define the configuration for Traefik as follows:

log:
  level: DEBUG
api:
  # Enable the Traefik dashboard, see
  # https://doc.traefik.io/traefik/operations/dashboard/
  dashboard: true
  debug: true
# Define the http and https entrypoints and configure the http entrypoint to
# redirect all traffic to the https entrypoint
entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
serversTransport:
  insecureSkipVerify: true
# Configure the Docker provider, by setting the Docker endpoint to the mounted
# "docker.sock" file and the file provider by setting the name of the
# configuration file, which should be used
providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    filename: /config.yaml
# Configure the DNS provider (in our case Cloudflare), which should be used for
# creating the TXT records, required by the DNS-01 challenge to issue the
# certificates
#
# For more information regarding the DNS-01 challenge, see
# https://letsencrypt.org/docs/challenge-types/#dns-01-challenge
certificatesResolvers:
  cloudflare:
    acme:
      storage: acme.json
      caServer: https://acme-v02.api.letsencrypt.org/directory
      # caServer: https://acme-staging-v02.api.letsencrypt.org/directory
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "1.0.0.1:53"

Last but not least, we can create the config.yaml file by running touch data/config.yaml. Then, we can start Traefik with the following command:

docker-compose -f docker-compose.yaml up -d --force-recreate

If we now take a look at the acme.json file (cat data/acme.json) we should see the issued certifacte for *.homelab.ricoberger.dev by Let's Encrypt:

{
  "cloudflare": {
    "Account": {
      "Email": "",
      "Registration": {
        "body": {
          "status": "valid"
        },
        "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/"
      },
      "PrivateKey": "",
      "KeyType": "4096"
    },
    "Certificates": [
      {
        "domain": {
          "main": "homelab.ricoberger.dev",
          "sans": ["*.homelab.ricoberger.dev"]
        },
        "certificate": "",
        "key": "",
        "Store": "default"
      }
    ]
  }
}

If we create a new DNS record similar to the one for the VPN, but using the local IP address instead of the public one (ipconfig getifaddr en0 || ipconfig getifaddr en1) for traefik.homelab.ricoberger.dev, we should be able to access the Traefik dashboard afterwards via our browser.

export CLOUDFLARE_ZONE_ID=
export CLODUFLARE_API_TOKEN=

curl https://api.cloudflare.com/client/v4/zones/${CLOUDFLARE_ZONE_ID}/dns_records \
  -H "Authorization: Bearer ${CLOUDFLARE_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "content": "",
    "name": "traefik.homelab.ricoberger.dev",
    "proxied": false,
    "ttl": 300,
    "type": "A"
  }'

At this point, we have created a wildcard certificate through Let's Encrypt, allowing us to use it for our services, such as the Traefik Dashboard, which can be accessed via HTTPS. In the final step, we will examine how to expose the Ollama API and Open WeUI from the blog post Mac mini as AI Server.

For the Ollama API, we are utilizing the file provider of Traefik since it runs on our host system rather than within a Docker container. To make the Ollama API accessible via ollama.homelab.ricoberger.dev, we need to add the following configuration to the data/config.yaml file:

http:
  # Create a new "ollama" router, which uses the "https" entrypoint and is used
  # when the "ollama.homeelab.ricoberger.de" domain is used
  routers:
    ollama:
      entryPoints:
        - "https"
      rule: "Host(`ollama.homelab.ricoberger.dev`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: ollama

  # Create a "ollama" service for the "ollama" router from above, where we set
  # the url of the Ollama API. Since the Ollama API is running on port 11434 on
  # host system, we have to use the "host.docker.internal" address
  services:
    ollama:
      loadBalancer:
        servers:
          - url: "http://host.docker.internal:11434"
        passHostHeader: true

For the Open WebUI we can create the following Docker compose file and run it via docker-compose -f docker-compose.yaml up -d --force-recreate

name: open-webui

services:
  open-webui:
    container_name: open-webui
    image: ghcr.io/open-webui/open-webui:main
    restart: always
    # Use the "proxy" network, so that the Open WebUI container is using the
    # same network as the Traefik container
    networks:
      - proxy
    volumes:
      - ./data:/app/backend/data
    # Add the extra hosts, so that the Open WebUI is able to access the Ollama
    # API
    extra_hosts:
      - host.docker.internal:host-gateway
    # Add the following labels, so that Traefik knows, the port of the Open
    # WebUI, the entrzpoint which should be used and the domain which should be
    # used
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.open-webui.rule=Host(`open-webui.homelab.ricoberger.dev`)"
      - "traefik.http.routers.open-webui.entrypoints=https"
      - "traefik.http.routers.open-webui.tls=true"
      - "traefik.http.services.open-webui.loadbalancer.server.port=8080"

networks:
  proxy:
    external: true

If we add a DNS record for ollama.homelab.ricoberger.dev and open-webui.homelab.ricoberger.dev to Cloudflare as before, we can access the Ollama API and the Open WebUI through their domains³.

That's it for today's post. You can find the complete setup in the ricoberger/playground repository.

If your router doesn't support this function, you can create the WireGuard setup manually by following one of the many available tutorials. The easiest one for me was the one from the Pi-hole documentation: Installing the WireGuard server. ↩︎
An improved version of the script can be found in the ricoberger/playground repository. ↩︎
If you prefer not to add DNS records for your homelab services to your DNS provider, you can run your own DNS server. I might write a follow-up blog post about this. ↩︎

Use Cloudflare Tunnels to Access your Homelab

Rico Berger — Sun, 16 Mar 2025 09:00:00 +0000

In our last blog post, we looked at how to set up an AI server on a Mac Mini and how to access the server in our homelab. In today's post, we will make the server available through Cloudflare Tunnels, allowing us to access it from anywhere.

Cloudflare Tunnel provides us with a secure way to connect our resources to Cloudflare without a publicly routable IP address. With Tunnel, we do not send traffic to an external IP — instead, a lightweight daemon in our infrastructure (cloudflared) creates outbound-only connections to Cloudflare's global network. Cloudflare Tunnel can connect HTTP web servers, SSH servers, remote desktops, and other protocols safely to Cloudflare. This way, our origins can serve traffic through Cloudflare without being vulnerable to attacks that bypass Cloudflare.¹

How It Works

cloudflared establishes outbound connections (tunnels) between our resources and Cloudflare's global network. Tunnels are persistent objects that route traffic to DNS records. Within the same tunnel, we can run as many cloudflared processes (connectors) as needed. These processes will establish connections to Cloudflare and send traffic to the nearest Cloudflare data center.

Setup a Tunnel

In the following, we will set up a Cloudflare Tunnel to expose the Open WebUI from the last blog post via openwebui-homelab.ricoberger.dev. We will create a new tunnel and DNS entry, connecting the Open WebUI through the tunnel with Cloudflare. This will route traffic from openwebui-homelab.ricoberger.dev to the localhost:3000 address, where the Open WebUI is running.

To proceed, we need to add a website to Cloudflare or change the domain nameservers for ricoberger.dev to Cloudflare. Once this is done, we can follow these steps to create the Cloudflare Tunnel and make our Open WebUI accessible via the specified domain:

Create an API token with the following permissions:

Type Item Permission

Account Cloudflare Tunnel Edit

Zone DNS Edit

Type	Item	Permission
Account	Cloudflare Tunnel	Edit
Zone	DNS	Edit

Create a tunnel, by making a POST request to the Cloudflare Tunnel endpoint:

export CLOUDFLARE_ACCOUNT_ID=
export CLOUDFLARE_API_TOKEN=

curl "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/cfd_tunnel" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  --data '{
    "name": "homelab",
    "config_src": "cloudflare"
  }'

{
  "success": true,
  "errors": [],
  "messages": [],
  "result": {
    "id": "",
    "account_tag": "",
    "created_at": "2025-03-15T12:46:52.095257Z",
    "deleted_at": null,
    "name": "homelab",
    "connections": [],
    "conns_active_at": null,
    "conns_inactive_at": "2025-03-15T12:46:52.095257Z",
    "tun_type": "cfd_tunnel",
    "metadata": {},
    "status": "inactive",
    "remote_config": true,
    "credentials_file": {
      "AccountTag": "",
      "TunnelID": "",
      "TunnelName": "homelab",
      "TunnelSecret": ""
    },
    "token": ""
  }
}

Copy the id and token values shown in the output. We will need these values to configure and run the tunnel.
```
export CLOUDFLARE_TUNNEL_ID=
export CLOUDFLARE_TUNNEL_TOKEN=
```

Make a PUT request to route our local service URL to a public hostname. Our ingress rules must include a catch-all rule at the end. In the following example, cloudflared will respond with a 404 status code when the request does not match any of the previous hostnames.

curl --request PUT \
  "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/cfd_tunnel/$CLOUDFLARE_TUNNEL_ID/configurations" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  --data '{
    "config": {
      "ingress": [
        {
          "hostname": "openwebui-homelab.ricoberger.dev",
          "service": "http://localhost:3000",
          "originRequest": {}
        },
        {
          "service": "http_status:404"
        }
      ]
    }
  }'

{
  "success": true,
  "errors": [],
  "messages": [],
  "result": {
    "tunnel_id": "",
    "version": 1,
    "config": {
      "ingress": [
        {
          "service": "http://localhost:3000",
          "hostname": "openwebui-homelab.ricoberger.dev",
          "originRequest": {}
        },
        {
          "service": "http_status:404"
        }
      ],
      "warp-routing": {
        "enabled": false
      }
    },
    "source": "cloudflare",
    "created_at": "2025-03-15T13:08:36.465147Z"
  }
}

Create a DNS record for our application. This DNS record allows Cloudflare to proxy openwebui-homelab.ricoberger.dev traffic to our Cloudflare Tunnel (.cfargotunnel.com).

export CLOUDFLARE_ZONE_ID=

curl "https://api.cloudflare.com/client/v4/zones/$CLOUDFLARE_ZONE_ID/dns_records" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  --data '{
    "type": "CNAME",
    "proxied": true,
    "name": "openwebui-homelab.ricoberger.dev",
    "content": ".cfargotunnel.com"
  }'

{
  "result": {
    "id": "",
    "name": "openwebui-homelab.ricoberger.dev",
    "type": "CNAME",
    "content": ".cfargotunnel.com",
    "proxiable": true,
    "proxied": true,
    "ttl": 1,
    "settings": {
      "flatten_cname": false
    },
    "meta": {},
    "comment": null,
    "tags": [],
    "created_on": "2025-03-15T13:17:18.56921Z",
    "modified_on": "2025-03-15T13:17:18.56921Z"
  },
  "success": true,
  "errors": [],
  "messages": []
}

Install and run the tunnel

brew install cloudflared
sudo cloudflared service install $CLOUDFLARE_TUNNEL_TOKEN

Verify tunnel status

curl "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/cfd_tunnel/$CLOUDFLARE_TUNNEL_ID" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN"

{
  "success": true,
  "errors": [],
  "messages": [],
  "result": {
    "id": "",
    "account_tag": "",
    "created_at": "2025-03-15T12:46:52.095257Z",
    "deleted_at": null,
    "name": "homelab",
    "connections": [...],
    "conns_active_at": "2025-03-15T13:20:53.912617Z",
    "conns_inactive_at": null,
    "tun_type": "cfd_tunnel",
    "metadata": {},
    "status": "healthy",
    "remote_config": true
  }
}

That's it! Now we can open openwebui-homelab.ricoberger.dev in our browser to access the Open WebUI from anywhere.

Cloudflare Documentation ↩︎

Mac mini as AI Server

Rico Berger — Fri, 14 Mar 2025 09:00:00 +0000

In today's blog post, we will explore how to run a local AI server on a Mac mini. We will use Ollama to run a large language model locally and Open WebUI as the web interface to access Ollama.

Ollama

Ollama¹ is a lightweight, extensible framework designed for running large language models locally. It enables users to easily set up and run models such as Llama 3.3, facilitating tasks like code generation and natural language processing. To install and run Ollama via Homebrew on macOS we can use the following commands:

brew install ollama
brew services start ollama

Once Ollama is running we can pull a model via the ollama pull command. In the following we are using Llama 3, but every other model should also work. A list of all available models can be found Ollama website.

ollama pull llama3

pulling manifest
pulling 6a0746a1ec1a... 100% ▕███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 4.7 GB
pulling 4fa551d4f938... 100% ▕███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  12 KB
pulling 8ab4849b038c... 100% ▕███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  254 B
pulling 577073ffcc6c... 100% ▕███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  110 B
pulling 3f8eb4da87fa... 100% ▕███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  485 B
verifying sha256 digest
writing manifest
success

When the model is pulled, we can begin chatting with it using the ollama run command:

ollama run llama3

>>> What are large language models?
Large language models (LLMs) are a type of artificial intelligence (AI) that have revolutionized the field of natural language processing (NLP). They're incredibly powerful AI systems that can process, understand, and generate human-like language at unprecedented scales.

Here's what makes them so remarkable:

1. **Scale**: LLMs are trained on massive datasets, often consisting of billions or even trillions of tokens (words, phrases, sentences) from various sources, such as books, articles, conversations, and more.
2. **Complexity**: These models use complex algorithms, such as transformer architectures, to process and analyze the vast amounts of text data. This enables them to capture subtle patterns, relationships, and nuances in language.
3. **Generative capabilities**: LLMs can generate new text based on a given prompt or context, which is known as natural language generation (NLG). They can produce coherent, context-specific responses, such as chatbot conversations, summaries, or even entire stories.

Some key features of large language models include:

1. **Attention mechanism**: This allows the model to focus on specific parts of the input text that are relevant to the current task.
2. **Self-supervised learning**: LLMs can learn from unlabeled data, which enables them to develop a deep understanding of language without explicit human feedback.
3. **Multi-turn dialogue generation**: They can engage in conversations that span multiple turns, responding to context and previous utterances.

Large language models have many applications, such as:

1. **Chatbots**: Conversational AI for customer service, entertainment, or education.
2. **Language translation**: Machine translation systems that can translate text from one language to another.
3. **Summarization**: Automatic summarization of long texts into concise summaries.
4. **Text generation**: Generating text based on a prompt or context for various purposes (e.g., creative writing, content creation).
5. **Question answering**: Answering complex questions by analyzing large amounts of text data.

Some notable examples of large language models include:

1. BERT (Bidirectional Encoder Representations from Transformers)
2. RoBERTa (Robustly Optimized BERT Pretraining Approach)
3. transformers
4. XLNet

These models have the potential to transform various industries, such as education, healthcare, marketing, and more, by enabling more accurate language understanding and generation capabilities.

Would you like to know more about a specific aspect of large language models or their applications?

Open WebUI

Until now, we have only been able to access our large language model via the command line. In the next step, we will install Open WebUI² to enable access through our browser.

Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution. To run Open WebUI via Docker the following command can be used:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Alternatively, we can start Open WebUI using the following Docker Compose file by running docker-compose up:

name: open-webui

services:
  open-webui:
    ports:
      - 3000:8080
    extra_hosts:
      - host.docker.internal:host-gateway
    volumes:
      - open-webui:/app/backend/data
    container_name: open-webui
    restart: always
    image: ghcr.io/open-webui/open-webui:main

volumes:
  open-webui:
    external: true
    name: open-webui

Now we can open our browser to access Open WebUI, in my case the Mac mini is reachable via ricos-mac-mini.local, so that the Open WebUI can be accessed via http://ricos-mac-mini.local:3000. When Open WebUI is opened the first time we have to create a admin user, afterwards we can start using our large language model through the web interface provided by Open WebUI.

That's it! We now have our local AI server running, accessible from any device on our home network. In the next blog post, we will explore how to access our local AI server from anywhere, even when we are outside our home network.

Mac mini as Home Server

Rico Berger — Sun, 09 Mar 2025 12:00:00 +0000

Until now, my home server setup was powered by a Raspberry Pi, which had its limits since I also used it for watching TV in my office. I decided to replace it with a Mac mini, and I was fortunate to find the current M4 base model at a low price. In the following post, I will discuss the basic setup to enable remote access to the Mac mini and prepare it for the various server workloads we want to run.

Why a Mac mini

I choosed the Mac mini M4 base model, which features 10 CPU and GPU cores, 16 GB of memory, and a 256 GB SSD. There are a few reasons why a Mac mini would make a good server:

Energy Efficiency - The M series chip falls between a Raspberry Pi and a 15W x86 chip.
Powerful Performance - The M series chip delivers powerful performance, easily handling various server tasks. It is also well-suited for running smaller AI tasks.
Silence - Silence is important because our home server is located in our office.
Content Caching - With multiple Macs, iPads, iPhones, and an Apple TV in the house, we have several devices that can utilize local caching.

Setup

In the following, we will examine the settings needed to reduce the power consumption of the Mac mini, enable remote access, and provide the basic tools for initial hacking. We will use the following system settings:

System Settings -> Bluetooth -> Disabled
System Settings -> Firewall -> Enabled
System Settings -> Energy -> Prevent automatic sleeping when the disply is off (Enabled) / Wake for network access (Enabled) / Start up automatically after power failure (Enabled)
System Settings -> Apple Intelligence & Siri -> Disabled
System Settings -> Spotlight -> Disable all Options
System Settings -> Notifications -> Show previews (Never) / Allow notifications when then display is sleeping (Disabled) / Allow notifications when the screen is locked (Disabled) / Allow notifications when mirroring or sharing the display (Disabled)
System Settings -> Sound -> Play sound on startup (Disabled) / Play user interface sound effects (Disabled) / Play feedback when volume is changed (Disabled)
System Settings -> Lock Screen - Start screen saver when inactive (Never) / Turn display off when inactive (10 Minutes) / Require password after screen saver begins or display is turned off (Never) / Show large clock (Never)
System Settings -> Privacy & Security -> Location Services -> Disabled
System Settings -> Login Password -> Automatically log in after a restart -> Enabled
System Settings -> Users & Groups -> Automatically log in as user
System Settings -> Game Center -> Disabled
System Settings -> General -> Sharing -> Advanced -> Remote Management (Enabled) / Remote Login (Enabled) / Remote Application Scripting (Enabled) / Local hostname (ricos-mac-mini.local)
- Remote Management: Always show remote management status in the menu bar (Enabled) / Anyone may request permission to control screen (Enabled) / Allow access for (All users) / Options (Enable all)
- Remote Login: Allow full disk access for remote users (Enabled) / Allow access for (All users)
- Remote Application Scripting: Allow access for (All users)

At this point, we can log in to our Mac mini via the Screen Sharing application and SSH. Before we continue, we will copy Ghostty's terminfo entry and our SSH key to the Mac mini:

infocmp -x | ssh ricos-mac-mini.local -- tic -x -

ssh-copy-id -i /Users/ricoberger/.ssh/id_rsa.pub ricoberger@ricos-mac-mini.local
scp /Users/ricoberger/.ssh/id_rsa ricoberger@ricos-mac-mini.local:/Users/ricoberger/.ssh/id_rsa
scp /Users/ricoberger/.ssh/id_rsa.pub ricoberger@ricos-mac-mini.local:/Users/ricoberger/.ssh/id_rsa.pub

Now we can log in to the Mac mini (ssh ricoberger@ricos-mac-mini.local) and adjust the SSH configuration to allow logins only via the copied SSH key. For this, we add the following lines to the configuration file at /etc/ssh/sshd_config:

PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no

To match the configuration on my MacBook Pro, we will install the Xcode Command Line Tools, Homebrew and our dotfiles from ricoberger/dotfiles.

xcode-select --install
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

mkdir -p /Users/ricoberger/Documents/GitHub/ricoberger
cd /Users/ricoberger/Documents/GitHub/ricoberger
git clone git@github.com:ricoberger/dotfiles.git

brew bundle install --file=Brewfile

sudo sh -c "echo $(which zsh) >> /etc/shells"
chsh -s $(which zsh)

./install.sh && source ~/.zshrc

We installed Colima as our container runtime through the Brewfile because we want to deploy most of our services using Docker. To autostart Colima and our services, use brew services start colima. We can also increase the resources of the VM created by Colima by adjusting the following values in the Colima configuration file at ~/.colima/default/colima.yaml:

# Number of CPUs to be allocated to the virtual machine.
cpu: 8

# Size of the disk in GiB to be allocated to the virtual machine.
disk: 100

# Size of the memory in GiB to be allocated to the virtual machine.
memory: 12

That's it! Now we are ready to experiment with our new Mac mini home server.

Convert HTML to PDF / PNG with Puppeteer

Rico Berger — Tue, 04 Mar 2025 20:00:00 +0000

Today, I automated one of the final annoying tasks related to my new website. I wanted to provide downloadable versions of the cheat sheets in PNG or PDF format. Until now, I had to create the PNGs manually using the FireShot Safari extension, which was quite frustrating. This process is now automated with Puppeteer. In the following sections, we will explore how Puppeteer can be used to generate PDFs and PNGs from HTML sites.

Create a New Node.js Project

Create a new folder for your project, navigate to the directory, and initialize a new Node.js project:

mkdir html-to-pdf-png
cd html-to-pdf-png
npm init

Install Puppeteer

Install Puppeteer as a dependency:

npm install puppeteer --save

This will create a node_modules directory in your project folder and add Puppeteer as a dependency to your package.json file.

Create a New File

In the same project, create an index.js file. This is where we will write our code to convert HTML into PDF and PNG.

touch index.js

Create a Browser Instance and a New Page

Inside the index.js file, we have to import puppeteer first. Afterwards we can create a new browser instance and a new page:

const puppeteer = require("puppeteer");

const browser = await puppeteer.launch({
  headless: true,
  args: ["--no-sandbox", "--disable-setuid-sandbox"],
});

const page = await browser.newPage();

The headless and args options are important for using it within a GitHub Action; otherwise, we will encounter the following error:

Error: Failed to launch the browser process!
[2178:2178:0304/190806.915940:FATAL:zygote_host_impl_linux.cc(127)] No usable sandbox! If you are running on Ubuntu 23.10+ or another Linux distro that has disabled unprivileged user namespaces with AppArmor, see https://chromium.googlesource.com/chromium/src/+/main/docs/security/apparmor-userns-restrictions.md. Otherwise see https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md for more information on developing with the (older) SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
[0304/190806.925145:ERROR:file_io_posix.cc(145)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
[0304/190806.925188:ERROR:file_io_posix.cc(145)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)


TROUBLESHOOTING: https://pptr.dev/troubleshooting

    at ChildProcess.onClose (/home/runner/work/ricoberger/ricoberger/node_modules/@puppeteer/browsers/lib/cjs/launch.js:318:24)
    at ChildProcess.emit (node:events:530:35)
    at ChildProcess._handle.onexit (node:internal/child_process:293:12)

Resize the Page and Navigate to a URL

In the next step, we will set the page size for our PDFs and PNGs, and then navigate to the URL:

await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1 });

await page.goto("https://ricoberger.de/cheat-sheets/gh/", {
  waitUntil: "networkidle0",
});

The value of waitUntil determines whether the navigation is considered successful. The default value is load, which means navigation is deemed complete when the load event fires. However, we want to wait until there are no more than 0 network connections for at least 500ms by using the value networkidle0.

Configure the Output

By default, page.pdf() generates a PDF of the page using print CSS media. To create a PDF that resembles what we see on the screen, we will use the screen media. Add page.emulateMediaType('screen') before downloading the PDF:

await page.evaluate((sel) => {
  var elements = document.querySelectorAll(sel);
  for (var i = 0; i < elements.length; i++) {
    elements[i].parentNode.removeChild(elements[i]);
  }
}, "#header");

await page.emulateMediaType("screen");

const pageHeight = await page.evaluate(
  () => document.documentElement.offsetHeight,
);

We remove the page header with the ID #header because it is unnecessary in the downloadable version of the cheat sheet. Additionally, we need the page height to use it as the height of the generated PDF page.

Download the PDF

Next, call page.pdf() to download the PDF with the following options passed to the method:

await page.pdf({
  path: "cheat-sheet.pdf",
  printBackground: true,
  width: "1920px",
  height: pageHeight + "px",
});

path: This is the file path where the PDF will be saved, and it is mandatory. If we do not specify the path, the file will not be saved to the disk, and we will receive a buffer instead.
printBackground: This parameter controls whether the background graphics of the web page are printed. The default value is false. You may want to set this to true, as some images will be missing in the PDF if it remains false.
width: Sets the width of paper. You can pass in a number or a string with a unit.
height: Sets the height of paper. You can pass in a number or a string with a unit.

Download the PNG

Next we call page.screenshot() to download the PNG with the following options passed to the method:

await page.screenshot({
  path: "cheat-sheet.png",
  fullPage: true,
  type: "png",
});

path: The file path to save the image to. The screenshot type will be inferred from file extension. If path is a relative path, then it is resolved relative to current working directory. If no path is provided, the image won't be saved to the disk.
fullPage: When true, takes a screenshot of the full page.

Close the Browser

Finally, close the browser instance after downloading the PDF and PNG:

await browser.close();

Our final code appears as follows:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ["--no-sandbox", "--disable-setuid-sandbox"],
  });

  const page = await browser.newPage();

  await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1 });

  await page.goto("https://ricoberger.de/cheat-sheets/gh/", {
    waitUntil: "networkidle0",
  });

  await page.evaluate((sel) => {
    var elements = document.querySelectorAll(sel);
    for (var i = 0; i < elements.length; i++) {
      elements[i].parentNode.removeChild(elements[i]);
    }
  }, "#header");

  await page.emulateMediaType("screen");

  const pageHeight = await page.evaluate(
    () => document.documentElement.offsetHeight,
  );

  await page.pdf({
    path: "cheat-sheet.pdf",
    printBackground: true,
    width: "1920px",
    height: pageHeight + "px",
  });

  await page.screenshot({
    path: "cheat-sheet.png",
    fullPage: true,
    type: "png",
  });

  await browser.close();
})();

The adjusted code for our cheat sheets is available in the build-cheat-sheets-assets.js file. You can find the usage within the GitHub Action in the deploy.yml file.

Neovim: Custom snacks.nvim Picker

Rico Berger — Mon, 03 Mar 2025 19:00:00 +0000

Welcome to another blog post about my dotfiles. Today, I will briefly discuss how to implement a custom picker using snacks.nvim. Inspired by a Reddit post, I wanted to create my own command palette, similar to the one in Visual Studio Code. Below, I will share the result.

One feature I have always liked about Visual Studio Code is the command palette, which allows you to quickly change a file's type, run code formatting, and more. Inspired by a Reddit post, I wanted to create my own command palette in Neovim, which allows me to run commands without creating a keymap, as I don't use them frequently enough.

The commands for the command palette are defined in a table, where each entry has a name and an action. The action can be a command if it starts with :, a keymap if it's a string not starting with : or a function if it's a function, similar to the actions in the snacks.nvim dashboard.

The commands are passed as items to the Snacks.picker. When a command is selected, the confirm function is executed, which runs the corresponding action in a similar way to how it is done in the snacks.nvim dashboard.

The command palette is exported as a module and registered in the keys section of the snacks.nvim plugin, so that the command palette can be opened via leader + p.

return {
  {
    "folke/snacks.nvim",
    priority = 1000,
    lazy = false,
    keys = {
      {
        "p",
        function()
          require("command-palette").show_commands()
        end,
        desc = "Command Palette",
      },
    },
  },
}

local M = {}

M.commands = {
  {
    name = "Copilot: Actions",
    action = "ca",
  },
  {
    name = "Dim: Toggle",
    action = function()
      local snacks_dim = require("snacks").dim
      if snacks_dim.enabled then
        snacks_dim.disable()
      else
        snacks_dim.enable()
      end
    end,
  },
  {
    name = "Tab: Close",
    action = ":tabclose",
  },
  {
    name = "Tab: New",
    action = ":tabnew",
  },
  {
    name = "Todo Comments: Quickfix List",
    action = ":TodoQuickFix",
  },
  {
    name = "Todo Comments: Location List",
    action = ":TodoLocList",
  },
}

function M.show_commands()
  local items = {}

  for idx, command in ipairs(M.commands) do
    local item = {
      idx = idx,
      name = command.name,
      text = command.name,
      action = command.action,
    }
    table.insert(items, item)
  end

  Snacks.picker({
    title = "Command Palette",
    layout = {
      preset = "default",
      preview = false,
    },
    items = items,
    format = function(item, _)
      return {
        { item.text, item.text_hl },
      }
    end,
    confirm = function(picker, item)
      if type(item.action) == "string" then
        if item.action:find("^:") then
          picker:close()
          return picker:norm(function()
            picker:close()
            vim.cmd(item.action:sub(2))
          end)
        else
          return picker:norm(function()
            picker:close()
            local keys = vim.api.nvim_replace_termcodes(item.action, true, true, true)
            vim.api.nvim_input(keys)
          end)
        end
      end

      return picker:norm(function()
        picker:close()
        item.action()
      end)
    end,
  })
end

return M

Neovim: Extend snacks.nvim Explorer

Rico Berger — Sun, 02 Mar 2025 19:00:00 +0000

In today's blog post, I want to take a quick look at the snacks.nvim explorer and how I extended it with some useful actions, so that I can search within a directory, diff selected files, and provide multiple copy options.

The first action copy_file_path is used to provide multiple copy options for the file under the cursor. The function returns a selection menu from which I can select whether I want to copy the basename, extension, filename, path, the relative path from the working directory or home directory, or the file URI.

The second action search_in_directory is used to search within the directory under the cursor. It opens a new snacks.nvim picker where the working directory is set to the selected directory from the explorer and allows me to search for all files with a specific search term.

The third action diff is used to compare two selected files. In the explorer, two files can be selected with Tab or Shift + Tab, and afterwards, a diff between these two files is opened in a new tab.

The code for the mentioned actions can be found in the following code snippet. If you want to know more about my Neovim configuration you can have a look at my dotfiles repository or my last blog post. If you have some useful addtions to the snacks.nvim explorer please let me know.

return {
  {
    "folke/snacks.nvim",
    priority = 1000,
    lazy = false,
    opts = {
      picker = {
        enabled = true,
        sources = {
          explorer = {
            auto_close = true,
            hidden = true,
            layout = {
              preset = "default",
              preview = false,
            },
            actions = {
              copy_file_path = {
                action = function(_, item)
                  if not item then
                    return
                  end

                  local vals = {
                    ["BASENAME"] = vim.fn.fnamemodify(item.file, ":t:r"),
                    ["EXTENSION"] = vim.fn.fnamemodify(item.file, ":t:e"),
                    ["FILENAME"] = vim.fn.fnamemodify(item.file, ":t"),
                    ["PATH"] = item.file,
                    ["PATH (CWD)"] = vim.fn.fnamemodify(item.file, ":."),
                    ["PATH (HOME)"] = vim.fn.fnamemodify(item.file, ":~"),
                    ["URI"] = vim.uri_from_fname(item.file),
                  }

                  local options = vim.tbl_filter(function(val)
                    return vals[val] ~= ""
                  end, vim.tbl_keys(vals))
                  if vim.tbl_isempty(options) then
                    vim.notify("No values to copy", vim.log.levels.WARN)
                    return
                  end
                  table.sort(options)
                  vim.ui.select(options, {
                    prompt = "Choose to copy to clipboard:",
                    format_item = function(list_item)
                      return ("%s: %s"):format(list_item, vals[list_item])
                    end,
                  }, function(choice)
                    local result = vals[choice]
                    if result then
                      vim.fn.setreg("+", result)
                      Snacks.notify.info("Yanked `" .. result .. "`")
                    end
                  end)
                end,
              },
              search_in_directory = {
                action = function(_, item)
                  if not item then
                    return
                  end
                  local dir = vim.fn.fnamemodify(item.file, ":p:h")
                  Snacks.picker.grep({
                    cwd = dir,
                    cmd = "rg",
                    args = {
                      "-g",
                      "!.git",
                      "-g",
                      "!node_modules",
                      "-g",
                      "!dist",
                      "-g",
                      "!build",
                      "-g",
                      "!coverage",
                      "-g",
                      "!.DS_Store",
                      "-g",
                      "!.docusaurus",
                      "-g",
                      "!.dart_tool",
                    },
                    show_empty = true,
                    hidden = true,
                    ignored = true,
                    follow = false,
                    supports_live = true,
                  })
                end,
              },
              diff = {
                action = function(picker)
                  picker:close()
                  local sel = picker:selected()
                  if #sel > 0 and sel then
                    Snacks.notify.info(sel[1].file)
                    vim.cmd("tabnew " .. sel[1].file)
                    vim.cmd("vert diffs " .. sel[2].file)
                    Snacks.notify.info("Diffing " .. sel[1].file .. " against " .. sel[2].file)
                    return
                  end

                  Snacks.notify.info("Select two entries for the diff")
                end,
              },
            },
            win = {
              list = {
                keys = {
                  ["y"] = "copy_file_path",
                  ["s"] = "search_in_directory",
                  ["D"] = "diff",
                },
              },
            },
          },
        },
      },
    },
  },
}

My Dotfiles

Rico Berger — Sat, 01 Mar 2025 19:00:00 +0000

In this blog post, I'll take you through my complete dotfiles, detailing the tools and configurations that empower my development process. From my terminal choice to the editor I rely on, and even the GitHub CLI commands that simplify my workflow, I'll share insights and tips that might just inspire you to optimize your own environment. Whether you're a seasoned developer or just starting, there's something here for everyone looking to enhance their macOS experience. Let's dive in!

OS Setup

I'm using macOS as my daily development environment, and together with Raycast, I couldn't be happier with it. Raycast allows me to quickly create notes and reminders, view my calendar entries, GitHub pull requests and issues, and Jira tickets. I'm also using it as a replacement for the built-in Spotlight search and for managing spaces¹ and windows.

Terminal

By the end of last year, I switched from Alacritty to Ghostty as my go-to terminal emulator (yes, the hype caught me 😅). Ghostty provides most of the features out of the box that I used tmux for in the past, such as multiple windows, tabs, and panes. It is super fast, and the available configuration options are on point (not too much, not too little). The only thing I would wish for is an API so that windows, tabs, and panes can be created programmatically, which is currently only possible via AppleScript. In my Ghostty configuration² I set my prefered color scheme, font and some key bindings.

As my shell³, I'm using Zsh with Zinit to manage the following plugins:

zsh-users/zsh-completions: Additional completion definitions for Zsh
zsh-users/zsh-autosuggestions: Fish-like fast/unobtrusive autosuggestions for Zsh
Aloxaf/fzf-tab: Replace Zsh's default completion selection menu with fzf

For the customization of my prompt⁴, I'm using Starship to show the current OS and user, the directory I'm working in, the Git branch and status, the exit status of the last command, the Kubernetes context and namespace, and the current time.

Other important tools I'm using are:

tmux⁵: tmux is a terminal multiplexer, and while I replaced most of its functionality with Ghostty, I'm still using it on servers where the benefit of persistent sessions is unmatched.
fzf: fzf is a command-line fuzzy finder.
ripgrep: ripgrep is a line-oriented search tool that recursively searches the current directory for a regex pattern.
Yazi⁶: Yazi is a blazing fast terminal file manager written in Rust, based on async I/O. It is fairly new in my workflow, but I want to try it out more to stay within the terminal when doing file oeprations instead of running open . and doing these operations via the Finder.

Editor

Neovim has become my daily editor of choice due to its powerful features and flexibility. As an extensible text editor based on Vim, it allows me to tailor it to my specific workflow, whether I'm coding, writing, or debugging issues. Spending most of my time within the terminal, it is also faster and feels more natural to stay within the terminal when editing files instead of opening Visual Studio Code.

Currently, I'm using the following plugins for tasks such as fuzzy searching files, Git integrations, code completion, formatting, linting, and some AI features:

lazy.nvim: A modern plugin manager for Neovim
catppuccin: Soothing pastel theme for Neovim
lualine.nvim: A blazing fast and easy to configure Neovim statusline plugin written in pure lua
snacks.nvim: A collection of QoL plugins for Neovim
gitsigns.nvim: Git integration for buffers
diffview.nvim: Single tabpage interface for easily cycling through diffs for all modified files for any git rev
nvim-treesitter: Nvim Treesitter configurations and abstraction layer
nvim-treesitter-context: Show code context
nvim-treesitter-textobjects: Syntax aware text-objects, select, move, swap, and peek support
nvim-lspconfig: Quickstart configs for Nvim LSP
vim-helm: Vim syntax for helm templates (yaml + gotmpl + sprig + custom)
conform.nvim: Lightweight yet powerful formatter plugin for Neovim
nvim-lint: An asynchronous linter plugin for Neovim complementary to the built-in Language Server Protocol support
blink.cmp: Performant, batteries-included completion plugin for Neovim
blink-copilot: Configurable GitHub Copilot blink.cmp source for Neovim
friendly-snippets: Set of preconfigured snippets for different languages
multicursor.nvim: Multiple cursors in Neovim
todo-comments.nvim: Highlight, list and search todo comments in your projects.
copilot.lua: Fully featured & enhanced replacement for copilot.vim complete with API for interacting with Github Copilot
CopilotChat.nvim: Chat with GitHub Copilot in Neovim

While all the plugins mentioned above are awesome and fulfill a specific need, I want to give a special shoutout to multicursor.nvim, which was the last missing plugin for me to fully abandon Visual Studio Code.

GitHub CLI

Last but not least, I also want to take a look at my workflow with GitHub because it is an important part of my daily work, and I'm a bit proud of it. I'm a heavy user of the GitHub CLI (gh) and the gh-dash extension, because it allows me to stay for most of my work within the terminal.

I use the GitHub CLI to create pull requests via the gh pr create command, to add / remove labels, to view the workflow runs for a pull request and to merge them. All with the help of some nice helper functions which can be found in the .bin directory in my dotfiles.

# Add a label to the pull request, by selecting the label via fzf from a list of
# all available labels in the repository
gh pr edit $1 --add-label "$(gh label list --json name --jq ".[].name" | fzf)"

# Delete a label from the pull request, by selecting the label via fzf from a
# a list of all labels from the pull request
gh pr edit $1 --remove-label "$(gh pr view $1 --json labels --jq ".labels.[].name" | fzf)"

# View the logs of a workflow run, by selecting the workflow run via fzf
branch=$(git rev-parse --abbrev-ref HEAD)
workflow=$(gh run list --branch $branch --json databaseId,workflowName,createdAt,status --template '{{range .}}{{printf "%.0f" .databaseId}}{{"\t"}}{{.status}}{{"\t"}}{{.createdAt}}{{"\t"}}{{.workflowName}}{{"\n"}}{{end}}' | fzf)
gh run view "$(echo $workflow | awk '{print $1}')" --log

# Squash the commits into one commit and merge it into the base branch, also
# delete the local and remote branch after merge
gh pr merge $1 --squash --delete-branch --admin

These commands are also integrated into my gh-dash configuration⁷. gh-dash is an extension for the GitHub CLI to display a dashboard with pull requests and issues. It integrates beautifully with tmux and Neovim:

Open the diff of a pull request in Neovim diffview.nvim via cd {{.RepoPath}} && gh pr checkout {{.PrNumber}} && nvim -c ":DiffviewOpen origin/HEAD...HEAD --imply-local" or with tmux via tmux new-window -n "{{.RepoName}}/{{.PrNumber}}" -c {{.RepoPath}} 'gh pr checkout {{.PrNumber}} && nvim -c ":DiffviewOpen origin/HEAD...HEAD --imply-local"'
Open a pull request in Neovim octo.nvim via cd {{.RepoPath}} && nvim -c ":Octo pr edit {{.PrNumber}}" or with tmux via tmux new-window -n "{{.RepoName}}/{{.PrNumber}}" -c {{.RepoPath}} 'nvim -c ":Octo pr edit {{.PrNumber}}"'
Open an issue in Neovim octo.nvim via cd {{.RepoPath}} && nvim -c ":Octo issue edit {{.IssueNumber}}" or with tmux via tmux new-window -n "{{.RepoName}}/{{.IssueNumber}}" -c {{.RepoPath}} 'nvim -c ":Octo issue edit {{.IssueNumber}}"'

Welcome to My New Website

Rico Berger — Sun, 23 Feb 2025 15:00:00 +0000

I spent the past few days creating a new website for my domain ricoberger.de. Previously, I only used the domain as a landing page with links to my social media profiles. This time, I wanted to add my cheat sheets, which were previously hosted in my ricoberger/cheat-sheets GitHub repository. I also aimed to include a small blog where I can write about topics I'm interested in. In the following post, we will explore the technologies used to create the website and the features it offers.

To include my cheat sheets on the website, I decided to create my own site generator in Go instead of using an existing static site generator like Hugo. The site generator is located in the main.go file and utilizes the html/template package to generate HTML files for the website based on various templates available in the templates directory.

Every site uses the base.html template, which provides the basic HTML layout structure, including the tag and site navigation. We then select a specific template for each site to generate the final HTML layout using the buildTemplate function. We also provide a destination path to the function, indicating where the site will be available and where the index.html file will be created. Finally, we can pass a Data struct to the template, which includes the Metadata for each site and custom data specific to each site.



  
    {{ .Metadata.Title }}
  

  
    

    {{ template "content" . }}

{{ define "content" }}

{{ end }}

type Data struct {
	Metadata Metadata
	Content  any
}

type Metadata struct {
	Title       string
	Description string
	Author      string
	Keywords    []string
	BaseUrl     string
	Url         string
	Image       string
	Prism       bool
}

func buildTemplate(tmpl string, distPath string, data Data) error {
	if err := os.MkdirAll(distPath, os.ModePerm); err != nil {
		return err
	}

	templates, err := template.New("base.html").Funcs(template.FuncMap{
		"formatMarkdown": func(s string) template.HTML {
			md := goldmark.New(
				goldmark.WithExtensions(
					extension.Table,
					extension.Strikethrough,
				),
				goldmark.WithRendererOptions(
					html.WithUnsafe(),
				),
			)

			var buf bytes.Buffer
			if err := md.Convert([]byte(s), &buf); err != nil {
				slog.Error("Failed to convert markdown", slog.Any("error", err))
			}
			return template.HTML(buf.String())
		},
	}).ParseFiles("templates/base.html", fmt.Sprintf("templates/%s.html", tmpl))
	if err != nil {
		return err
	}

	f, err := os.Create(fmt.Sprintf("%s/index.html", distPath))
	if err != nil {
		return err
	}

	if err := templates.Execute(f, data); err != nil {
		return err
	}

	return nil
}

For the website's styling, we use Tailwind CSS. All our styles are defined in the input.css file, which is used to generate the final CSS file (output.css) using @tailwindcss/cli.

In the input.css file, we specify the location of the source files so that Tailwind can detect all the used classes. We also define some theme variables and the styling for each HTML tag used.

@import "tailwindcss" source(none);

@source "../../**/*.html";

/* The used colors are based on the awesome Catppuccin theme: https://catppuccin.com/ */
@theme {
  --color-base: #24273a;
  --color-mantle: #1e2030;
  --color-crust: #181926;
  --color-surface: #5b6078;
  --color-text: #cad3f5;
  --color-primary: #8aadf4;
  --color-red: #ed8796;
  --color-yellow: #eed49f;
  --color-green: #a6da95;
  --color-blue: #8aadf4;
}

@layer base {
  body {
    @apply bg-base text-text;
  }

  /* ... */
}

Last but not least, we are using Alpine.js and Tailwind CSS to create a user-friendly dropdown menu for small screens.

Cheat Sheets

As mentioned at the beginning of the post, an important aspect for me was the ability to include my cheat sheets on the websites. The cheat sheets are written as YAML files and have the following structure:

---
# The title, description, author and keywords for the cheat sheet
title: Vim
description: Vim Cheat Sheet
author: Rico Berger
keywords:
  - Vim
  - Neovim
# Each cheat sheet can have multiple pages with a title and a defined number of
# columns
pages:
  - title: Vim
    columns: 5
    # Each page of a cheat sheet can have multiple sections with a title, which
    # are rendered dynamically, within the defined number of columns
    sections:
      - title: Registers
        # Each section can have multiple items, which are written in Markdown
        items:
          - "`:register` - Show registers content"
          - ...
        # Each section can also have a tip, which is rendered as a box below the
        # defined items. Besides the actual description, each tip can also have
        # a list of items
        tip:
          description: |
            "**Tip:** Registers are being stored in ~/.viminfo, and will be
            loaded again on next restart of vim. Special registers:"
          items:
            - "`0` - Last yank"
            - ...

The YAML files are decoded using the github.com/goccy/go-yaml package. Then we are using the cheat-sheet.html template to render the cheat sheet via the buildTemplate function. The decoded cheat sheet is passed to the function within the Data struct. The result for the rendered cheat sheet then looks as follows (Vim):

Blog Posts

Blog posts are written as markdown files and rendered via the blog-post.html template. The markdown files are parsed and rendered to HTML via the github.com/yuin/goldmark package. Each markdown file also contains a metadata section with the following information:

---
Title: Welcome to My New Website
Description: |
  I spent the past few days creating a new website for my domain ricoberger.de.
  Previously, I only used the domain as a landing page with links to my social
  media profiles. This time, I wanted to add my cheat sheets, which were
  previously hosted in my ricoberger/cheat-sheets GitHub repository. I also
  aimed to include a small blog where I can write about topics I'm interested
  in. In the following post, we will explore the technologies used to create the
  website and the features it offers.
AuthorName: Rico Berger
AuthorTitle: Site Reliability Engineer
AuthorImage: /assets/img/authors/ricoberger.webp
PublishedAt: 2025-02-23 15:00:00
Tags:
  - alpinejs
  - blog
  - cheat-sheets
  - go
  - projects
  - tailwindcss
Image: /blog/posts/welcome-to-my-wew-website/assets/landing-page.png
---

The document metadata is parsed using the github.com/yuin/goldmark-meta extension for goldmark and is used to render the header of each blog post and the meta tags in the HTML file. We include the metadata for the Open Graph protocol and X Cards in every blog post, to make them look great when they are shared:

Since I'm a big fan of RSS feeds (you might want to have a look at FeedDeck 😉), we also include an RSS feed for the blog and an RSS feed for each tag, which can be specified for a blog post. The RSS feed is generated using the buildRssFeed function. While generating the feed and creating the feed.xml file, we go through the parsed HTML of each post to replace all relative links with absolute ones via the github.com/PuerkitoBio/goquery package.

Hosting via GitHub Pages

To host the new website, we are using GitHub Pages like for the old website. GitHub Pages is perfect for hosting static sites and integrates very well with GitHub Actions. Within the deploy.yaml GitHub Action, we are building and deploying our website:

---
name: Deploy

on:
  push:
    branches:
      - main

jobs:
  build-website:
    name: Build Website
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pages: write
      id-token: write

    steps:
      # Checkout the repository, setup Go and Node.js, install the Go and
      # Node.js dependencies and build the "generator" binary
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
          cache: true
          cache-dependency-path: go.sum

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: npm
          cache-dependency-path: package-lock.json

      - name: Setup Pages
        uses: actions/configure-pages@v5

      - name: Install Dependencies / Build Binary
        run: |
          go mod download
          go build -o generator .
          npm install

      # Run the generator to create the static files for our website in the
      # "dist" directory
      #
      # To be able to use a custom domain for our GitHub page we also create a
      # file named "CNAME" within the "dist" directory, which contains our
      # custom domain (see https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site#configuring-a-subdomain)
      - name: Generate Website
        run: |
          ./generator
          npm run build
          echo "ricoberger.de" > ./dist/CNAME

      # In the last build step we upload the "dist" directory as artifact, so
      # that it can be deployed to a GitHub Page
      - name: Upload Artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: ./dist

  deploy-website:
    name: Deploy Website
    runs-on: ubuntu-latest
    needs: build-website
    permissions:
      contents: read
      pages: write
      id-token: write
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}

    # Deploy the uploaded artifact from the build job to GitHub Pages
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v4

Since we are using GitHub Pages for hosting our site, we can also create a custom 404 page, by placing a file named 404.html (generated via the 404.html template) in the root directory of our website. This site will display a custom 404 error page when people try to access nonexistent pages on our site.

Final Words

As we conclude this blog post, I hope you gained a better understanding of the internals behind ricoberger.de and perhaps learned something new. I aim to enhance my writing skills in future posts. If you don't want to miss them, feel free to subscribe to the RSS feed or follow me on social media:

If you have any suggestions for future cheat sheets, blog posts, or if you have some nice articles about the topics of Site Reliability Engineering, Platform Engineering, Cloud Native, or Kubernetes, feel free to contact me via social media or create an issue in my GitHub repository.