Home Run self-hosted CI/CD agents on Azure Kubernetes Service - Part 4 - Azure DevOps
Post
Cancel

Run self-hosted CI/CD agents on Azure Kubernetes Service - Part 4 - Azure DevOps

Posted Feb 11, 2023 Updated Feb 25, 2023
By Adam Kielar
7 min read

Are you looking for a way to run your Azure DevOps builds and deployments on your own infrastructure? If so, you’ll want to check out Azure DevOps self-hosted runners. Self-hosted runners are agents that allow you to run your build and deployment jobs on machines that you control, giving you more flexibility and control over your environment. In this blog post, we’ll take a closer look at what Azure DevOps self-hosted runners are, why you might want to use them, and how to set them up for your projects. Whether you’re looking to save costs, ensure security and isolation, or build on specialized hardware or software configurations, self-hosted runners can help you improve your Azure DevOps workflow.

This post is a continuation of our journey with self-hosted CI/CD agents. I encourage you to check part 1, part 2 and part 3 if you want to see a different approach to that topic.

Overview

We will create and configure the following resources:

  • AKS cluster with workload identity and kubelet identity.
  • Azure Container Registry.
  • Role assignment for kubelet identity and workload identity.
  • Kubernetes deployment object for self-hosted runner.
  • Azure Devops pipeline.
Here is a link to GitHub repo with all files for reference.

Video walkthrough

Create AKS cluster and ACR

  1. Run script.
    1
2
3

    # After running above script, if there were no errors, variables should be available in terminal.
chmod +x aks.sh
./aks.sh 'add user id, for me, it is my email of AAD user'
  2. Get credentials to AKS, oidcUrl and test connection.
    1
2
3
4
5

    az aks get-credentials --resource-group $resourceGroup --name $aksName
export oidcUrl="$(az aks show --name $aksName \
--resource-group $resourceGroup \
--query "oidcIssuerProfile.issuerUrl" -o tsv)"
kubectl get nodes

Set up workload identity

  1. Create workload identity.
    1
2
3
4
5

    workloadIdentity="workload-identity"
workloadClientId=$(az identity create --name $workloadIdentity \
--resource-group $resourceGroup --query clientId -o tsv)
workloadPrincipalId=$(az identity show --name $workloadIdentity \
--resource-group $resourceGroup --query principalId -o tsv)
  2. Create Kubernetes Service Account in devops namespace.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

    # devops-runner-sa.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: devops
  labels:
 name: devops
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
 azure.workload.identity/client-id: <insert workloadClientId> # echo $workloadClientId
  labels:
 azure.workload.identity/use: "true"
  name: workload-sa
  namespace: devops

    1

    kubectl apply -f devops-runner-sa.yaml
  3. Create ClusterRole and ClusterRoleBinding for Service Account. Update workloadPrincipalId in file.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

    # devops-roles.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
  name: devops
rules:
- apiGroups:
  - '*'
  resources:
  - statefulsets
  - services
  - replicationcontrollers
  - replicasets
  - podtemplates
  - podsecuritypolicies
  - pods
  - pods/log
  - pods/exec
  - podpreset
  - poddisruptionbudget
  - persistentvolumes
  - persistentvolumeclaims
  - endpoints
  - deployments
  - deployments/scale
  - daemonsets
  - configmaps
  - events
  - secrets
  verbs:
  - create
  - get
  - watch
  - delete
  - list
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
 rbac.authorization.kubernetes.io/autoupdate: "true"
  name: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: devops
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:devops
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: # echo $workloadPrincipalId

    1

    kubectl apply -f devops-roles.yaml
  4. Create federated identity credentials.
    1
2
3
4
5
6

    az identity federated-credential create \
--name "aks-federated-credential" \
--identity-name $workloadIdentity \
--resource-group $resourceGroup \
--issuer "${oidcUrl}" \
--subject "system:serviceaccount:devops:workload-sa"
  5. Create custom role for workload identity. Create acrbuild.json file with following definition. Replace {YOUR SUBSCRIPTION} with your subscription id.
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

    {
  "Name": "AcrBuild",
  "IsCustom": true,
  "Description": "Can read, push, pull and list builds.",
  "Actions": [
 "Microsoft.ContainerRegistry/registries/read",
 "Microsoft.ContainerRegistry/registries/pull/read",
 "Microsoft.ContainerRegistry/registries/push/write",
 "Microsoft.ContainerRegistry/registries/scheduleRun/action",
 "Microsoft.ContainerRegistry/registries/runs/*",
 "Microsoft.ContainerRegistry/registries/listBuildSourceUploadUrl/action"
  ],
  "AssignableScopes": [
 "/subscriptions/{YOUR SUBSCRIPTION}"
  ]
}

    1

    az role definition create --role-definition acrbuild.json
  6. Assign AcrBuild role to workload identity.
    1
2

    az role assignment create --assignee $workloadClientId \
--role 'AcrBuild' --scope $acrId
  7. Assign Azure Kubernetes Service Cluster User Role and Azure Kubernetes Service RBAC Writer to workload identity.
    1
2
3
4

    az role assignment create \
--role "Azure Kubernetes Service Cluster User Role" \
--assignee $workloadPrincipalId \
--scope $aksId

    1
2
3
4

    az role assignment create \
--role "Azure Kubernetes Service RBAC Writer" \
--assignee $workloadPrincipalId \
--scope "$aksId/namespaces/devops"

Install Azure Devops self-hosted agent

Personal access token (PAT) and Agent Pool

As a first step, we must register the runner. We need to have permissions to administer the agent queue if we want to complete that step.

  1. Sign in to Azure DevOps organization: https://dev.azure.com/{your_organization}.
  2. Create personal access token. Azure DevOps user settings Create PAT token
  3. Create an Agent pool Azure DevOps Organization settings

Create an image for Azure DevOps runner

We will build custom image useing new version of the runner, pre-release v3.217.1 . We will also add az cli, kubectl.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

FROM ubuntu:22.04
USER root
RUN apt-get -y update && apt-get install -y curl && \
    curl -sL https://aka.ms/InstallAzureCLIDeb | bash && az aks install-cli && \
    curl -fsSL https://get.docker.com -o get-docker.sh && sh ./get-docker.sh && \
    mkdir devops-runner && cd devops-runner && \
    curl -o vsts-agent-linux-x64-3.217.1.tar.gz -L https://vstsagentpackage.azureedge.net/agent/3.217.1/vsts-agent-linux-x64-3.217.1.tar.gz && \
    tar xzf ./vsts-agent-linux-x64-3.217.1.tar.gz && \
    apt-get clean

RUN addgroup --gid 110 devops && adduser devops --uid 111 --system && adduser devops devops && \
  chown -R devops:devops devops-runner

USER devops

1

az acr build -f Dockerfile.runner -t devops-runner:v1.0.0 -r $acrName -g $resourceGroup .

Create Storage Class and Persistent Volume Claim

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# devops-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: devops-azurefile
provisioner: file.csi.azure.com
mountOptions:
  - uid=110
  - gid=111
allowVolumeExpansion: true
volumeBindingMode: Immediate
reclaimPolicy: Delete
parameters:
  skuName: Standard_LRS

1
2
3
4
5
6
7
8
9
10
11
12
13

# devops-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: devops-pvc
  namespace: devops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
  storageClassName: devops-azurefile

1
2

kubectl apply -f devops-storageclass.yaml
kubectl apply -f devops-pvc.yaml

Create Deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# devops-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: devops-deployment
  namespace: devops
  labels:
    app: devops-runner
    azure.workload.identity/use: "true"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devops-runner
      azure.workload.identity/use: "true"
  template:
    metadata:
      labels:
        app: devops-runner
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: workload-sa
      containers:
      - name: devops-runner
        image: devopsacr14044.azurecr.io/devops-runner:v1.0.0 # Add your ACR respository
        command:
        - sleep
        args:
        - 99d
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        volumeMounts:
          - mountPath: "/home/devops"
            name: runner-data
      volumes:
        - name: runner-data
          persistentVolumeClaim:
            claimName: devops-pvc

1

kubectl apply -f devops-deployment.yaml

Set up runner inside pod

  1. Check if pod is running and connect to container.
1
2

kubectl -n devops get pods
kubectl -n devops exec -it <POD NAME> -- sh
  1. Configure runner
    1

    ./devops-runner/config.sh

    Configure Azure Devops runner

We need to provide additional information here.

  • Azure DevOps URL eg., https://dev.azure.com/org-name
  • PAT token
  • Name of the agent pool registered in Azure DevOps Services. Defaults to ‘Default’ pool
    1. Start runner
      1

      ./devops-runner/run.sh

      We can always automate this process later. Connect runner to Azure Devops

Create Azure Devops pipeline

  1. In .ci folder create azure-pipelines.yml.

Update name of ACR registry and optionally AKS name and resource group.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

name: $(Date:yyyyMMdd)$(Rev:.r)

pool: DevOpsOnAks

trigger:
  batch: true

stages:
  - stage: buildPushDeploy
    displayName: Build, Push and Deploy
    jobs:
      - job: build
        steps:
          - checkout: self
            clean: true

          - script: |
              set -euo pipefail
              az login --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --federated-token $(cat $AZURE_FEDERATED_TOKEN_FILE)
              az acr build -f $(Agent.BuildDirectory)/s/Dockerfile -t devopsacr14044.azurecr.io/devops-demo:$(Build.SourceVersion) -r devopsacr14044 .
            displayName: Log in and push image to ACR

          - script: |
              set -euo pipefail
              az aks get-credentials --resource-group devops-runner-rg --name aks-devops-runner --overwrite-existing
              kubelogin convert-kubeconfig -l workloadidentity
              sed 's|IMAGE|devopsacr14044.azurecr.io/devops-demo|g; s/TAG/$(Build.SourceVersion)/g' $(Agent.BuildDirectory)/s/app.yaml | kubectl apply -f -
            displayName: Get AKS credentials and deploy app
  1. Push changes to GitHub and check status of each resource in Azure Devops.

Job output

Azure Devops output 1

Azure Devops output 2

Kubectl output

We successfully managed to deploy our application.

azure aks azure devops kubernetes
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents