Passing variables between stages in Azure DevOps pipelines

The other day I was working on an Infrastructure as Code project that involved deploying an Azure Container registry. That ACR is typically a resource that you deploy just ones in your production environment and not one per environment. You do that because you want container images to be used as immutable artifacts that are progressively deployed across all your environments.

The problem you then get is that you do need to, for example, assign permissions on that ACR for the AKS user in your dev and test environment (your kubelet identity needs the ‘ACR Pull’ permissions). When you deploy your dev or test resources, you usually cannot do that. That connection you use there should not have permission to assign that role to the production environment.

Multi-stage pipeline

To come around that, we can pass the AKS identity to the production stage and do the role assignment there. The Bicep sources and pipeline can be found here. Let’s go over some details.

First, the main Bicep file will create a resource group, the identity used for the nodes in AKS, and the registry:

targetScope = 'subscription'

@allowed([
  'dev'
  'test'
  'prod'
])
param environmentShort string
param location string = 'westeurope'
param locationShort string = 'we'

resource rgshared 'Microsoft.Resources/[email protected]' = {
  name: 'rg-shared-${environmentShort}-${locationShort}-001'
  location: location
}

var kubeletIdentityName = 'id-demo-kubelet-${environmentShort}-${locationShort}-001'

module kubeletIdentity 'userAssignedIdentities.bicep' = {
  scope: rgshared
  name: 'kubeletIdentity'
  params: {
    identityName: kubeletIdentityName
    location: location
  }
}

var acrName = 'acrdemo${environmentShort}${locationShort}001'

module acr 'registries.bicep' = if (environmentShort == 'prod') {
  scope: rgshared
  name: 'acr'
  params: {
    location: location
    name: acrName
  }
}

output kubeletIdentityPrincipalId string = kubeletIdentity.outputs.principalId
output acrId string = environmentShort == 'prod' ? acr.outputs.id : ''

Notice how the kubelets principalId and the ID of the ACR are returned to the Azure DevOps pipeline. The ID of the ACR will only be there on the production deployment.

The pipeline then contains a task that converts that output into a pipeline variable:

- task: [email protected]
  displayName: 'Set outputs'
  name: SetOutputs
  inputs:
    targetType: inline
    script: |
      Write-Host $env:MAINDEPLOYMENTS_OUTPUTS
      $armOutputObj = $env:MAINDEPLOYMENTS_OUTPUTS | convertfrom-json
      Write-Host $armOutputObj

      $kubeletPrincipalId = $armOutputObj.kubeletIdentityPrincipalId.value

      Write-Host "kubeletPrincipalId:"
      Write-Host $kubeletPrincipalId

      Write-Host "##vso[task.setvariable variable=kubeletPrincipalId;isOutput=true]$kubeletPrincipalId"

In the next stage, we can grab that variable and use that as input by declaring it as a variable on the stage:

- stage: DeployProd

  variables:
  - name: devKubeletPrincipalId 
    value: $[ stageDependencies.DeployDev.DeployResources.outputs['DeployResources.SetOutputs.kubeletPrincipalId'] ]

The last task in the pipeline can then use that in the role assignments:

- task: [email protected]
    displayName: Role assignments on ACR
    inputs:
        azureSubscription: ${{ variables.ProdAzureResourceManagerConnection }}
        scriptType: pscore
        scriptLocation: inlineScript
        addSpnToEnvironment: true
        inlineScript: |
            $armOutputObj = $env:MAINDEPLOYMENTS_OUTPUTS | convertfrom-json
            write-host "prod ID"
            write-host $armOutputObj.kubeletIdentityPrincipalId.value

            write-host "Dev ID"
            write-host $(devKubeletPrincipalId)

            write-host "ACR ID"
            write-host $armOutputObj.acrId.value

            az role assignment create --role "AcrPull" --assignee-object-id $(devKubeletPrincipalId) --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal
            az role assignment create --role "AcrPull" --assignee-object-id $armOutputObj.kubeletIdentityPrincipalId.value --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal

Here’s the complete pipeline. Don’t forget to create the environments used in the pipeline, and the identity used to run the production stage needs to have permission to do a role assignment.

trigger: none

pool:
  vmImage: ubuntu-latest

variables:
  ModulesFolderPath: $(Build.SourcesDirectory)/multi-stage-pass-variables/Bicep
  ParameterFilesFolderPath: $(Build.SourcesDirectory)/multi-stage-pass-variables/Pipelines

  NonProdSubscriptionId: 'f0e483fc-9d2f-4a4b-8aee-887a398ff27e'
  NonProdAzureResourceManagerConnection: 'Azure MVP Sponsorship'

  ProdSubscriptionId: 'f0e483fc-9d2f-4a4b-8aee-887a398ff27e'
  ProdAzureResourceManagerConnection: 'Azure MVP Sponsorship'

stages:
- stage: 'PublishTemplates'
  displayName: 'PublishTemplates'

  variables:
    MainBicepFilePath: '${{ variables.ModulesFolderPath }}/mainDeployment.bicep'
    MainTemplateFilePath: '${{ variables.ModulesFolderPath }}/mainDeployment.json'
    MainTemplateTestParameterFilePath: '${{ variables.ParameterFilesFolderPath }}/parameters.dev.json'

  jobs:
    - job: PublishTemplates
      displayName: Publish Templates

      steps:
        - checkout: self
          path: src

        - task: [email protected]
          displayName: Bicep Build
          inputs:
            targetType: 'inline'
            script: |
              az bicep build --file ${{ variables.MainBicepFilePath }}
            pwsh: true

        - task: [email protected]
          displayName: Validate ARM Template
          inputs:
            azureResourceManagerConnection: ${{ variables.NonProdAzureResourceManagerConnection }}
            deploymentScope: 'Subscription'
            subscriptionId: '${{ variables.NonProdSubscriptionId }}'
            location: 'West Europe'
            templateLocation: 'Linked artifact'
            csmFile: '${{ variables.MainTemplateFilePath }}'
            csmParametersFile: '${{ variables.MainTemplateTestParameterFilePath }}'
            deploymentMode: 'Validation'

        - task: [email protected]
          displayName: 'Copy templates to $(build.artifactstagingdirectory)/templates'
          inputs:
            SourceFolder: ${{ variables.ModulesFolderPath }}
            Contents: '*.json'
            TargetFolder: '$(build.artifactstagingdirectory)/templates'

        - task: [email protected]
          displayName: 'Copy parameters to $(build.artifactstagingdirectory)/templates'
          inputs:
            SourceFolder: ${{ variables.ParameterFilesFolderPath }}
            Contents: '*.json'
            TargetFolder: '$(build.artifactstagingdirectory)/templates'

        - task: [email protected]
          displayName: 'Publish Artifact: drop'

- stage: DeployDev

  displayName: Deploy template to dev environment
  jobs:
    - deployment: DeployResources
      environment: 'IaCDevEnvironment'
      strategy:
        runOnce:
          deploy:
            steps:
            - template: resourcesDeployment.yml
              parameters:
                SubscriptionId: ${{ variables.NonProdSubscriptionId }}
                Location: West Europe
                AzureResourceManagerConnection: ${{ variables.NonProdAzureResourceManagerConnection }}        
                Environment: 'dev'            

- stage: DeployProd

  variables:
  - name: devKubeletPrincipalId 
    value: $[ stageDependencies.DeployDev.DeployResources.outputs['DeployResources.SetOutputs.kubeletPrincipalId'] ]

  displayName: Deploy template to prod environment
  jobs:
  - deployment:
    environment: 'IaCProdEnvironment'
    strategy:
      runOnce:
        deploy:
          steps:
          
          - template: resourcesDeployment.yml
            parameters:
              SubscriptionId: ${{ variables.ProdSubscriptionId }}
              Location: West Europe
              AzureResourceManagerConnection: ${{ variables.ProdAzureResourceManagerConnection }}        
              Environment: 'prod'

          - task: [email protected]
            displayName: Role assignments on ACR
            inputs:
              azureSubscription: ${{ variables.ProdAzureResourceManagerConnection }}
              scriptType: pscore
              scriptLocation: inlineScript
              addSpnToEnvironment: true
              inlineScript: |
                $armOutputObj = $env:MAINDEPLOYMENTS_OUTPUTS | convertfrom-json
                write-host "prod ID"
                write-host $armOutputObj.kubeletIdentityPrincipalId.value

                write-host "Dev ID"
                write-host $(devKubeletPrincipalId)

                write-host "ACR ID"
                write-host $armOutputObj.acrId.value

                az role assignment create --role "AcrPull" --assignee-object-id $(devKubeletPrincipalId) --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal
                az role assignment create --role "AcrPull" --assignee-object-id $armOutputObj.kubeletIdentityPrincipalId.value --scope $armOutputObj.acrId.value --assignee-principal-type ServicePrincipal