Azure DevOps Pipeline YAML Guide

Basic Structure

Azure DevOps pipelines use a hierarchical structure with three main levels:

• Stages: Logical groupings of work (e.g., Build, Test, Deploy)
• Jobs: Units of work that run on agents (can run in parallel within a stage)
• Steps: Individual tasks or scripts within a job (run sequentially)

Every pipeline starts with these fundamental elements:

# Pipeline name (optional)
name: 'My Pipeline $(Date:yyyyMMdd)$(Rev:.r)'

# Trigger configuration
trigger:
  branches:
    include:
      - main
      - develop
  paths:
    exclude:
      - docs/*
# Variables
variables:
  buildConfiguration: 'Release'
  vmImageName: 'ubuntu-latest'
# Stages, Jobs, and Steps
stages:
- stage: Build
  jobs:
  - job: BuildJob
    pool:
      vmImage: $(vmImageName)
    steps:
    - script: echo Hello World

Triggers

Triggers determine when your pipeline runs automatically. There are several types:

Branch Triggers

Branch triggers run the pipeline when code is pushed to specified branches. You can include or exclude specific branches and file paths.

trigger:
  branches:
    include:
      - main
      - feature/*
    exclude:
      - experimental/*
  paths:
    include:
      - src/*
    exclude:
      - docs/*
      - README.md

PR Triggers

Pull Request triggers run the pipeline when PRs are created or updated. This is essential for validating changes before they’re merged.

pr:
  branches:
    include:
      - main
  paths:
    exclude:
      - docs/*

Scheduled Triggers

Scheduled triggers use cron expressions to run pipelines at specific times. The always: true parameter ensures the pipeline runs even if there are no code changes.

schedules:
- cron: "0 0 * * *"
  displayName: Daily midnight build
  branches:
    include:
    - main
  always: true

Disable Triggers

Use this when you want to run pipelines manually only or trigger them from other pipelines.

trigger: none
pr: none

Variables

Variables store values that can be reused throughout your pipeline. They make pipelines more maintainable and flexible.

Pipeline Variables

Variables can be simple strings, computed values, or conditional based on pipeline context.

variables:
  # Simple variables
  buildConfiguration: 'Release'
  projectName: 'MyApp'
  
  # Computed variables
  buildNumber: $[counter(variables['Build.SourceBranchName'], 100)]
  
  # Conditional variables
  ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
    environment: 'production'
  ${{ else }}:
    environment: 'development'

Variable Groups

Variable groups are collections of variables stored in Azure DevOps that can be shared across multiple pipelines. They’re ideal for storing environment-specific configurations and secrets.

variables:
- group: 'shared-variables'
- name: localVar
  value: 'local-value'

Runtime Variables

Runtime variables are evaluated during pipeline execution and can use expressions to make dynamic decisions based on pipeline context.

variables:
  isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
  deployToProduction: $[and(eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables['Build.Reason'], 'Manual'))]

Stages, Jobs, and Steps

This section demonstrates the hierarchical structure and different job types available in Azure DevOps pipelines.

Multi-Stage Pipeline

Multi-stage pipelines separate concerns (build, test, deploy) and can have dependencies between stages. The dependsOn property controls execution order, and condition determines when stages run.

stages:
- stage: Build
  displayName: 'Build Stage'
  jobs:
  - job: BuildJob
    steps:
    - script: echo Building...

- stage: Test
  displayName: 'Test Stage'
  dependsOn: Build
  condition: succeeded()
  jobs:
  - job: UnitTests
    steps:
    - script: echo Running tests...
- stage: Deploy
  displayName: 'Deploy Stage'
  dependsOn: 
  - Build
  - Test
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
  - deployment: DeployJob
    environment: 'production'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo Deploying...

Job Types and Configurations

Azure DevOps supports different job types:

• Regular jobs: Standard build/test jobs
• Matrix jobs: Run the same job with different variable combinations (great for testing multiple versions)
• Deployment jobs: Special jobs for deploying to environments with approval gates

jobs:
# Regular job
- job: BuildJob
  displayName: 'Build Application'
  pool:
    vmImage: 'ubuntu-latest'
  timeoutInMinutes: 60
  cancelTimeoutInMinutes: 5
  
# Matrix job
- job: TestMatrix
  strategy:
    matrix:
      Python38:
        python.version: '3.8'
      Python39:
        python.version: '3.9'
      Python310:
        python.version: '3.10'
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '$(python.version)'

# Deployment job
- deployment: DeployToProduction
  displayName: 'Deploy to Production'
  environment: 'production'
  strategy:
    runOnce:
      deploy:
        steps:
        - script: echo Deploying to production

Agent Pools

Agent pools determine where your pipeline jobs run. You can use Microsoft-hosted agents (managed VMs) or self-hosted agents (your own machines).

Get Manu Francis’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Microsoft-hosted agents are convenient but have usage limits. Self-hosted agents give you more control but require maintenance. The demands property ensures jobs run on agents with specific capabilities.

pool:
  # Microsoft-hosted agents
  vmImage: 'ubuntu-latest'  # or 'windows-latest', 'macOS-latest'

# Self-hosted agent pool
pool:
  name: 'MyAgentPool'
# Demands for self-hosted agents
pool:
  name: 'MyAgentPool'
  demands:
  - Agent.OS -equals Linux
  - docker

Steps and Tasks

Steps are the individual actions your pipeline performs. Azure DevOps provides built-in tasks for common operations, but you can also run custom scripts.

Built-in Tasks

Built-in tasks are pre-configured actions that handle common scenarios like copying files, publishing artifacts, and running tests. They provide consistent interfaces and error handling.

steps:
# Checkout code
- checkout: self
  clean: true
  fetchDepth: 1

# Download artifacts
- download: current
  artifact: 'drop'
# Publish artifacts
- publish: $(Build.ArtifactStagingDirectory)
  artifact: 'drop'
# Copy files
- task: CopyFiles@2
  inputs:
    sourceFolder: '$(Build.SourcesDirectory)'
    contents: '**/*.dll'
    targetFolder: '$(Build.ArtifactStagingDirectory)'
# Archive files
- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)'
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
# Publish test results
- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/test-results.xml'
    mergeTestResults: true

Script Tasks

Script tasks let you run custom commands. You can use inline scripts or reference external script files. Different script types (bash, PowerShell, generic script) are available depending on the agent OS.

steps:
# Inline script
- script: |
    echo "Building application..."
    npm install
    npm run build
  displayName: 'Build with npm'

# PowerShell script
- powershell: |
    Write-Host "Running PowerShell script"
    Get-ChildItem -Path $(Build.SourcesDirectory)
  displayName: 'List files'
# Bash script
- bash: |
    echo "Running bash script"
    ls -la $(Build.SourcesDirectory)
  displayName: 'List files with bash'
# Script from file
- script: './scripts/build.sh'
  workingDirectory: '$(Build.SourcesDirectory)'

Conditions and Dependencies

Conditions and dependencies control when jobs and steps execute. This enables complex workflows where certain actions only happen under specific circumstances.

Job Conditions

Job conditions determine when jobs run based on the success/failure of previous jobs, branch names, build reasons, and other factors. Common condition functions include succeeded(), failed(), always(), and canceled().

jobs:
- job: BuildJob
  steps:
  - script: echo Building

- job: TestJob
  dependsOn: BuildJob
  condition: succeeded()
  steps:
  - script: echo Testing
- job: DeployJob
  dependsOn: 
  - BuildJob
  - TestJob
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  steps:
  - script: echo Deploying
# Custom conditions
- job: ConditionalJob
  condition: |
    and(
      succeeded(),
      eq(variables['Build.Reason'], 'PullRequest'),
      contains(variables['System.PullRequest.TargetBranch'], 'main')
    )

Step Conditions

Step conditions work similarly to job conditions but apply to individual steps within a job. This allows fine-grained control over which steps run based on various factors.

steps:
- script: echo "Always runs"

- script: echo "Only on main branch"
  condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
- script: echo "Only on manual builds"
  condition: eq(variables['Build.Reason'], 'Manual')
- script: echo "Only if previous step succeeded"
  condition: succeeded()
- script: echo "Runs even if previous steps failed"
  condition: always()

Templates

Templates promote reusability and consistency across pipelines. They allow you to define common patterns once and reuse them multiple times with different parameters.

Job Template (jobs/build-job.yml)

Job templates encapsulate entire jobs that can be parameterized and reused. Parameters can be simple values or complex objects like arrays. The ${{ each }} syntax enables loops over arrays.

parameters:
- name: buildConfiguration
  type: string
  default: 'Release'
- name: projects
  type: object
  default: ['**/*.csproj']

jobs:
- job: Build
  displayName: 'Build ${{ parameters.buildConfiguration }}'
  steps:
  - ${{ each project in parameters.projects }}:
    - task: DotNetCoreCLI@2
      inputs:
        command: 'build'
        projects: '${{ project }}'
        arguments: '--configuration ${{ parameters.buildConfiguration }}'

Using Templates

Templates are referenced using the template keyword. You can pass parameters to customize the template behavior for different scenarios.

stages:
- stage: Build
  jobs:
  - template: jobs/build-job.yml
    parameters:
      buildConfiguration: 'Release'
      projects:
      - 'src/WebApp/WebApp.csproj'
      - 'src/Api/Api.csproj'

Step Template

Step templates contain reusable sequences of steps. They’re ideal for common build patterns like Node.js, .NET, or Python workflows that you use across multiple projects.

# templates/npm-steps.yml
parameters:
- name: workingDirectory
  type: string
  default: '$(Build.SourcesDirectory)'

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '16.x'
- script: npm ci
  workingDirectory: ${{ parameters.workingDirectory }}
- script: npm run build
  workingDirectory: ${{ parameters.workingDirectory }}
- script: npm test
  workingDirectory: ${{ parameters.workingDirectory }}

Resources

Resources allow your pipeline to use external artifacts, code repositories, and other pipelines. This enables complex CI/CD scenarios where pipelines depend on each other.

Repository Resources

Repository resources let you access code from other Git repositories during your pipeline execution. This is useful for accessing shared libraries, templates, or configuration files.

resources:
  repositories:
  - repository: templates
    type: git
    name: MyProject/pipeline-templates
    ref: refs/heads/main
  - repository: external
    type: github
    endpoint: MyGitHubConnection
    name: owner/repo

stages:
- stage: Build
  jobs:
  - template: build-template.yml@templates

Pipeline Resources

Pipeline resources create dependencies between pipelines. When an upstream pipeline completes, it can automatically trigger downstream pipelines. This enables complex CI/CD workflows where multiple pipelines work together.

resources:
  pipelines:
  - pipeline: upstream
    source: 'MyProject-CI'
    trigger:
      branches:
        include:
        - main

stages:
- stage: Deploy
  jobs:
  - job: DeployJob
    steps:
    - download: upstream
      artifact: 'drop'

Environments and Deployments

Environments represent deployment targets (dev, staging, production) and can have approval gates, security policies, and deployment history tracking.

Environment Configuration

Deployment jobs are special job types designed for deploying applications. They provide deployment history, rollback capabilities, and integration with Azure DevOps environments.

stages:
- stage: Deploy
  jobs:
  - deployment: DeployToStaging
    displayName: 'Deploy to Staging'
    environment: 'staging'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo Deploying to staging

  - deployment: DeployToProduction
    displayName: 'Deploy to Production'
    environment: 'production'
    dependsOn: DeployToStaging
    condition: succeeded()
    strategy:
      runOnce:
        preDeploy:
          steps:
          - script: echo Pre-deployment checks
        deploy:
          steps:
          - script: echo Deploying to production
        postDeploy:
          steps:
          - script: echo Post-deployment verification

Deployment Strategies

Azure DevOps supports different deployment strategies:

• runOnce: Simple deployment that runs once
• rolling: Gradually updates instances to minimize downtime
• canary: Gradually increases traffic to new version for safe rollouts

# Rolling deployment
strategy:
  rolling:
    maxParallel: 2
    preDeploy:
      steps:
      - script: echo Pre-deploy
    deploy:
      steps:
      - script: echo Deploy
    postDeploy:
      steps:
      - script: echo Post-deploy

# Blue-Green deployment
strategy:
  canary:
    increments: [10, 25, 50, 100]
    preDeploy:
      steps:
      - script: echo Pre-deploy
    deploy:
      steps:
      - script: echo Deploy $(strategy.canary.increment)%
    postDeploy:
      steps:
      - script: echo Post-deploy

Advanced Features

These features enable sophisticated pipeline scenarios and dynamic behavior.

Loops and Iterations

Template expressions support loops and conditional logic at compile time. This enables generating multiple stages, jobs, or steps based on parameters, making pipelines highly dynamic and reusable.

parameters:
- name: environments
  type: object
  default:
  - name: 'dev'
    vmImage: 'ubuntu-latest'
  - name: 'staging'
    vmImage: 'windows-latest'
  - name: 'prod'
    vmImage: 'ubuntu-latest'

stages:
- ${{ each env in parameters.environments }}:
  - stage: Deploy_${{ env.name }}
    displayName: 'Deploy to ${{ env.name }}'
    jobs:
    - job: Deploy
      pool:
        vmImage: ${{ env.vmImage }}
      steps:
      - script: echo Deploying to ${{ env.name }}

Container Jobs

Container jobs run inside Docker containers, providing consistent environments and pre-configured tooling. You can also use service containers for dependencies like databases or caches during testing.

jobs:
- job: ContainerJob
  container:
    image: 'node:16'
    options: '--cpus="2.0" --memory="4g"'
  steps:
  - script: |
      node --version
      npm --version

# Multiple containers
jobs:
- job: MultiContainer
  pool:
    vmImage: 'ubuntu-latest'
  container: 'node:16'
  services:
    postgres:
      image: 'postgres:13'
      env:
        POSTGRES_PASSWORD: 'password'
      ports:
      - 5432:5432
  steps:
  - script: npm test

Expressions and Functions

Azure DevOps provides powerful expression syntax for dynamic behavior. Expressions are evaluated at runtime and can manipulate strings, make decisions, and access pipeline context. Common functions include string manipulation, logical operations, and dependency checking.

variables:
  # String functions
  branchName: $[replace(variables['Build.SourceBranch'], 'refs/heads/', '')]
  shortSha: $[substring(variables['Build.SourceVersion'], 0, 7)]
  
  # Conditional expressions
  deploymentRing: $[if(eq(variables['Build.SourceBranch'], 'refs/heads/main'), 'prod', 'dev')]
  
  # Dependencies
  shouldDeploy: $[and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))]

steps:
- script: echo Branch: $(branchName)
- script: echo Short SHA: $(shortSha)
- script: echo Deploy to: $(deploymentRing)

Error Handling and Retry

Azure DevOps provides several mechanisms for handling failures:

• continueOnError: Allows pipeline to continue even if a step fails
• condition: always(): Ensures cleanup steps run regardless of previous failures
• Custom retry logic in scripts for resilient operations

steps:
- script: |
    # Script that might fail
    exit 1
  continueOnError: true
  displayName: 'Step that might fail'

- script: |
    # Retry logic
    for i in {1..3}; do
      if command_that_might_fail; then
        break
      fi
      sleep 5
    done
  displayName: 'Retry command'
- script: echo "This runs even if previous steps failed"
  condition: always()

Security and Best Practices

Security is crucial in CI/CD pipelines, especially when dealing with sensitive data and production deployments.

Secret Variables

Secret variables are encrypted and masked in logs. They should be stored in variable groups with appropriate permissions. Never hardcode secrets in YAML files — always use secure variable storage.

variables:
- group: 'production-secrets'  # Contains secret variables

steps:
- script: |
    # Use secrets in scripts
    echo "Connecting to $(DatabaseConnectionString)"
  env:
    DATABASE_CONNECTION: $(DatabaseConnectionString)

Pipeline Security

Security best practices include:

• Limiting which branches can trigger pipelines
• Using environment approvals for production deployments
• Restricting pipeline permissions
• Using service connections for external resource access
• Regular security reviews of pipeline configurations

Key Concepts Summary:

1. Hierarchy: Stages contain jobs, jobs contain steps
2. Triggers: Control when pipelines run (branch, PR, schedule)
3. Variables: Store and reuse values across pipeline runs
4. Conditions: Control execution flow based on various factors
5. Templates: Promote reusability and consistency
6. Environments: Represent deployment targets with governance
7. Resources: Connect pipelines to external dependencies
8. Security: Protect sensitive data and control access

Best Practices:

• Use templates for common patterns
• Implement proper error handling
• Secure sensitive data in variable groups
• Use environments for deployment governance
• Keep pipelines simple and readable
• Test pipeline changes in non-production first

# Restrict pipeline to specific branches
trigger:
  branches:
    include:
    - main
    - release/*

# Require approval for sensitive environments
stages:
- stage: Production
  jobs:
  - deployment: ProductionDeploy
    environment: 'production'  # Configure approvals in Azure DevOps UI

This guide covers the majority of Azure DevOps pipeline YAML features. Start with the basic structure and gradually incorporate more advanced features as needed. Remember to use pipeline validation and testing to ensure your YAML is correct before running in production.