Azure Data Factory build and deploy with new CI/CD flow using GitHub Actions | by Jared Zagelbaum | Microsoft Azure | Medium

The data factory marketplace actions are no longer being maintained as Microsoft makes the transition to fully investing and integrating Data Factory functionality into Microsoft Fabric.

To support customers that are continuing to use Azure Data Factory along with GitHub Actions, I’ve developed a new sample yaml for the CICD workflow as a reference: Sample-CICD-for-ADF-with-GitHub-Actions/.github/workflows/cicd.yml at main · jzagelbaum/Sample-CICD-for-ADF-with-GitHub-Actions.

This action workflow follows all of the recommended practices for Azure Data Factory Releases (Automated publishing for continuous integration and delivery — Azure Data Factory | Microsoft Learn) including the pre and post deployment PowerShell script and uses the latest versions of the Azure marketplace action dependencies.

The authentication to Azure CLI leverages a user assigned managed identity and federated credentials per best practice and the setup steps are included in Olga Molocenco’s prior post: Azure Data Factory CI/CD with GitHub Actions | Microsoft Community Hub

The ADFUtilities NPM package has not received any recent updates as of late either. I hope this helps bridge any gaps in the meantime for those of you supporting Azure Data Factory implementations in your current environment.

name: Validate, Export and Deploy ARM Template and ADF pre / post deployment scripts

on:
  push:
    branches:
      - main

permissions:
  id-token: write
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Install Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '18.x'

    - name: Install npm packages
      run: npm install
      working-directory: ${{github.workspace}}/build # replace with the package.json folder

    - name: Validate Data Factory resources
      run: npm run build validate ${{github.workspace}} /subscriptions/<replace with dev sub id>/resourceGroups/<replace with dev rg>/providers/Microsoft.DataFactory/factories/<replace with dev adf name>
      working-directory: ${{github.workspace}}/build # replace with the package.json folder

    - name: Validate and Generate ARM template
      run: npm run build export ${{github.workspace}} /subscriptions/<replace with dev sub id>/resourceGroups/<replace with dev rg>/providers/Microsoft.DataFactory/factories/<replace with dev adf name> "ArmTemplate"
      # For using preview that allows you to only stop/start triggers that are modified, uncomment the below line and make sure the package.json contains the build-preview command.
      # run: npm run build-preview export ./<Root-folder-from-Git-configuration-settings-in-ADF> /subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/GartnerMQ2021/providers/Microsoft.DataFactory/factories/Dev-GartnerMQ2021-DataFactory "ArmTemplate"
      working-directory: ${{github.workspace}}/build # replace with the package.json folder

    - name: Upload ARM template artifact
      uses: actions/upload-artifact@v4
      with:
        name: ArmTemplates
        path: ${{github.workspace}}/build/ArmTemplate # replace with generated ARM template folder

  release:
    needs: build
    runs-on: ubuntu-latest

    steps:

        
 # we 1st download the previously uploaded artifact so we can leverage it later in the release job     
      - name: Download a Build Artifact
        uses: actions/download-artifact@v4
        with:
          name: ArmTemplates # (5) Artifact name 


 # Display the structure of the downloaded files for debugging purposes
 #     - name: Display structure of downloaded files
 #       run: ls -R

      - name: Login via Az module
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
          enable-AzPSSession: true       


      - name: Run Pre-deployment PowerShell Script File
        uses: azure/powershell@v2
        with:
         inlineScript: |
          ./PrePostDeploymentScript.ps1 `
          -armTemplate ${{github.workspace}}/ARMTemplateForFactory.json `
          -ArmTemplateParameters ${{github.workspace}}/ARMTemplateParametersForFactory.json `
          -ResourceGroupName '<replace with target resource group for deployment>' `
          -DataFactoryName '<replace with target data factory name>'`
          -predeployment $true `
          -deleteDeployment $false
         azPSVersion: "latest"

      - name: Run ARM deploy
        uses: azure/arm-deploy@v2
        with:
            resourceGroupName: '<replace with target resource group for deployment>'  
            template: 'ARMTemplateForFactory.json' 
            parameters: ARMTemplateParametersForFactory.json factoryName=<replace with target data factory name>  #add additional overide parameters if needed here ''

            
      - name: Run Post-deployment PowerShell Script File
        uses: azure/powershell@v2
        with:
         inlineScript: |
          ./PrePostDeploymentScript.ps1 `
          -armTemplate ${{github.workspace}}/ARMTemplateForFactory.json ` 
          -ArmTemplateParameters ${{github.workspace}}/ARMTemplateParametersForFactory.json `
          -ResourceGroupName '<replace with target resource group for deployment>' `
          -DataFactoryName '<replace with target data factory name>'`
          -predeployment $false `
          -deleteDeployment $true
         azPSVersion: "latest"

A previous blog post in Technet, Azure Data Factory CI/CD with GitHub Actions, outlined how to setup GitHub Actions for ADF using the manual publish flow. The initial setup steps and requirements are the same up through to “Configure the GitHub Secrets”.

The updated flow:

Updated guidance on creating the workflow to build and deploy the ARM template:

In addition to the ADFUtilities NPM package, there are marketplace actions published by Azure that make it even easier to implement the build and deploy job steps into your workflow. For validate and generate ARM templates we can use data-factory-export and for release there is data-factory-deploy.

Putting it all together, we can implement this workflow as follows:

1. Follow steps from Azure Data Factory CI/CD with GitHub Actions through to “Configure the GitHub Secrets”.
2. Add an additional secret that is the name of the Azure Data Factory deployment target.
3. Add a .yaml workflow definition file in the .github/workflows directory of your collaboration branch. It can have any name; in my repo I just named it after the branch:

Sample .yaml:

name: Deploy ARM template to Azure Data Factory

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Validate ARM template
        uses: Azure/data-factory-validate-action@v1.1.5
     

      - name: Export Azure Data Factory
        id: export
        uses: Azure/data-factory-export-action@v1.2.0
     
      
      - name: Publish ARM template
        uses: actions/upload-artifact@v2
        with:
          name: adf-armtemplate
          path: ${{ steps.export.outputs.arm-template-directory }}
          if-no-files-found: error
  
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Login via Az module
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}
          enable-AzPSSession: true 
     
      - name: Download ARM template
        uses: actions/download-artifact@v3
        with:
          name: adf-armtemplate

      - name: Deploy resources
        uses: Azure/data-factory-deploy-action@v1.2.0
        with:
          resourceGroupName: ${{ secrets.AZURE_RG }} 
          dataFactoryName: ${{ secrets.AZURE_DATA_FACTORY }} 

          

For an explanation of the .yaml file, I phoned a friend:

5. Add a branch protection rule to require PRs against the collaboration branch (at minimum):

6. Disable Publish (from ADF Studio):

Now you have a baseline workflow for implementing the new ADF ci/cd flow using GitHub Actions. Instead of manually publishing updates to the ADF instance we initiate the build and deploy when committing the PR.

Releases to higher environments can be managed by cloning the workflow deploy (and build) steps and replacing triggers and secret references as needed. The use of marketplace actions allows us the implement best practices for ADF deployment with a minimum amount of code.

Azure Data Factory build and deploy with new CI/CD flow using GitHub Actions was originally published in Microsoft on Medium, where people are continuing the conversation by highlighting and responding to this story.