3.1.3.2. Stages, Jobs, Templates, and Matrix Strategies

3.1.3.2. Stages, Jobs, Templates, and Matrix Strategies

With YAML fundamentals in place, multi-stage pipelines combine stages, jobs, and templates into complete deployment workflows.

💡 First Principle: The fundamental purpose of pipeline design is to create an automated, version-controlled, and repeatable workflow that transforms source code into a deployed application, ensuring consistency, quality, and traceability at every step.

🏭 Think of a CI/CD pipeline like an automated assembly line — raw materials (source code) enter at one end, pass through stations (build, test, scan, deploy) in a defined sequence, and a finished product (deployed application) emerges at the other end. The line is defined in code (YAML), versioned, and reproducible.

Scenario: Your organization needs to automate the build, test, and deployment of a new web application. They want to use Azure Pipelines, ensure consistency across environments (dev, test, prod), and require manual approval before deploying to production. They also have specific build tools that require a custom build environment.

What It Is: Pipeline design and implementation is the process of defining, building, and deploying automated workflows for software delivery, from code commit to production release.

Selecting a deployment automation solution involves choosing between platforms like GitHub Actions and Azure Pipelines. GitHub Actions excels in open-source projects and integrates deeply with the GitHub ecosystem, while Azure Pipelines offers robust enterprise features and seamless integration with Azure services.

Designing agent infrastructure, whether GitHub-hosted runners or Azure DevOps agents (Microsoft-hosted or self-hosted), requires considering cost, toolchain compatibility, licensing, network connectivity, and maintainability. Self-hosted agents provide control over the environment, crucial for specific tool requirements or restricted networks.

Pipeline trigger rules automate execution based on events. Common types include Continuous Integration (CI) triggers for code commits, pull request (PR) triggers for validation, and scheduled triggers for routine tasks (e.g., nightly builds, security scans).

Developing pipelines using YAML offers significant benefits: version control, reusability (via templates), and consistency across environments. This declarative approach ensures that pipeline definitions are treated as code.

Complex pipeline scenarios, such as hybrid deployments (e.g., deploying to on-premises servers), leveraging VM templates, or utilizing self-hosted runners or agents, address diverse infrastructure needs. These enable deployments across on-premises and cloud environments.

Reusable pipeline elements like YAML templates, task groups, variables, and variable groups promote modularity and reduce duplication. Templates define common patterns, while variables manage configuration specifics.

Finally, designing and implementing checks and approvals, often via YAML-based environments, provides critical gatekeeping for deployments. This ensures that code meets quality standards and requires human sign-off before progressing to sensitive environments.

Key Components of Pipeline Design and Implementation:

• Platforms: GitHub Actions, Azure Pipelines.
• Agent Infrastructure: Microsoft-hosted agents, self-hosted agents, GitHub-hosted runners, self-hosted runners.
• Triggers: CI, PR, Scheduled.
• Definition Language: YAML.
• Reusable Elements: YAML templates, task groups, variables, variable groups.
• Quality Gates: Checks and Approvals.

⚠️ Common Pitfall: Creating pipelines with hardcoded values (e.g., server names, connection strings). This makes the pipeline inflexible and difficult to reuse across different environments. Use variables and variable groups instead.

Key Trade-Offs:

• Microsoft-Hosted vs. Self-Hosted Agents: Microsoft-hosted agents are convenient and require no maintenance but offer limited customization. Self-hosted agents provide full control over the environment and software but require you to manage and maintain them.

Practical Implementation: Azure Pipelines YAML with Stages and Approvals

stages:
- stage: Build
  jobs:
  - job: BuildJob
    # ... build steps ...

- stage: Deploy_Staging
  dependsOn: Build
  jobs:
  - deployment: DeployWeb
    environment: 'Staging'
    strategy:
      runOnce:
        deploy:
          # ... deployment steps ...

- stage: Deploy_Production
  dependsOn: Deploy_Staging
  jobs:
  - deployment: DeployWeb
    environment: 'Production' # This environment has an approval check configured
    strategy:
      runOnce:
        deploy:
          # ... deployment steps ...

GitHub Actions YAML Structure:

While Azure Pipelines uses stages → jobs → steps, GitHub Actions uses workflows → jobs → steps. A workflow is triggered by events (push, pull_request, schedule, workflow_dispatch). Jobs run in parallel by default (use needs: for sequential ordering). Each job runs on a runner (GitHub-hosted or self-hosted). Steps execute actions (reusable marketplace components) or run scripts.

name: CI Pipeline
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-dotnet@v4
      with:
        dotnet-version: '8.0.x'
    - run: dotnet build --configuration Release
    - run: dotnet test --collect:"XPlat Code Coverage"
    - uses: actions/upload-artifact@v4
      with:
        name: build-output
        path: ./artifacts/

  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging
    steps:
    - uses: actions/download-artifact@v4
    - uses: azure/webapps-deploy@v3
      with:
        app-name: myapp-staging

Matrix Builds and Parallel Execution:

Matrix builds run the same job across multiple configurations simultaneously. This is essential for cross-platform testing, multi-version compatibility, or running test suites in parallel shards.

In GitHub Actions:

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest]
    dotnet-version: ['6.0.x', '8.0.x']
runs-on: ${{ matrix.os }}

In Azure Pipelines:

strategy:
  matrix:
    Linux_DotNet6:
      vmImage: 'ubuntu-latest'
      dotnetVersion: '6.0.x'
    Windows_DotNet8:
      vmImage: 'windows-latest'
      dotnetVersion: '8.0.x'

Pipeline Trigger Rules — Complete Reference: