CI/CD pipeline for PowerShell

powershell jenkins ci-cd

One of the first step I took in the PowerShell journey of my team, and the topic of this post, was to address the need for a Continuous Integration and Delivery (CI/CD) pipeline for our PowerShell modules. Since the modules are built, published and consumed by our internal company infrastructure, there are some nuances that I had to account for when designing our approach for the pipeline. Having a well-defined, rock-solid CI/CD pipeline becomes a must-have as PowerShell develops into a key tool in our automations and processes.

Currently, the PowerShell pipeline has the following goals:

  • Install the module dependencies
  • Validate PowerShell code syntax
  • Run static code checker
  • Run tests in order of complexity: unit tests, then component tests, etc.
  • Package the module to be published

Before we begin

Some of the concepts and techniques that we are going to cover on this post are not necessarily at a beginner’s level; it is intended for those already familiar with PowerShell, instead. The implementation that we discuss below, builds mainly on existing work and knowledge shared by Kevin Marquette, Warren Frame and Mark Kraus. If you need to cover some ground on PowerShell modules and pipelines, you should checkout first the following posts:

You’re back! Awesome, let’s first cover some of the dependencies that we use to build the pipeline with.

Modules in use

To run the pipeline, we use several modules from the community, as well as some developed internally, that facilitate achieving the goals outlined for our system earlier in this post.

Community modules

As I mentioned them in a previous post about PowerShell, these modules are among the building blocks for any serious endeavors in PowerShell, well at least for me :).

Besides these excellent PowerShell modules, we use an internal module to generate a .nupkg file for the current module in the pipeline, so it can be stored to Artifactory, which act as our internal PowerShell gallery.

Pipeline

The pipeline is composed of two main components: a PowerShell implementation, that handles everything from validating the code, running tests and packaging the module; and the Jenkins component, that oversees the whole CI/CD starting from Git, passing through the PowerShell pipeline and ending it with the deployment to Artifactory.

This separation was drawn by our infrastructure itself, which only allows to publish to Artifactory repositories from our internal Jenkins servers. Enforcing this separation allows contributors to run the majority of the pipeline on their local boxes, thus improving the feedback cycle, quality and productivity.

PowerShell

The PowerShell component of the pipeline handles the following tasks:

  • Run static analysis, which includes tokenization to detect errors in PowerShell files using PSParser, verification of compliance with PSScriptAnalyzer rules (run as Pester tests) and presence of Set-StrictMode -Version Latest at the top of the files.
  • Build the Module (which includes copying all relevant files to the Output folder and update the module manifest (.psd1 file) accordingly.
  • Run unit tests and integration tests with Pester to verify the business logic of the module.
  • Publish the module as a .nupkg file in the Output location.

build.ps1

The build.ps1 file is the entry point for the PowerShell automation. As part of its execution, it:

  • Makes sure that NuGet packaged provider is configured (a prerequisite) and that module dependencies specified in the Development.depend.psd1 are locally met by using PSDepend.
  • Sets the environment information, i.e. detecting whether is running on a CI tool (e.g. Jenkins, Appveyor), a git repo is reachable in the path, etc., using BuildHelpers.
  • Finally, triggers the execution of the task-based portion of the automation, defined using InvokeBuild.

The implementation is heavily based on the build.ps1 from https://github.com/KevinMarquette/PSGraphPlus

<#
.Description
Installs and loads all the required modules for the build.
Derived from the PSGraphPlus repo (https://github.com/KevinMarquette/PSGraphPlus)
#>

[CmdletBinding()]
param (
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string]
    $Task = 'Default',
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string]
    $RepositoryName = 'TempFeed',
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string]
    $RepositoryLocation
)

Set-StrictMode -Version Latest

if (-not $PSBoundParameters.ContainsKey('RepositoryLocation')) {
    $RepositoryLocation = "$PSScriptRoot/$RepositoryName"
}

Write-Output "Starting build"

# Grab nuget bits, install modules, set build variables, start build.
Write-Output "  Install Dependent Modules for CI"
try {
    Get-PackageProvider -Name NuGet -ForceBootstrap | Out-Null
    Save-Module -Name PSDepend -Path "$PSScriptRoot/Dependencies" -RequiredVersion 0.3.0 -Force -Repository Artifactory
    Import-Module -Name "$PSScriptRoot/Dependencies/PSDepend" -Force
    Invoke-PSDepend -Path "$PSScriptRoot\Development.depend.psd1" -Install -Import -Force
}
catch {
    Write-Error $_
    exit 1
}

Write-Output "  Import Dependent Modules"
Import-Module InvokeBuild, BuildHelpers

Write-Output "  Set Build Environment"
Set-BuildEnvironment -Force

if (($env:BHBranchName -eq 'HEAD') -and (-not [string]::IsNullOrEmpty($env:BRANCH_NAME))) {
    Write-Output "  Update BHBranchName envvar"
    $env:BHBranchName = $env:BRANCH_NAME
}

$params = @{
    RepositoryName = $RepositoryName
    RepositoryLocation = $RepositoryLocation
    Result = 'Result'
}

Write-Output "  InvokeBuild"
Invoke-Build $Task @params

if ($Result.Error)
{
    exit 1
}
else
{
    exit 0
}

Development.depend.psd1

The Development.depend.psd1 file, similar to the one shown below, lays out the required dependencies for the module during the CI process as well as common options used to describe where/how those dependencies should be installed. As options, we specified the following:

  • Modules should be downloaded from our internal Artifactory repository, so we don’t hit rate limits on the PSGallery for the third-party modules in use.
  • Modules should be added to environment path variable, so they can be imported by name.
  • Modules should be saved on a local folder at the root repo level, so they are local to the CI repo workspace and avoid polluting the powershell host with unused modules for developers and the CI nodes.
@{
    PSDependOptions = @{
        Target = '$DependencyFolder/Dependencies'
        AddToPath = $True
        Parameters = @{
            Repository = 'Artifactory'
            SkipPublisherCheck = $true
        }
    }

    Pester = '4.6.0'
    PSScriptAnalyzer = '1.17.1'
    InvokeBuild = '5.4.2'
    BuildHelpers = '2.0.7'

    LocalRepositoryManager = '2.0.0'
}

Module.build.ps1

The Module.build.ps1 file is in charge of running the common tasks of a CI/CD process. The default workflow, represented by the Default task, spans all the defined tasks from building the module and running the associated tests, to generating a *.nupkg file that can be published to our internal Artifactory gallery. At glance, we can group the tasks as follows:

  • First, the build-related tasks, formed by Build, Clean, CopyToOutput, SetDirectoryStructure and BuildPSD1 take care that the latest files are copied to the Output folder (where the nupkg file will be created) with up-to-date information.
  • Then, the tests tasks, which include Pester, StaticAnalysisTests, UnitTests, IntegrationTests and SupportTests verify the different requirements and functionalities of the codebase.
  • Finally, the PublishToLocal task generates the required *.nupkg file to be sent to our internal repo.
#requires -Modules InvokeBuild, BuildHelpers, PSScriptAnalyzer, Pester

# Parameters
param(
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string]
    $RepositoryName = 'TempFeed',
    [Parameter()]
    [ValidateNotNullOrEmpty()]
    [string]
    $RepositoryLocation
)

Set-StrictMode -Version Latest

if (-not $PSBoundParameters.ContainsKey('RepositoryLocation')) {
    $RepositoryLocation = "$PSScriptRoot/$RepositoryName"
}

# Variables
$script:ModuleName = Get-ChildItem -File -Depth 1 -Filter *.psm1 | Select-Object -First 1 -ExpandProperty BaseName

$script:Source = Join-Path -Path $BuildRoot -ChildPath $ModuleName
$script:TestsPath = Join-Path -Path $BuildRoot -ChildPath 'Tests'
$script:Output = Join-Path -Path $BuildRoot -ChildPath 'Output'
$script:Destination = Join-Path -Path $Output -ChildPath $ModuleName
$script:ModulePath = Join-Path -Path $Destination -ChildPath "$ModuleName.psm1"
$script:ManifestPath = Join-Path -Path $Destination -ChildPath "$ModuleName.psd1"
$script:HelpRoot = Join-Path -Path $Output -ChildPath 'Help'


function TaskX($Name, $Parameters) {task $Name @Parameters -Source $MyInvocation}


Task Default Pester, PublishToLocal
Task Build Clean, SetDirectoryStructure, CopyToOutput, BuildPSD1
Task Pester StaticAnalysisTests, ImportModule, UnitTests, IntegrationTests, SupportTests


Task Clean {
    If (Test-Path -Path $Output)
    {
        Remove-Item $Output -Recurse -ErrorAction Ignore | Out-Null
    }

    if (Test-PSRepository -Name $RepositoryName) {
        Unregister-LocalRepository -Name $RepositoryName
    }
}


Task StaticAnalysisTests SetDirectoryStructure, {
    $testResults = Invoke-Pester -Path "$TestsPath\StaticAnalysis.Tests.ps1" -PassThru -Strict -OutputFormat NUnitXml -OutputFile "$Output\TestResults_StaticAnalysis.xml"

    if ($testResults.FailedCount -gt 0)
    {
        Write-Error "Failed [$($testResults.FailedCount)] Pester tests"
        $testResults | Format-List
    }
}


Task UnitTests ImportModule, {
    $testResults = Invoke-Pester -Path "$TestsPath\Unit" -PassThru -Strict -OutputFormat NUnitXml -OutputFile "$Output\TestResults_Unit.xml"

    if ($testResults.FailedCount -gt 0)
    {
        Write-Error "Failed [$($testResults.FailedCount)] Pester tests"
        $testResults | Format-List
    }
}


Task IntegrationTests ImportModule, {
    $testResults = Invoke-Pester -Path "$TestsPath\Integration" -PassThru -Strict -OutputFormat NUnitXml -OutputFile "$Output\TestResults_Integration.xml"

    if ($testResults.FailedCount -gt 0)
    {
        Write-Error "Failed [$($testResults.FailedCount)] Pester tests"
        $testResults | Format-List
    }
}


Task SupportTests {
    $testResults = Invoke-Pester -Path "$TestsPath\Support.Tests.ps1" -PassThru -Strict -OutputFormat NUnitXml -OutputFile "$Output\TestResults_Support.xml"

    if ($testResults.FailedCount -gt 0)
    {
        Write-Error "Failed [$($testResults.FailedCount)] Pester tests"
        $testResults | Format-List
    }
}


Task SetDirectoryStructure {
    New-Item -Path $Output -ItemType Directory -Force
}


Task CopyToOutput {
    Write-Output "  Create Directory [$Destination]"
    New-Item -Type Directory -Path $Destination -ErrorAction Ignore | Out-Null

    Get-ChildItem $Source -File |
        Where-Object { $_.Name -NotMatch "$ModuleName\.psd1" } |
        Copy-Item -Destination $Destination -Force -PassThru |
        ForEach-Object { "  Copy [.{0}]" -f $_.FullName.Replace($PSScriptRoot, '') }

    Get-ChildItem $Source -Directory |
        Copy-Item -Destination $Destination -Recurse -Force -PassThru |
        ForEach-Object { "  Copy [.{0}]" -f $_.FullName.Replace($PSScriptRoot, '') }
}


TaskX BuildPSD1 @{
    Inputs  = (Get-ChildItem $Source -Recurse -File)
    Outputs = $ManifestPath
    Jobs    = {

        Write-Output "  Update [$ManifestPath]"
        Copy-Item "$Source\$ModuleName.psd1" -Destination $ManifestPath

        Import-Module $ModulePath

        $version = [version] (Get-ModuleVersion)

        Write-Output "  Using version: $version"

        $updateParams = @{
            Path = $ManifestPath
            PropertyName = 'ModuleVersion'
            Value = $version
        }

        Update-Metadata @updateParams

        if ($env:BHBranchName -ne "master") {
            $updateParams.PropertyName = 'Prerelease'
            $updateParams.Value = Format-ModuleVersionSuffix -ReleaseType $env:BHBranchName -Revision $env:BHBuildNumber

            Write-Output "  Using prerelease suffix: $($updateParams.Value)"
            Update-Metadata @updateParams
        }

        Write-Output "  Export functions to manifest"
        Set-ModuleFunction -Path $ManifestPath
    }
}


Task ImportModule Build, {
    if ( -not (Test-Path $ManifestPath))
    {
        Write-Output "  Module [$ModuleName] is not built, cannot find [$ManifestPath]"
        Write-Error "Could not find module manifest [$ManifestPath]. You may need to build the module first"
    }
    else
    {
        if (Get-Module $ModuleName)
        {
            Write-Output "  Unloading Module [$ModuleName] from previous import"
            Remove-Module $ModuleName
        }
        Write-Output "  Importing Module [$ModuleName] from [$ManifestPath]"
        Import-Module $ManifestPath -Force
    }
}


Task PublishToLocal Build, {
    Register-LocalRepository -Name $RepositoryName -Location $RepositoryLocation

    $publishParams = @{
        RepositoryName = $RepositoryName
    }

    $ManifestPath | Publish-ModuleToLocalRepository @publishParams
}


function Format-ModuleVersionSuffix
{
    [CmdletBinding()]
    [OutputType([string])]
    param(
        [Parameter(Mandatory)]
        [ValidateNotNullOrEmpty()]
        [string]
        $ReleaseType,
        [Parameter(Mandatory)]
        [ValidateScript({$_ -ge 0})]
        [int]
        $Revision
    )

    "$(Format-ReleaseType -ReleaseType $ReleaseType)b$Revision"
}

function Format-ReleaseType
{
    [CmdletBinding()]
    [OutputType([string])]
    param(
        [Parameter(Mandatory)]
        [ValidateNotNullOrEmpty()]
        [string]
        $ReleaseType
    )

    $result = $ReleaseType -replace "[^a-zA-Z0-9]", ""

    $result.ToLower()
}

StaticAnalysis.Tests.ps1

The StaticAnalysis.Tests.ps1 file contains the tests that verifies the quality of PowerShell code written in the module. As mentioned in the beginning of this section, it takes care of:

  • Tokenizing the code to verify there are no parsing errors, both for the tests and module code
  • Running the allowed PSScriptAnalyzer rules, from the PSScriptAnalyzerSettings.psd1 at the module root folder, against the module source code.
  • And, verifying that all files from the module and tests folder have set the strict mode to Latest, so scripts will always use the latest version available and take the advantage of the improvements of the coding rules verification.
Set-StrictMode -Version Latest

$projectRoot = Resolve-Path -Path "$PSScriptRoot\.."

Describe "General Module syntax validation" {
    BeforeAll {
        function RunSyntaxTests([string] $Path)
        {
            $scripts = Get-ChildItem -Path $Path -Include *.ps1, *.psm1, *.psd1 -Recurse |
                Where-Object {$_.FullName -notmatch 'powershell'}

            # TestCases are splatted to the script so we need hashtables
            $testCases = $scripts | Foreach-Object {@{file = $_}}

            It "Script <file> should be valid powershell" -TestCases $testCases {
                param($file)

                $file.FullName | Should Exist

                $contents = Get-Content -Path $file.FullName -ErrorAction Stop
                $errors = $null
                $null = [System.Management.Automation.PSParser]::Tokenize($contents, [ref]$errors)
                $errors.Count | Should Be 0
            }
        }
    }

    Context 'Modules' {
        $path = Join-Path -Path $projectRoot -ChildPath 'Module'

        RunSyntaxTests -Path $path
    }

    Context 'Tests' {
        $path = Join-Path -Path $projectRoot -ChildPath 'Tests'

        RunSyntaxTests -Path $path
    }
}

Describe 'Static Analysis Checker' {
    BeforeAll {
        $settingsPath = Join-Path -Path $PSScriptRoot -ChildPath "../Module/PSScriptAnalyzerSettings.psd1"
        $analyzerConfig = Import-PowerShellDataFile -Path $settingsPath
        $rules = Get-ScriptAnalyzerRule -Severity $analyzerConfig.Severity

        function RunStaticAnalysis([string] $Path)
        {
            $dirs = @(
                Get-ChildItem -Path $Path -Directory |
                    Where-Object {
                        $_.FullName -notmatch "powershell"
                    }
            )
            foreach ($rule in $rules) {
                if ($rule -notin $analyzerConfig.ExcludeRules) {
                    It "should not return any violation for rule: $($rule.RuleName)" {
                        foreach ($dir in $dirs) {
                            Invoke-ScriptAnalyzer -Settings $settingsPath -Path $dir.FullName -IncludeRule $rule.RuleName -Recurse | Should -BeNullOrEmpty
                        }
                    }
                }
            }
        }
    }
    Context 'Module Source' {
        $path = Join-Path -Path $projectRoot -ChildPath 'Module'

        RunStaticAnalysis -Path $path
    }
}

# Based on the pester repo project
# at: https://github.com/pester/Pester/blob/master/Pester.Tests.ps1#L395
Describe 'Set-StrictMode to latest for files' {

    BeforeAll {
        function RunCheckForStrictMode([string] $Path)
        {
            $files = @(
                Get-ChildItem -Path $Path -File -Include *.ps1, *.psm1 -Recurse
            )

            It "files in $Path start with explicit declaration of StrictMode set to Latest" {
                $UnstrictTests = @(
                    foreach ($file in $files) {
                        $lines = [System.IO.File]::ReadAllLines($file.FullName)
                        $lineCount = $lines.Count
                        if ($lineCount -lt 3) {
                            $linesToRead = $lineCount
                        }
                        else {
                            $linestoRead = 3
                        }
                        $n = 0
                        for ($i = 0; $i -lt $linestoRead; $i++) {
                            if ($lines[$i] -match '\s+Set-StrictMode\ -Version\ Latest' -or $lines[$i] -match 'Set-StrictMode\ -Version\ Latest' ) {
                                $n++
                            }
                        }
                        if ( $n -eq 0 ) {
                            $file.FullName
                        }
                    }
                )
                if ($UnstrictTests.Count -gt 0) {
                    throw "The following $($UnstrictTests.Count) files doesn't contain strict mode declaration in the first three lines: $([System.Environment]::NewLine)$([System.Environment]::NewLine)$($UnstrictTests -join "$([System.Environment]::NewLine)")"
                }
            }
        }

    }

    Context 'Module Source' {
        $path = Join-Path -Path $projectRoot -ChildPath 'Module'

        RunCheckForStrictMode -Path $path
    }

    Context 'Tests' {
        $path = Join-Path -Path $projectRoot -ChildPath 'Tests'

        RunCheckForStrictMode -Path $path
    }
}

Jenkins

The Jenkins part of the pipeline takes care of interacting with the following tasks:

  • Fetch the latest source code data for the branch from the git repository.
  • Invoke the default behavior for the PowerShell pipeline (which includes, building the module, testing and packaging as .nupkg file).
  • Publish the artifact to Artifactory, along with build metadata, first to our powershell-dev repo, which is used to store prerealese versions of the module, so there is the possibility of further manual testing if required, and optionally promoted to our powershell-internal repo if we are releasing from master, so we can keep these versions long term.
#!/usr/bin/env groovy

//noinspection GroovyUnusedAssignment
@Library('jenkinsLibrary@release/2.x') _

buildTag = env.BUILD_TAG
branchName = env.BRANCH_NAME
buildNumber = env.BUILD_NUMBER
buildUser = env.BUILD_USER_ID

releaseMap = [master: 'Release', release: 'RC', develop: 'WIP']

def releaseType() {
    def branchReleaseName = branchName.startsWith('release') ? 'release' : branchName

    releaseMap.get(branchReleaseName, null)
}

def shouldPublish() {
    releaseType() != null
}

def isRelease() {
    releaseType() == 'Release'
}

timestamps {
    try {
        currentBuild.result = 'SUCCESS'
        node('BuildVM-PowerShell') {
            stage('Git') {
                echo "Pull git"
                checkout([
                            $class: 'GitSCM',
                            branches: scm.branches,
                            extensions: scm.extensions + [[$class: 'CleanCheckout']],
                            userRemoteConfigs: scm.userRemoteConfigs
                        ])
            }

            def repositoryLocation = "${pwd()}/Repository_${buildNumber}"

            try {
                stage('Build') {
                    def params = """
                        \$params = @{
                            RepositoryName='${buildTag}'
                            RepositoryLocation='${repositoryLocation}'
                        }
                    """

                    powershell """
                        ${params}
                        ${pwd()}/build.ps1 @params
                    """
                }

                stage('Publish Test Results') {
                    nunit testResultsPattern: "**/*Test*.xml", failIfNoResults: true, keepJUnitReports: true
                }

                if (shouldPublish()) {
                    def version = powershell(
                            script: "${pwd()}/GetVersion.ps1",
                            returnStdout: true
                        ).trim()

                    def artifactId =  powershell(
                            script: "${pwd()}/GetArtifactId.ps1",
                            returnStdout: true
                        ).trim()

                    def repoToPublish = 'powershell-dev'

                    stage('Publish Artifacts') {
                        final buildInfo = Artifactory.newBuildInfo()

                        artifactory.upload(buildInfo, [
                            productId  : 'PSM',
                            repoName   : repoToPublish,
                            groupId    : '.',
                            artifactId : artifactId,
                            version    : version,
                            uploads    : ["${repositoryLocation}": "*.nupkg"],
                        ])

                        if (isRelease()) {
                            stage('Promote To Internal') {
                                echo "Promoting nuget artifacts from 'powershell-dev' to 'powershell-internal' artifact repository ..."

                                artifactory.promote(buildUser, buildInfo, 'powershell-dev', 'powershell-internal')
                            }
                        }
                    }
                }
            }
            catch (Exception ex) {
                throw ex
            }
            finally {
                stage('Cleanup') {
                    powershell """
                        \$params = @{
                            RepositoryName='${buildTag}'
                            RepositoryLocation='${repositoryLocation}'
                            Task = 'Clean'
                        }
                        ${pwd()}/build.ps1 @params
                    """
                }
            }

        }
    }
    catch (Exception ex) {
        currentBuild.result = 'FAILURE'
        final stackTrace = commonFunctions.GetStackTrace(ex);
        println stackTrace;
        throw ex
    }
    finally {
        commonFunctions.SendBuildReport('mail@mail.com')
    }
}

Future Improvements

As always in software, there is still plenty of room for improvements. On the PowerShell side, the focus is on leveling up the security of the pipeline, by introducing detection mechanisms for security issues such as InjectionHunter, and automating the generation of Markdown documentation, based on the comment-based help for the cmdlets using PlatyPS. From an infrastructure perspective, we want to start leveraging docker and containers to build and release the modules to reduce the dependency on specific VM maintained for PowerShell modules in our build infrastructure.

Conclusions

Thank you so much for reading this post. It was longer than I originally expected, but I hope that anyways you liked reading it as much as I did writing it. Stay tuned for more!!

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer’s view in any way.