Large Scale Image Pipeline – Part 3 – Query Artifactory via PowerShell

For the next part of this Image Pipeline project, I needed to search Artifactory for all OS Image artifacts within a repo that have a specific property key, filter for the newest artifact with the same key value, download the items, and then upload OVA’s to vCenter.

When working on large automation projects, there’s always a couple challenges that pop up. Usually one of these issues stems from the lack of examples to complete a step with the selected scripting language. This was the case with using PowerShell to search Artifactory.

Searching Artifactory is easy to do with CURL, especially with all of the examples on the JFrog site, but it was challenging to complete using PowerShell for a couple reasons. I’m using PowerShell for this process because we already have Windows scripting servers utilizing PowerCLI to pull data from all of our vCenter Servers, roughly 40, so there’s no additional infrastructure or setup needed.

The first challenge is that Artifactory Query Language (AQL) requires authentication. It supports multiple types of authentication, such as basic or Tokens, but for our deployment model I need to utilize basic username and password. With CURL this is easy to do as basic authentication works via “-u username:password” but using Invoke-WebRequest in PowerShell this won’t work. To get this working I had to encode the credentials via Base64.

After retrieving proper username and password (which are stored in encrypted files) they are encoded via the following:

# Base64 Encode Artifactory Credentials
$artpair = "$($ArtUser):$($artPass)"
$artencodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($artpair))
$ArtifactoryAuthValue = "Basic $artencodedCreds"

 

The next challenge is sending the query data itself. With CURL this is done either by sending a file with the query via “-T filename.txt” or “–data ‘query info'”.

When using Invoke-WebRequest query info will be put into the BODY of the request. It is in JSON-like format and sent as plain text. For this example I will search the “OS-images” repo for all artifacts with a property “image.type” and value of “vcenter_ova”.

# Build web request
$Body = "items.find("
$Body = $Body + "  {"
$Body = $Body + "    `"repo`": {`"`$eq`":`"OS-images`"},"
$Body = $Body + "    `"@image.type`": {`"`$eq`":`"vcenter_ova`"}"
$Body = $Body + "  }"
$Body = $Body + ").include(`"name`",`"repo`",`"path`",`"modified`",`"property`",`"actual_sha1`",`"size`").sort({`"`$desc`" : [`"modified`"]})"
$ArtUrl = "https://ArtifactoryAddress/api/search/aql"
$Headers = @{
    Authorization = $ArtifactoryAuthValue
}

 

I also want to return specific information for each artifact. This is shown above via the “.include” statement. I am asking for multiple items that are needed later in the pipeline process. With Name, Repo, and Path I have enough info to build a URL to download artifacts, if needed. This would be put together in the format:

$itemurl = "https://ArtifactoryAddress/" + $repo + "/" + $path + "/" + $name

 

The rest of the process is straight forward. Next, I put all request parts together and submit it to the Artifactory server. Returned info will be in JSON format and stored into a variable.

$webresponse = Invoke-WebRequest -Uri $ArtUrl -body $Body -method "POST" -ContentType "text/plain" -Headers $headers

 

Now that I’ve submitted the query, I validate artifact data was returned and, if so, convert it to an array so it’s easier to use within PowerShell.

# Check for returned data and convert JSON to an array
If($webresponse) {
    $response = $webresponse.content | ConvertFrom-Json
} else {
    Write-Host -ForegroundColor red "No Artifactory search responses returned."
    break
}

 

Our image pipeline process uploads every image version to Artifactory as a new artifact but with the same “image.name” key value. Because of this I need to get a list of every “image.name” value and then only keep the newest item with each value.

#  Get just the latest artifact for each image name
$artifactnames = $response.results.properties | Where-Object {$_.key -eq "image.name"} | Select-Object -expandproperty value -Unique
$latestartifacts = ForEach ($image in $artifactnames) {$response.results | Where-Object {$_.properties.key -eq "image.name" -and $_.properties.value -eq $image} | Select-Object -last 1}

 

I now have a unique array of Artifactory stored OVA items that can be downloaded and utilized in Part 4 our image pipeline.