The last process in this Image Pipeline series is getting OVA’s into LOCAL Content Libraries across our environment. To do this I compare the latest list of OVA’s in the Artifactory generated array via my previous post to what’s currently in the Content Library. Then I have to add the OVA if it doesn’t exist or check to see if it needs updated if it does exist and update it if it does.
As with any product, there’s new features and API’s available with every release. When we originally implemented this process we were still on vCenter 6.5 because the version of another product we use had interoperability limitations with 6.7. Until it was upgraded we couldn’t upgrade all of our vCenter Servers to 6.7.
I’ll split the import process up into two posts for a couple reasons. First, as stated above there are people who can’t upgrade for various reasons. They may still need this. Second, 6.7 allows imports directly from HTTP/S sources so I want to show both methods.
In this post I’ll cover:
- Checking for and getting Content Library info
- See if an OVA exists within the Content Library
- How to decide if a Content Library Item needs updated
- How to get OVA into Content Library
- Problems with this method and versions
As a refresher, I setup this script to run for each Content Library of type LOCAL, which then replicates to other Content Libraries in that network space. The script starts with the last post where it queried Artifactory for a list of all OVA’s, filtered them for the latest OVA’s with the same image.name property, and stored them into an array. We now need to process this list against the Content Library.
As with any script, there’s environment setup and vCenter connections made. There are many posts with this information. A few other things I needed to do before getting started are:
- Space to download OVA’s to before importing to vCenter. For this I created a 50GB drive on the scripting VM.
- Check to make sure there’s enough free space on the drive above.
- Disable First Run Customization for Internet Explorer.
- Invoke-WebRequest won’t run until completed or reg entry applied.
# Regedit to disable Internet Explorer First-Launch mandate
$registryPath = "HKCU:\Software\Policies\Microsoft\Internet Explorer\Main"
$Name = "DisableFirstRunCustomize"
$value = "1"
IF(!(Test-Path $registryPath)) {
New-Item -Path $registryPath -Force | Out-Null
}
New-ItemProperty -Path $registryPath -Name $name -Value $value -PropertyType DWORD -Force | Out-Null
- Connect to vCenter REST API via PowerCLI
- Connect-CisServer
As a reminder, I won’t be posting scripting logic or basics. If you have questions about how to integrate these sections into your script hit me up on Twitter.
Checking for and getting Content Library:
This part was pretty easy thanks to William Lam and his PowerCLI Example Scripts.
The first thing I need to do is see if the LOCAL Content Library exists, which happens to be the first function in William’s list. We’ll also want to stop this script if the library doesn’t exist.
Note: I’m using the global FQDN vCenter variable and our naming standards to get the Content Library name.
# make sure Content Library exists
$library_ID = (Get-ContentLibrary -Name $LibraryName).ID
IF(!$library_ID){
write-host -ForegroundColor red "$(get-date -format g): $($LibraryName) does not exist. Exiting process."
Exit
}
See if an OVA exists within the Content Library:
Now that we know the Content Library exists and we have its ID, we compare its contents against my list of OVA’s stored in the array variable $latestartifacts. In the final script I would loop through the array but for this example I’m going to step through a single item by assigning variables as needed.
Let’s use the OVA listed in the first row off the array $latestartifacts[0] to see if it exists in the Content Library. I first have to set some variables from the array and then compare against all items within the Content Library.
### See if Content Library Item exists
# Get image.name value from array Properties value
$clitemName = $latestartifacts[0] | Select-Object -ExpandProperty Properties | Where-Object {$_.key -eq "image.name"} | Select-Object -ExpandProperty Value
$existingClItem = Get-ContentLibraryItems -LibraryName $LibraryName -LibraryItemName $clitemName
If the $existingClItem variable is NULL then I need to create a new Content Library item. If it’s populated then I need to update it.
How to decide if a Content Library Item needs updated:
If the OVA does not exist in the Content Library I jump to the next section. If it does exist, I need to see if it needs updated. To do this there must be a way to compare current Content Library items against what is in Artifactory and a source of truth for this value.
I was originally adding date stamps to the names of VM’s built with Packer and trying to use that. But we didn’t want those date stamps on the Content Library item because that was making it hard for the Automation Team to know which one to deploy from (and couldn’t use tags b/c of 6.5 issue). It was also a mess because I couldn’t force all teams using the process to have the same format and coding for all of them was terrible!!! There had a be a better way.
On the Artifactory side I discovered that an item uploaded gets the modified date property created and/or updated. This property stays the same when replicated to other Artifactory peers, even when replicated across the globe. That works for a source of truth.
On the Content Library side I needed a way to compare against the Artifactory modified property. I started down the road of setting the modified date when creating or updating the item via the Content Library but I couldn’t stop someone from making a change via the UI, which changed the modified value. I tried removing rights for people to do this but there were reasons why someone might have to update them manually (revert to old image, delete images, etc.)
I then decided to add the Artifactory object Modified date to the Description/Notes field (depending on what UI/API used). While testing this worked great because I could use PowerShell to compare the dates and if someone deleted or changed the item description to just text the compare process would re-update the item, which updated the field with the proper date.
This had even more benefits. Anyone can look/query items in a Content Library and know what date objects were created. The automation team can also copy that date to the Notes section of a VM deployed from that template to keep track of where the VM was sourced.
Here’s what library items look like once imported.
Let’s test the example we are working on to see if the existing Content Library item is newer than what is currently in Artifactory. Below I am comparing the Artifactory modified date to see if it’s greater than the date in the Description field of the Content Library item. My script will then update the item if it needs to or complete the loop for this Artifactory item and continue to the next one.
If($existingClItem){
# Check to see it artifactory item is newer. If so, update it.
If((get-date $latestartifacts[0].modified) -gt (get-date $existingClItem.Description)){
Write-host -ForegroundColor yellow "$(get-date -format g): Artifactory Image: $($clitemName) exists and needs updated."
} else {
write-host -ForegroundColor yellow "$(get-date -format g): No update needed for $($clitemName)"
}
}
How to get OVA into Content Library
Now it’s time to get the OVA’s into the Content Libraries. As stated above, with vCenter 6.5 and non-PowerCLI 11.5 this will be a multi-step process. Also, with this process and PowerCLI 11.5 cmdlets, testing will need to be done to compare import times against replication times.
If an item update tries to run while Content Library replication is going on (or vice versa), OR a VM deployment from the Template is running during update – it will lock up both processes. See the bottom of this blog for more info on this and other issues.
The following steps will be taking to complete this phase:
- Download OVA to scripting VM
- Import VM into vCenter Cluster
- Clone VM to Content Library Item
- Clean up VM and OVA
Download OVA to scripting VM:
This step is pretty straight forward. There’s 3 ways to do this within PowerShell and they all have pros/cons (listed well in article). Take your pick for what works to you. Main thing is to generate a variable for the path that the OVA is downloaded to. You’ll need for multiple steps later. I’ll use $ovaPath for this.
Import VM into vCenter Cluster:
There’s a ton of posts on how to use PowerCLI to import VM to vCenter. A couple things to keep in mind when doing this “at scale” are:
- Import-to clusters will be of different types and sizes.
- Getting OVA vNIC connection info so import works properly.
Our main script config has a variable ($cluster) telling which vCenter cluster to do the import to. Based on that variable we automatically choose a host, datastore, and PortGroup to import the VM to. Two things to notice below. One is the datastore variable checks for availability, not-local, and not some types of datastores. The other is the Network variable checks for VDS first and if it can’t find one it’ll map to a Standard Switch PortGroup.
$vmHost = get-cluster $cluster | Get-VMHost | Where-Object {$_.ConnectionState -eq "Connected"} | sort MemoryUsageGB | Select -first 1
$vmDatastore = Get-VMHost $vmHost | Get-Datastore | Where-Object {$_.FileSystemVersion -lt 6 -and $_.State -eq "Available" -and $_.Name -NotLike "local*" -and $_.Type -ne "VVOL"} | sort FreeSpaceGB | select -Last 1
$vmNetwork = Get-vmhost $vmHost | Get-VDSwitch | Get-VDPortgroup | sort VMs | select -first 1
if(!$vmNetwork){
$vmNetwork = Get-vmhost $vmHost | Get-VirtualPortGroup | Where-Object {$_.VLanId -GT 0 -and $_.Name -notlike "vmotion"} | select -First 1
}
Since we were using OFVTool to export our Packer builds the OVA vNIC was still set to connect to a specific PortGroup. When importing via the UI it sees this and allows you to update it. When using PowerCLI, you will get errors unless you find the existing connection info and map it to the existing $vmNetwork variable above. Here’s how I auto handle this for all of our VM’s.
$ovaconfig = Get-OvfConfiguration $ovaPath
$ovaconfig.NetworkMapping.($ovaconfig.NetworkMapping | Get-Member -Type CodeProperty | foreach {$_.Name}).value = $vmNetwork
After that it’s just the Import-vApp cmdlet with parameters above.
Clone VM to Content Library Item:
Cloning the VM to the Content Library was a big challenge. It’s easy to do via UI but I couldn’t find a good way of doing it via PowerCLI. I finally came across a post by “Stuart” with a function for what I needed. It uses the REST API to complete this, along with REST API calls for Content Library lookup as well (wish I had that earlier!!!).
The function shows how to ADD a new item to the Content Library but not UPDATE an item. For this I added another parameter to the function ($item_ID), that isn’t mandatory, and the line below is added to the com.vmware.vcenter.ovf.library_item config spec. Add logic to see if the parameter value exists. If so, then it’s added to the config spec. If not, we use what’s in the post above.
$createOvfTarget.library_item_id = $item_ID
The only other deviation I did from this post is that I update the Description field AFTER I’ve verified the item updated successfully. If you submit something to the Description field at the same time it will update that field at the start of the process and not revert if the import/update process failed.
I do this by getting the current Content Version of the item first via $existingClItem.version from above, then checking that against the value once the Clone to Content Library step is done.
Clean up VM and OVA:
Cleanup is pretty easy. You’ll get the VM you imported above and pipe that to Remove-VM to clean it up. Make sure to add -DeletePermanently parameter or the files will hang around and fill up your datastore. Then run Remove-Item $ovaPath to clean up the downloaded OVA.
Problems with this method and versions:
One hiccup I ran into that I want to call out. Our automation team wants to utilize tags instead of relying on object names. While testing in our lab, I found that we can tag Content Library items in vCenter 6.5 but doing so breaks replication. As soon as tags were remove, replication works again. Tagging works fine in 6.7, though.
Another issue with this method is the Clone to Template in Content Library method will hang if there’s an active deployment or Content Library replication going on. I briefly talked about this in the Content Library setup post. To get around this I had to time imports and change replication schedules so that the chance of collision is minimal.
Description field must be updated after verifying the add/update process succeeded. If updating in one step, this gets updated right away and doesn’t revert if the add/update fails. I need to check against this so I want to make sure it succeeds first.
Now that was a long post. There’s roughly 500 lines on PS code to run this.
Next I’ll show how to do this in vCenter 6.7, which allows us to bypass downloading the OVA to the scripting VM AND importing to a cluster first. This uses more lines of code but there’s a tremendous benefit to it. Stay tuned to find out what that is!!
vEric: 
Leave a Reply