VMware cloud account validation failed – alternative workaround on KB88531

Recently we had an issue with the validation of a VMware cloud account in vRA. The validation didn’t work.
A colleague at the customer site found the correct VMware KB artikel that addressed this issue ( it is KB article 88531)
This post is about an alternative approach for the same workaround mentioned in the KB article. This is not an in-depth article. You should have knowledge about vRA and using REST API calls.

Issue

The issue is that the certificate info has not been stored in the cloud account.
And this can happen when the vCenter SSL certificate is renewed, and in vRA you accept the new certificate, but you didn’t hit the ‘SAVE ‘ button.
When accepting the new certificate, you stored it in the certificate store of vRA. But because you didn’t hit save, that info wasn’t stored with the cloud account registration.
According to the article this has been solved in vRA 8.9 and for prior versions there is a workaround

Workaround

The workaround is correct. But what I don’t like about it, is that there is no explanation about what you are doing.
The workaround uses the REST API interface of vRA to store the correct certificate with the cloud account. And if they would have mentioned that, maybe the reader would think… wait…. REST API…. can I use the swagger interface…..

Yes you can.
The API calls we need are:

  • GET /iaas/api/cloud-accounts
    To find the cloud-account-id of the vCenter Cloud Account we are going to update
  • PATCH /iaas/api/cloud-accounts/{cloud-account-id}
    To store the new certificate information with the vCenter Cloud account

Alternative

  1. Store the vCenter certificate (including the chain) as a PEM file.
  2. Go to the swagger ui (which can be found at {root-url}/automation-ui/api-docs/
  3. Go to the ‘Infrastructure as a Service section
    In this section you will find the API calls we need.
  4. Authenticate with vRA using a Bearer token.
    (Tip: you can get a Bearer token using the REST API calls on the swagger ui and/or create a vRO action to get a Bearer token)
  5. Search for the cloud account id by using the ‘GET /iaas/api/cloud-accounts’
  6. Convert the PEM file to a single line where the line ends are replaced with \n
  7. update the cloud account with the new certificate information using the ‘PATCH /iaas/api/cloud-accounts/{cloud-account-id}? apiVersion=2021-07-15’ API call.
    (note: using the url parameter apiVersion is crucial)
    You should get a HTTP status code of 202 for confirmation.
  8. run the validation of the cloud account in vRA, it should work now

invoke-command instead of invoke-expression

In my previous post (which you can find here ) I use the invoke-expression cmdlet for running a Powershell script which was downloaded with invoke-webrequest.
And this was a good solution. The code that was downloaded and executed was a powershell script that would run a private function. This private function then was formated with the three scriptblocks Begin,End and Process.
Parameters where with a same construct being downloaded from a git repository and placed in a powershell Object called $P.
With this approach I separated code parameters from the actual code.
Using GIT I was able to versioning my parameters file, separate from my script code. This setup is working great. And it gives flexibility by leaving the code untouched when changing parameters.

But…

Yeah, a but…. I still needed a way to pass parameters /arguments on the command line. Using invoke-expression… well that wasn’t possible.
So I looked into invoke-command, which has an -argumentlist parameter., making it possible to pass one or more arguments to the script. Using named parameters isn’t possible, which is not what I was looking for.
So to support naming parameters, I decided to introduce just one parameter. And this parameter should be a JSON string, making it possible to pass multiple parameters merged into a JSON object.

The only challenge with this is that all the interpreters that the code was going to pass, should leave the JSON string intact, including the quotes. And I didn’t want to escape any quotes. That would be messy and prone to errors. But encoding it, should solve this issue. The argument is a base64 encoded JSON string.

Is it secure?

Well, No …. not at all… it is a base64 encoding. My goal was not to make it more secure, but that the string wouldn’t be changed by the different shell interpreters.
Off course, you can make it more secure by using private/public key-pairs. You could use a docker volume containing the encoding keys, or other secure methods. When using base64 coding, just don’t pass any sensitive data (passwords) with it. There are other, more secure, approaches for this with containers.

parameter approaches

This setup gives me different approaches to pass parameters to the script. The more static parameters are stored in a .json file, stored in a GIT repository.
And the more dynamic parameters (like VM names to start), are passed via the base64 encoded JSON string.

What changed ?

I changed to following items:

  • changed entrypoint string
    • using invoke-command instead of invoke-expression
    • placing invoke-webrequest inside a scriptblock
    • using argumentlist to pass a base64 string, encoded a JSON string
  • changed powershell wrapper script to decode inpu

Docker Entrypoint

The previous docker entrypoint was something like

pwsh -Command invoke-expression '$(Invoke-WebRequest -SkipCertificateCheck -uri ' + <git URI> + ' -Headers @{"Cache-Control"="no-store"} )'

The new entrypoint is looking like

pwsh -Command invoke-command -scriptblock ([scriptblock]::Create( (Invoke-WebRequest -SkipCertificateCheck -uri <git URI> -Headers @{"Cache-Control"="no-store"} ).content ) ) -ArgumentList <base64 coded JSON string>

As you can see, the one-liner has grown.
I used the -scriptblock and the -ArgumentList parameter from the invoke-command. The -scriptblock contains the Invoke-webrequest cmdlet which downloads the RAW version of the powershell script on the GIT repository.
The invoke-command cmdlet then executes this scriptblock and passing the argument from the argumentlist to this script.

Script Layer

The script has a wrapper layer, a main layer (containing the Begin,End and Process blocks) and the Process block containing the specific code to run.

<#
.SYNOPSIS
    template.ps1 powershell
.PARAMETER inputObject 
    A JSON string base64 (UTF-8) encoded
#>

param(
    [string][Parameter(
        ValueFromPipeline = $true, 
        ValueFromPipelineByPropertyName = $true,
        HelpMessage="JSON string base64 (UTF-8) encoded.")]$inputObject=""
)

function main {

}
#-- calling the real powershell code to run
main

main layer (function)

I choose to use the function method to preserve my code format structure. For most of my powershell code I use the End,Begin and Process scriptblocks to structure the code. And I didn’t want to stepp down from that approach.

function main {
    <#
    .SYNOPSIS

    #>
    Begin{
        $uri = <url to RAW version of parameter file>
        #-- trying to load parameters into $P object, preferably json style
        try { $webResult= Invoke-WebRequest -SkipCertificateCheck  -Uri ($scriptrootURI+$scriptName+".json") -Headers @{"Cache-Control"="no-store"}  }
        catch  {
            write-host "uri : " + $scriptrootURI
            throw "Request failed for loading parameters.json with uri: " + $webResult 
        }
        # validate answer
        if ($webResult.StatusCode -match "^2\d{2}" ) {
            # statuscode is 2.. so convert content into object $P
            $P = $webResult.content | ConvertFrom-Json 
        } else {
            throw ("Failed to load parameter.json from repository. Got statuscode "+ $webRequest.statusCode)
        }

    #-- private functions
        function exit-script {
            ...
        }

    #--- proces inputObject (the argument passed via the cmd line
        # decode that inputObject as UTF-8 base64 and convert it to a powershell object
        $A= ConvertFrom-Json -InputObject ([System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($($inputObject)))) -ErrorAction SilentlyContinue -ErrorVariable err1
        if ($err1) {
            writ-host "Failed to proces input object"
            exit-script
        }
    }
    End {
        exit-script -exitcode 0
    }
    Process {
        #-- the code that is doing the real work
        write-host ($P.world) #-- from the parameter.json file
        write-host ($A.universe) #-- passed via the cmd line argument
    }
}

Final

So I hope this blog gives you some ideas with you code challenges.
I’m going to write a more structured set of articles , a deep dive into my FaaS – like setup. So keep following this blog, when interested.
And comments, are always welcome.

Dynamicly executing powershell code from GIT – a FaaS way

In this post I mentioned that I was tipping my toes in vRA.
And…. yeah, it is not dipping anymore, it is a deep dive… 🙂 but that is for another blog

This blog is about a challenge I solved by using a FaaS approach. FaaS stands for Function as a Service.
I have a small UPS running for my home environment. And found out that it could support my home environment for about 30 min. And then poof… power off…. So I was searching for a solution that my vsphere environment would be shutdown, triggered from a home assistant /Node-Red automation flow, monitoring my UPS status.

My first way of thinking was to use the REST API of vSphere / VMware esxi. But not all the actions I need are published. (Like shutdown of an ESXi host.)
And I want to be dynamic as possible.
I want to shutdown / suspend all VMs expect VMs that are tagged as coreVM. So the code that I should write, doesn’t contain the VM names or IDs, but would filter out all VMs that have a vSphere tag with coreVM.

The Idea

As mentioned, is that the automation would find all VMs that need to be suspended or shutdown ( depending if VMtools is running). And VMs that have a vSphere tag UPS/coreVM, would be ignored.

These VMs are controlled with the startup/shutdown feature of my ESXi host.
The total automation flow would be a 3 fase flow. Fase 1 shutdown all but the core VMs. Fase 2 shutdown the ESXi host, which would automaticly shutdown VMs controlled via the startup/shutdown feature. Fase 3, shutdown of Synology NAS and Home Assistant
The core VMs are my vCenter, Log Insight and router VM.

The tools

The tools I use are:

  • Home Assistant with
    • NUT intergration (monitoring my UPS)
    • Node-Red flow automation (ussing HTTP REST API calls to control docker via Portainer)
  • Docker CE with Portainer CE running on a synology NAS
    • Docker image vmware/powerCLI
    • Portainer controlling the docker environment and leveraging control via its APIs
  • Gitea GIT server running locally (containing my script and parameter files)

the FaaS way

For fase 2 I try to use a FaaS approach. Meaning I have a function for shutting down / suspending VMs. This function is run in a temporary runtime environment (docker container) and only available at runtime. The function is not part of the runtime environment, but on runtime it is downloaded from a GIT repository and executed.
This gives the advantage of maintaining a runtime environment seperate from scripts (or the other way around). And on every execution, it starts with a clean environment.
For the function I use powerCLI, because I haven’t found an API call in vSphere 7.0 that will shutdown an ESXi host. And filtering VMs on vsphere tags was (for me) a bridge to far.

Using a container gives me also the possibility to seperate my vSphere credentials from my script, by saving it in a docker volume, which is only mounted at runtime. The function itselfs contains no credentials. It looks to a file in that docker volume.

The hurdles I encountered were:

  • how to run the script from a git repo
    • by using a powershell one-line invoke-expression with invoke-webrequest using the url to the raw presentation of a file (script) in the git repo.
    • using this one-liner as the entrypoint parameter when creating the container
  • how to bypass caching by the web browser
    • using “Cache-Control”=”no-store” in the header
  • running a full powershell script via invoke-expression
    • write the full code block as a powershell function (with begin, End and Process blocks) and call that function within the script.

The powershell one-line is

pwsh -Command invoke-expression '$(Invoke-WebRequest -SkipCertificateCheck -uri ' + <git URI> + ' -Headers @{"Cache-Control"="no-store"} )'

The <git URI> is the url to the raw representation of the script file in the GIT repo. When leveraging the portainer/docker api to create the container you need to use the JSON notation for the entrypoint. The one-line will look something like:

["pwsh",
"-Command",
"invoke-expression",
'$(Invoke-WebRequest -SkipCertificateCheck -uri ' + <git URI> + ' -Headers @{"Cache-Control"="no-store"} )']

Powershell script formant / the snag

The snag with using invoke-expression is that it can’t handle a powershell script that has a Begin, End and Process code block. While I this is how I write my code, and wouldn’t like to deviate from it, it meant I had a snag.
The solution was to write a powershel script that contains a private function and execution of that function, like this

function main {
    <#
    .SYNOPSIS
        Example code for running full script from an URI.
    .DESCRIPTION
        Example code to run a powershell script with dynamic blocks Begin,Process and End.
        Loading parameters from a JSON file into an P object.
        By wrapping the code into the function main, we can use begin, process and end scriptblocks when calling with Invoke-Expression
        The Process block contains the main code to execute.
        The Begin and en blocks are mainly used for setting up the environment and closing it.
    .EXAMPLE
    Run the following cmdline in a powershell session.
    Invoke-Expression (Invoke-webrequest <URL>).content
    .NOTES    
    #>
    Begin {
        #=== Script parameters

        #-- GIT repository parameters for loading the parameters.json
        $scriptName="startVMs"
        $scriptGitServer = "https://....."   # IPv4 or FQDN of GIT server
        $scriptGitRepository = "organisation/repo/" # uri part containing git repository
        $scriptBranch = "master/" # GIT branch
        $scriptrootURI = $scriptGitServer+$scriptGitRepository+"raw/branch/"+$scriptBranch

        #==== No editing beyond this point !! ====
        $ts_start=get-date #-- Save current time for performance measurement

        #--- write log header to console
        write-host
        write-host "================================================================="
        write-host ""
        write-host "script: $scriptName.ps1"
        write-host ""
        write-host "-----------------------------------------------------------------"

        #-- trying to load parameters into $P object, preferably json style
        try { $webResult= Invoke-WebRequest -SkipCertificateCheck  -Uri ($scriptrootURI+$scriptName+".json") -Headers @{"Cache-Control"="no-store"}  }
        catch  {
            write-host "uri : " + $scriptrootURI
            throw "Request failed for loading parameters.json with uri: " + $webResult 
        }
        # validate answer
        if ($webResult.StatusCode -match "^2\d{2}" ) {
            # statuscode is 2.. so convert content into object $P
            $P = $webResult.content | ConvertFrom-Json 
        } else {
            throw ("Failed to load parameter.json from repository. Got statuscode "+ $webRequest.statusCode)
        }
    }
    End {

    }
    Process {

    }
}
main

What it does, is the script will run the function main. That function is basicly the full powershell script. In the begin code block the invoke-Webrequest is used to load a .json file and convert it to a powershell object called P.
This object contains all parameters used in the rest of script (like vCenter FQDN).

Result

The result is that from a monitoring trigger Node-Red will do some REST API calls to portainer to create run and delete a docker container based on a vmware/powerCLI image. During the lifetime of this container a volume is mounted with authorisation information and a powershell one-liner is executed which runs a powershell code directly loaded from a GIT repository.
With this setup I can run on demand any powershell script which doesn’t need user interaction, maintained in a GIT repository.

I hope you enjoyed this post. do you have questions / comments, please leave them below or reach out to me on twitter

IFO: remote-ssh VSC on photonOS

I’m using visual studio code a lot. It has a lot of extensions that will make your life a bit easier. And one my of my top favorites has become remote-ssh.

With remote-ssh you can use VSC on a remote server/VM. It uses the ssh protocol to connect to the remote server, install the remote component, and you can use the remote system as a local system.

What am I using it for ?

Well, I have a homelab to get accustomed to and explore VMware software. I use it to fiddle around with software solutions and get accustomed with solutions like GIT, Docker, Ansible and Kubernetes. And all those solutions are text based… Yes off course there are GUI shells for these solutions, but that will not help you to get sufficient with these solutions.

So you need an IDE, and my choice is VSC with remote-ssh
As a VMware fan boy, I like to use photonOS for my linux VMs.

How to configure photonOS for VSC & remote-ssh

I make the following assumptions:

  • it is a test/dev environment (we logging in as root) not a production environment
  • tdnf updateinfo and tndf -y update has been run
  • you have internet connection with the photonOS VM

To make remote-ssh work with photonOS you need to do these things

  1. install tar tdnf -y install tar
    Remote-ssh uses tar to extract its remote server software
  2. edit sshd_config at /etc/ssh/sshd_config and set the following settings
    1. PermitRootLogin yes
    2. AllowTcpForwarding yes
  3. (bonus / optional) Add your public SSH key to the <user>/.ssh/authorized_keys file.
    When using root to login, the location is /root/.ssh/authorized_keys
    else it is /home/<username>/.ssh.authorized_keys

See also

IFO: allow apps downloaded from anywhere on mac OS

.

So, Apple is big on security. Which is a good thing.
But sometimes, it is too strict.
I’m busy remodelling my homelab, and one of the actions is reĂŻnstalling a clean vCenter appliance.
And I thought let’s do it from the CLI !!!


Yeah…. so I ran vcsa-deploy and got the error that the app is downloaded from the internet and not to be trusted.
So you can allow it via the system preferences, but the MAC OSx gatekeeper keeps irritating you with all the warnings about the libraries that are loaded

After some googling around I found this site
3 Ways to Allow Installation of Apps from Anywhere in macOS Catalina (techsviewer.com)
And the cli option to allow apps downloaded from anywhere was winking at me. Yes that was the option I wanted. So even though it was for mac OS Catalina, why not try it for macOS Big Sur.

And it worked… to allow vcsa-deploy to function properly just do the following

  1. open terminal
  2. execute the command : sudo spctl –master-disable
  3. goto system preferences -> Security & Privacy
  4. tick the ‘anywhere’ option under Allow apps downloaded from:
  5. Run vcsa-deploy

Well to be security aware, the best practise is to remove the anywhere option, just follow these steps

  1. open terminal
  2. execute the command: sudo spctl –master-enable

and your done.

Making these changes is (off course) at your own risk.

IFO: max. lifetime SSL cert is 1 year

Yes, certificate missery.
In the wisdom of great corperations, for our safety, it is deiced that the maximum SSL/TLS certificate validity is one year.
Yes, really… don’t believe me, just check this search for it
ssl lifetime 1 year at DuckDuckGo

From the security side of things this is a good thing. Because it mitigates the risk of a hacked certificate.
But from an administration point of view….. HEADACHE.
Especially for certificates that are used internally in your production sites.
Now you need to replace the certificates every year for your servers.
At least for those servers that run web services because your browser is going to nag you that the site isn’t safe anymore. Yes another warning.
And you know what happens to warnings, in the end they will think for you, and won’t allow you to access the website anymore.

so SSL certificate monitoring becomes more importent, and having a plan / replacement strategy for SSL certificates would be a good thing to have.
Do you have an up-to-date overview of all the SSL certificates in your network ?
Maybe a good idea to have it up-to-date and monitor them.

truncate docker container log

sometimes you just want to clear logs of a docker container.
For instance, I’m running dnsmasq in a docker container and I’m troubleshooting raspberry PI PXE boot (yes to run ESXi-ARM stateless…)
And the dnsmasq is serving my homelab its domain.
Then it can be an annoyance when you run docker logs <container name> seeing all the log entries since the container started.
And you just want to start with a clean slate.

It is the way

Yes, there is a way. And when you google you’ll find more blog posts giving you the solution. Which is

pi@raspberrypi~ $ sudo docker inspect --format='{{.LogPath}}' <container name>
pi@raspberrypi~ $ sudo truncate -s 0 <path presented by previous command>

And it works great… but… typing 2 lines of code… copy / pasting the log path in the second command.. way too dificult 🙂
So what do you think abou this ?

pi@taspberrypi~$ sudo  truncate -s 0  $(docker inspect --format='{{.LogPath}}' <container name>)

Yes, it will erase all logging, but that is the purpose of this whole excercise.

Addendum

If the one-liner isn’t helping, how about creating a small script just for one purpose alone…. What if you could do something like

pi@raspberrypi~$ truncate-log dnsmasq

To truncate the docker logs of the dnsmasq docker container. How ? Well easy.
Just create a new file in /usr/local/bin

pi@raspberrypi~$ nano /usr/local/bin/truncate-log

and copy / paste this content into the file

#!/bin/sh
CONTAINER=$1
truncate -s 0 $(docker inspect --format='{{.LogPath}}' $CONTAINER)

After saving the file. (Ctrl-x), add the execution bit

pi@raspberrypi~$ chmod +x /usr/local/bin/truncate-log

And voila, the next time you need to truncate a docker container log, you just type truncate-log <docker container name>

rPI adventures bits III

Some of the default installations have a graphic desktop. But, when you use your rPIs headless, what’s the point of these interfaces.
Of course you can remove them, but you can also use remote connections.
And then you have by default VNC, for which you need to start the service first.
But a RDP service, wouldn’t that be usefull, and is it possible ?
Yes it is.

XRDP

It is called xrdp.
xrdp is an open-source remote desktop service for linux. And ues, you can run it on a rPI. To find more inf about xrdp check out their site http://xrdp.org/

Install XRDP

To install xrdp on a rpi run the following steps. (assuming you are running raspion OS

  • login to the rPI or with a terminal or via ssh
  • update software
pi@raspberrypi:~$ sudo apt update
pi@raspberrypi:~$ sudo apt full-upgrade
  • install xrdp
pi@raspberrypi:~$ sudo apt-get install xrdp
  • start a RDP client and connect to the rPI

And easy as pi 🙂

Purpose

Well, for me it is a stepping stone into my homelab.
I have rPI for running dns,tftp,dhcp,samba,http,ntp, unifi controller services in docker containers.
With this setup I can control my homelab from this stepping stone. And also acces the IPMI interface of my supermicro server, hosting my vSphere homelab.

rPI adventures bits II

ssh access on first boot

I’m using my rPI headless. Meaning no monitor, mouse and keyboard. Just an ethernet connection.
And that is great, but then you need to have ssh access.. and by default that is not running.
There are some small steps to have ssh running on boot, these are the steps
(assumption is that your network has a DHCP service running)

  1. insert SD card into your Windows / MacOS / Linux system
  2. create a file named ‘ssh’ on the root partition.
  3. Insert the SD card into the rPI
  4. boot the rPI
  5. check your DHCP service log for a new created IP lease
  6. SSH to the IP found in the previous step
    Default username: pi, password: raspberry

rPI adventures bits I

Yes, I’ve stept into the realm of raspberry pi.
Adding this to my homelab setup.

For now I’ll just scribble some stuff I need to remember. Later on I’ll write detailed blogs about the setup of my homelab.

Temperature check

rPI’s are getting hot. Especially the rpi 4B+.
To check it’s core temp you can run the following command

root@raspberrypi:~# /opt/vc/bin/vcgencmd measure_temp

To make life easier I created a small script called ‘temp’ and placed it in /usr/local/bin.

#!/bin/sh
/opt/vc/bin/vcgencmd measure_temp

Also make sure it is executable.

chmod +x /usr/local/bin/temp

Now you can check the temp by running the command temp.

root@raspberrypi:~# temp
temp=47.8'C
%d bloggers like this: