Ansible in a (docker) container

Ansible Playbooks vs Roles: Part I - The Playbook ...

A year ago, after following a few sessions at the dutch VMATechcon I decided to dive into the world of Ansible. With a background in industrial automation (programming PLCs) I have a soft spot for writing code. I saw the benefits of declarative languages, and decided to dive into Ansible.
But where to start. And what should I use it for.

Image

I’m working on a small VM which provides DHCP,DNS,TFTP,FTP,HTTP and NTP services so I can easily (PXE) boot nested ESXi hosts on VMware workstation and my homelab. All running in docker containers on Photon OS.
For maintenance (and learning) I decided to use Ansible for building, running these containers.

One of the issues I had, was that I had to maintain the VM and the software installed on it… I’m quite proficient with Linux, but it can still be a challenge.
And keeping installations logs update about how to install software… yeah…. that is an issue. A embraced the solutions docker, git and ansible are giving me. (And vagrant, when using VMware workstation).

Using docker to run the services, using ansible to automate workflows and using git to keep track of changes and store my code in my github repositories.
With this approach I can easily rebuild the VM when it breaks down. And I see the VM more as a processing entity, then a virtual computer that needs to be maintained.

The Ansible container

browsing the internet I found this blog post about running ansible inside a docker container and using it interactively on the docker host. This gave me the possibility to have ‘portable’ ansible installation, without any ties to the VM OS.
In my opinion it is great. With adding some small bash scripts so I don’t need to write something like ‘docker run -rm -it …..’ everytime, I have an ideal solution.
The original code for the container is based on alpine:3.7 and python 2. Everytime I build the docker image I received a warning that python 2 is outdated…. so I (finaly) updated the dockerfile using alpine:3.11 as the base image and using python 3.
You can find my code here on github, and the docker image here on dockerhub.

How to use

simple. There are 2 ways to use it.

  1. clone git repository
  2. pull image from dockerhub

Option 1, clone git repository

I’ve created a small installation script to make things easier. It will run docker build to build the ansible image and it will copy the scripts under ./scripts/ to /usr/local/bin/
And your all set… well almost

Option 2, pull image from dockerhub

Instead of building the image, you can pull it from dockerhub into your local docker environment. The scripts assume that the docker image is tagged like ansible:2.9.0, but when you pull it from dockerhub, the name is brtlvrs/ansible-helper:v.0.3. So you need to retag them (or edit the scripts).

docker image tag brtlvrs/ansible-helper:v.0.3 ansible:2.9.0
docker image tag ansible:2.9.0 ansible:latest

The wrapper scripts are stored in the image, in the folder /2installOnHost
To copy them out of the image do the following

docker create -n test ansible:2.9.0
docker cp test:/2InstallOnHost/ah* /usr/local/bin/
chmod +x /usr/local/bin/ah*
docker rm test

This will create a docker container named test, using the imported image ansible:2.9.0.
Then copies the files from inside the container to the /usr/local/bin/ and sets the executable bit on these files.
Finally it removes the created docker container.
Now you should be able to run ah or ah-playbook.

Both scripts will mount the folowing volumes into the container

localmountpointremark
~/.ssh/id_rsa/root/.ssh/id_rsaneeded to access localhost via ssh (ansible)
~/.ssh/id_rsa.pub/root/.ssh/id_rsa.pubneeded to access localhost via ssh (ansible)
$(pwd)/ansible/playbooksMountpoint is the default location inside the container.
/var/log/ansible/ansible.logMaking the ansible logs persistent.

Finishing touch

To make it work you need to do the following:

  • install docker-py on the docker host ( if you want ansible to be able to control docker)
  • add the ~/.ssh/id_rsa.pub content to ~/.ssh/authorized_keys file, so you don’t need to store the user password into the ansible inventory file.

Final

An now it should work. Type ‘ah’ and you’ll enter a temporary docker container running ansible. Type ah-playbook and you’ll execute the command ansible-playbook inside a temporary docker container.
Have fun.

Custom Welcome DCUI screen

Recently I discovered you can change the default ESXi DCUI screen.
This feature is already present for a long time…. just check one of my favorite blog sites (virtualGhetto) for this item here. Yes he blogged about it in 2010….. I know

Why would you want to change it

  1. Your Company wants a warning shown on the DCUI
  2. Security, don’t show hostname, ip etc…
  3. showing off ASCI-ART – just for fun
  4. ….

For me, it was point 3….
I run on my laptop VMware Workstation and am sometimes in need of spinning up nested ESXi with different kind of versions. For this reason I build a provisioning VM, so I can network boot empty VMs and choose from an iPXE menu the ESXi version I want to work with. (maybe some interesting stuff for a blog serie) And this setup wasn’t finished with a custom DCUI screen.

How does it work

Until version ESXi 7.0 there is a text file /etc/vmware/welcome, which you can edit. When it is succesfully edited, the DCUI wil show the result.
My screen shows: VMware product and version, license, Memory, IP, SSL Thumbprint and ASCI-ART ūüôā

You can also edit the logonbanner, shown when you access the host via SSH. And the support info

Until ESXi 7 ?

Yes, until… well almost. Until version 7.x you just could simply edit the file /etc/vmware/welcome. But it isn’t there anymore.
But there is a different way. Called CLI. You can edit the welcome message with esxcli and powercli. Information is found in the VMware knowledge base kb2046347. And this works for a lot off ESXi versions. I’ve tested it succesfully from 5.5 to 7.0.

ASCII-ART

Well, you can show your creative side. Some guidelines I found are:

  1. screen with is 125 characters
  2. max. rows about 25
  3. check for ASCI TAB character, ASCII nr 9
  4. use a site for creating art like this one

Especially number 3 is a killer. A lot of text editors will replace a certain amount of spaces with a TAB character… and this will mess up your screen.

ESXCLI method

The syntax is easy, just

esxcli system welcomemsg set -m 'your welcome'

And that is what is shown in the kb article. But how do you create a complete screen. Well, there are some undocumented parameters you can use. The article by William Lam (see above) shows some of them. But I haven’t find a site with all of them.
With those parameters you create in a text editor your screen…. like this one

Every line that uses a different background color then black is 125 characters long, not including the length of any fields that are used, like {color:white}.

After you created your file, you can place it in /etc/vmware or use esxcli to edit the welcome message.
If the file is placed in /tmp/welcome.txt then your instruction is like

esxcli system welcomemsg set -m "$(cat /tmp/welcome.txt)"

And voila, you have a custom welcome screen.

DCUI with SSH

The DCUI is not only shown on the console screen of ESXi, but also accessed via SSH by >dcui
The result is shown below.

DCUI screen when accessed in SSH

Enjoy.

migrate to vcsa 6.5 U3 GUI xlarge issue

For a customer I’m migrating their legacy vSphere 5.5 environment to vSphere 6.5 U3.
The migration is from a windows vCenter to the VCSA.
First, we tried to use the GUI and stumbled over an issue.

GUI

The GUI, is of course a nice, friendly way to do this process. But when we got at the stage to select the size of the VCSA, we only had the ‘xlarge’ option.
And well… that was a bit too much. Because we were aiming at the ‘small’ size.

So we did some searching on the internet and found out that we were not alone. Although it was an encouraging thought, we still didn’t find what we were looking for.
Most of the threads and blog posts pointed to database size, log size etc..
And yes, they need to be checked. We had a vcenter database table of 7 million records, the size of the database was 80 GB. After some cleaning up and shrinking the database it was only a few GB.
But al this effort, didn’t persuade the GUI to give us the desired ‘small’ size option.

CLI

Hoping that the validation in the GUI was different than when using the CLI, we decided to migrate using the vcsa-deploy CLI method.

Yes, it involves creating a json file. But the .iso file for the vcsa contains several template files, for several migration scenarios.
We used on of the templates, customized it to our needs (setting the size to ‘small’. And after some trial and error we finally got it working.

TIP: validate your json file with

vcsa-deploy --precheck-only \some_path_to_json_file\migrate.json

If you want to read more about the CLI way of migration, check the vmware docs here.

vcsa-deploy creates for every run a new log folder . When we checked the logs we found out that vcsa-deploy was content with the ‘small’ size option we configured in the json file.

Thoughts

Why not 6.7 U3, well due to dependencies 6.5 U3 is at the moment the most current version we can run.
Although this post is about migrating to 6.5 U3, it could also work for 6.7 U3, but no guarantees.
The gist of this all is, if the GUI doesn’t work, try doing it the CLI way.

TIP: Secure your My VMware account with MFA

I know, it is not the most interesting subject to blog about.
VMware gives you the opportunity for your My VMware account to use MFA.
And I would advise to use MFA.

Why ?

Well, depending on the situation, your my vmworld account can have access to different company accounts. Giving you, depending on roles and permissions, the ability to create support requests, download software, access licenses etc…
And that is great.
But it is a risk when your account gets hacked.
That is where MFA can help. It is an extra line of defense.

MFA

MFA stands for, multi factor authentication. And when MFA is enabled,a person needs to present multiple pieces of evidence to authenticate (see https://en.wikipedia.org/wiki/Multi-factor_authentication )
Most of the times these pieces are a password and a token, generated by a authentication app or device.
So, when you would know my password, you still can’t use my ‘my vmware’ account because I’ve enabled MFA. And the token is generated on my phone, which I have with me, and is locked.

My advise

  1. Use a strong password. Strong doesn’t mean a lot off difficult characters, although there are some rules you have to follow. Study has shown that also the length of your password is a big key in having a strong password.
  2. Enable MFA.
    Check this VMware KB on how to enable MFA for your My VMware account.

patch vCenter HA 6.7 U1

Maybe you think, we’ll how hard can it be ??
Yes, that was the same question I had. And to be honest… it is not that hard.
But there are some quarks or gotchas.
In this post I’ll explain the route I took for patching a vCenter HA setup.

Why don’t you use the VAMI ?

VAMI stands for ‘ Virtual Appliance Management Infrastructure ‘. It can be accessed via port 5480 like https://<FQDN VMware appliance>:5480 .
The VAMI of a vCenter Appliance (VCSA) has an update section, which you can use to patch the VCSA. This is a nice and easy way for patching the VCSA, but when you have vCenter configured as vCenter HA then this option won’t work. (I know from experience….)
After trying (and failing) I thought, why not read the manual….
VMware has a nice article about patching a vCenter in HA and you can find it here.
I still use the VAMI, but not for patching but for making a backup.

Continue reading “patch vCenter HA 6.7 U1”

Issue 3 – vmk0 removed when applying host profile

This article is part of a series of articles about issues I encountered during implementation of a vSphere stretched cluster based on vSphere 6.7 U1.
You can find the introduction article here

Issue 3

This issue is irritating, to say at least.
You configured your first ESXi host in a cluster, polished it etc… and created a host profile from this host. And then it should be easy to configure the other hosts. But, when you apply the host profile…. the host disconnects (because of a reboot….)… but never comes back….
What happened ??

What happened is that in the wisdom of the vSphere environment the vCenter instructed the host to delete vmk0… yes the kernel interface that vCenter uses to connect to ESXi (assuming the vSphere host is clean installed and has just one kernel port). Delete it to create vmk0 again…..
But it never gets to that part….
Do you recognize this issue ??

I googled for this issue and found blog posts going back to vSphere 5.1. So it looks like a difference in interpretation of how it should work.

Continue reading “Issue 3 – vmk0 removed when applying host profile”

VMware Enhanced Authentication Plugin Not Working

For several versions of vSphere vCenter it is possible to logon with your windows credentials. Making it easier to only tick a box and logon instead of typing your username and password.
This nice and neat trick is done via the VMware Enhanced Authentication Plugin.
The only issue… it does need to work every time, else it is going to be an annoyance.
And , yes, it became an annoyance for me… especially when using Firefox.
So let me give you to possible solutions

Continue reading “VMware Enhanced Authentication Plugin Not Working”

Change All Services icon through Rest API of vRA

A blog on my experience in using postman to change the All Services icon off vRA 7.2

Since vRA 7.1 you can change the “All Services” icon.
VMware has an article here on how to change this through the API.
I thought, maybe you could also do this via Postman. Which would also be a good exercise (for me) in understanding and using the Rest API via Postman.
The article below is one of many ways to solve this issue, feedback is welcome in the comments below.

Continue reading “Change All Services icon through Rest API of vRA”

Kanban with Outlook

For a while now I’m looking for ways to implement kanban in my workflows.
I do see the benefits, and I’m aware that it is not the golden solution for everything.

Requirements:

  • I need a visual representation of my WIP (work in progress)
  • not too much hassle like, logging on to different websites
  • not in public enviroments
  • direct accesible from my tooling (a.k.a. business laptop)
  • Intergration with task functions already available in my tooling (a.k.a. outlook tasks)

So I did a google on outlook and kanban. And after some searching I found this¬†site.¬†Which looks like a nice solution, using my outlook software etc…
So I tried to implement it.

And had one problem…. For security reasons the functionality to add a home page to an outlook folder has been disabled. And I understand it. But still wanted to see how this free solution would work.

So I found a site about enabling the home page function again. This site
The first option didn’t work for me, but adding the key¬† “EnableRoamingFolderHomepages”=dword:00000001 to¬†[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\Security]¬† was the magic I needed.

For now this solution will do. Using familiar software to get accustomed with using kanban.

restore data from Synology backup using virtual DSM

It happened to me….¬†

A crash off all my disks in a Synology DS 412+. Yeah….. For sometime my disks reported problems… but I thought.. just wait a few months more… then I can buy new disks…
But in the end…. the disks crashed all together.
I know what your first response is…. did you make backups ??
Answer: Yes. Yes I did.

How to restore

I used synology Hyper backup to make backups off my photos, documents and music folder to an external USB drive. And I new that these backups where very recent.
But to access this data  I had to restore it and needed a working Synology DSM. My options were:

  1. buy at least 2 new disks and re-install the DS 412+
  2. Find a friend with a DSM and use his/her DSM to restore my data and to move it into a cloud storage
  3. Looking into running DSM virtual

I choose option 3. Why ?
Well…. I was interested if it could be done.
Technicaly it could be possible. I had VMware workstation running on my laptop and new about xpenology. After some searching on the internet I found this  https://www.youtube.com/watch?v=a8YUq2QGhks
So it could be done….

Running DSM virtual

After some experiments I got it working, running DSM 6.x in a VM under VMware workstation 12.
I created a virtual DSM according to others blogs. Added a disk (VMDK) 200 GB in size. (large enough to restore a backup-set).
Then I started the DSM, did the usual configuration. Made a no-raid raid config with one disk. Created the volume in the newly created raid set. installed some DSM apps (at least the hyper-backup application).
And restored a backup from my external USB drive.

Moving data in to the cloud

Yes !!!!…. I had found a way to acces my data… but where should I store it ?
I used the Cloud sync app to sync the restored data to a cloud storage provider. If the provider supports webdav then it is possible to sync your local data to it. I needed to keep this running for several days due to the upload bandwith of my internet provider. but in the end I had my data accesible again.
xpenology and virtualization saved the day.