Tag: VMware

vExpert NUC 2022

As mentioned in my previous blog I received (as many other vExperts) a ‘NUC’ (sponsered by Cohesity) to be used for a home lab. We were asked to blog about how we are going to use it in our homelab. So this is part 1 of …. (I have no clue ) blogging about my adventures with this NUC

What are you ?

To drop the b… it is not a nuc.
a nuc is a small form factor PC manufactured by Intel. This devices is manufactured by Maxtang. It is a small form factor PC, a NUC look-a-like.
And it has its own page at their site, titled ‘Intel Elkhart Lake J6412 Processor based compact fanless mini pc’.

The specs

CPU: Intel Elkhart Lake Celeron J6412 (4 cores)
Memory: SO-DIMM DDR4 max 32 GB
Ethernet: 2x nic – Realtek RTL8111H – 10/100/1000 Mbps
Storage: 1x M.2 for 2242/2280 SSD (SATA)
I/O: 2x USB 2.0, 2x USB 3.2, 1x USB-C
Display: 2x HDMI 2.0

Full specs can be found here

Usage

My ‘nuc’ is going to be used for a pfsense router. At the moment I use pfSense in a VM for my firewaling and routing at home. A VM is great, but in my homelab server (a supermicro) memory is scares. And having the router virtualised, demands to much memory. At first my pfSense VM was only used for my homelab, but it became more and more my main firewall and router for my home network. Which introduced some challenges. Namely losing internet when updating ESXi.
So moving it a dedicated hardware, results in no internet loss in my home for my family when I update my homelab.

Challenges

Installing pfSense, easy. And it looked like it was running fine. But once in a while, my WIFI was gone. And my LAN networks weren’t available.
Scrolling through the logs I first found the UUID error.
Also googling for the realtek NICs and pfsense, I found out that the realtek drivers aren’t updated in the freeBSD image.

UUID error

For the UUID error I had to search in the source code of pfSense (which is available in github) for the error message. And looked on the community forum of pfsense (here) And I found out that pfSense checks the UUID against a blacklist of UUIDs. And guess what….
My UUID was blacklisted….
The solution was to change the UUID. For this I needed to boot into the EFI shell and run DMI edit for AMI. After a long search I found the needed software here.
I downloaded dmi-edit-efi-ami.zip, unpacked it to a USB flash drive, read the documentation and changed some DMI settings.
And this solved my issue. But it didn’t solve the network issue.

Realtek NIC

I knew already that the realtek nics aren’t supported for ESXi. But I hoped pfSense (or freeBSD for that matter) would.
But no. pfSense would work for a week. Ben then my network dropped. Especially my VLANs.
After some search I found that you could update the Realtek drivers for freeBSD. (here)

To install them I followed the following steps on the shell of pfSense.

adding the package

pkg add https://pkg.freebsd.org/FreeBSD:12:amd64/latest/All/realtek-re-kmod-197.00.pkg

Letting pfSense know it should load the realtek drivers on boot by creating/ editing the file loader.conf.local in /boot/

if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"

Then a cold reboot of pfSense and checking the OS boot log that the re1/re0 interfaces show the correct version (version 1.97.00)

This solved the issue. I haven’t seen the issue since.
So now I have my firewall/router running on dedicated hardware, making it independent from my ESXi homelab server. And leaving me with more available resource in my homelab to experiment with BOSH and CloudFoundry.

Back from Explore EU 2022 – first thoughts

This week I attended VMware Explore Europe. Finally a life event and back in Barcelona.
And for the first time being part of an orange ‘army’ the ITQ workforce.
I had a great time. Getting to know my colleagues, learning some interesting stuff, meeting customers.

Multi cloud

Multi cloud, yeah, you couldn’t have missed it during Explore. It was the buzz word (and even the WIFI password…..) It was no surprise of course. VMware is advertising for years that their vision is to run any application on any infrastructure on any cloud. And even when you think you are not using the bits and bytes from VMware, well there is a good change you still do. Does spring.io ring a bell ??

Eventhough compared to the last VMworld in Barcalona, VMware Explore was a bit tuned down, there was still more then enough to do. It just depends on what your focus is. For me this year, I wanted to focus more on cloud native session. And just some session about new features for vSphere. And just merge into the vExpert en Code communities.
Being part of the vExpert community, you had a treat this year if you signed up for a barebone nuc. The nuc, sponsored by Cohesity is going to be a great addition to my homelab and with make life a bit easier for my esxi host.

I will run pfsense on the nuc, so my esxi is dedicated for homelab, and not running also my pfsense. Which gives me more flexibility when upgrading esxi (impact at the moment would be that internet access is down, which is not appreciated at my home šŸ™‚ ). And I don’t have to reserve CPU and memory resources anymore.
But that is for another time, another blog(story).

key takeaways / highlights

My key takeaways / highlights of Explore 2022 are

  1. 90DaysOfDevops by Michael Cade
    Michael shared his journey of discovering DevOps and documented it on github
    My intention is to start the 90DaysOfDevops journey for myself, but also to check it out if it is interesting for sysadmins I encounter. There is a gap to breach for sysadmins, because the way of working is going to be more dev like. But the dev country can look overwhelming. And what I expect of this journey is to get a better understanding and hands on experiences with the DevOps tooling
  2. Accelerating Your Career: The power of Mentoring (PCB2454EUR)
    A great panel discussion about mentorship in a company, which benefits it will give for the company and for its employees.
  3. Keeping a University Medical Centre Running During a VCF Transition (MCLB1452EUR)
    An honest story about the successes and pitfalls of the transition. Co-presented by a buddy of mine.
  4. Buidling and Running Enterprise-Grade Spring Applications in the Cloud (CNAB1953EUR)
    I’ve heard of the Spring framework, but never knew it was a VMware product, and never knew what it was about. Now I do šŸ™‚
  5. How to Be Successful at Modernizing Apps and Data, and the Adoption of Cloud (CNAB2113EUR).
  6. And off topic …. There is a nice grill restaurant somewhere in Barcelona… I’ve marked the spot …..

What are your key take aways of VMware Explore 2022 , and why ?

dipping my toes in vRA 8

re-visiting an old friend ?

I’ve been quite on my blog for the last half year. That is what moving to a new house, seeing my son growing up, renovating a bathroom, setting up my home automation etc… will do…
So at the last day of 2021, looking to the morning sky from my office at home, I decided to blog about my reintroduction experience with vRA 8

view from my attic

Yeah, reintroduction.

A few years ago I followed the VMware ICM (installation, configuration and management) course for vRA 7. Five days of learing about fabric groups, blueprints, orchestrator, the several VMs needed to build a vRealize Automation platform. By trade I’m a developer / automation engineer. The course was more a meaning to an end. But that end never happened. Yeah I’ve been building a vRA 7 platform. And on another assignment, rescuing a vRA 7 platform ( it was falling apart)… But really developing automation with vRA.. Never happened.
And my warm feelings for vRA 7 went away. A complex platform, a memory game in finding the correct terminology and links in the vRA portal… and that awfull java client for orchestrator. No succes stories for me.

Until

Until this year…
I’m working with a customer who has already a vRealize platform in place, but needs support in developing. In helping their administrators developing a developer mindset. And they have vRA 8

vRealize automation 8

Of course I had seen VMware’s presentation about vRA 8. And to be honest I started to become positive about vRA. No mix of appliances and windows VMs, no MS SQL VM… just one type of appliance, running kubernetes and distributed the several services over kubernetes pods.
vRA8 is underneath the hood a completely different than its predecessor. And no java client. And the principe that everything should be code…

In the last two weeks I’ve (with some help) deployed a new vRA 8 platform, and developed some automation, using the exisiting vRA 7 platform. We decided not to use the migration tool. But to rebuild the functionality. The main reason for this is the different approaches between vRA 7 and 8.
vRA 8 is focused on tag policy driven placement. Meaning you tag resources with several kind of metadata, like enviroment (DMZ,Dev,Production,Test) , OS, storage SLA, backup SLA. And you use constraints in blueprints and projects to guide the deployment.
You use vRO actions as external sources for option values, default values in the vRA request forms to help the end user in making selections.
You don’t develop one big monolithic automation, but need to slice it up in smaller parts. Thinking of ‘can I reuse it’, ‘can I make it more task general and dynamic’, etc.
And in the end you have some simple workflows and blueprints, but build a catalog with dynamic items, helping the administrator and/or developers in deploying VMs

My experience with developing in vRA 8 is a positive one. As for any language or automation platform, the main points to take away are:

  • find out how some programming constructs should be set up, like if…then…else, for..loop, regular expressions etc…
  • learn to code structured. Use readable names for variables, constants, objects etc… Use comments in your code to highlight, explain what the next line of instructions do.
  • The more high level your code is, the more you should use comments to explain what that code, workflow should do.
  • Learn by failing… take one function you are trying to use, build a new workflow around it, and test it… yes it can be time consuming… but you’ll learn
  • work with other developers, learn from their way of structure code. From their approach to automation
  • First try to get a picture of what needs to be automated, describe it in manual actions, how would an administrator solve this task by hand ?
    Try to get a broader sence of how the administrators are working , and ask about why things are as they are, what is the reason / decision for the way of working…..
    This is one of the hard parts of automation. Don’t start right away on the keyboard, but try to understand what is being asked to automate

Final

As you can tell, my experiences with vRA8 are positive. You need to invest time to understand the platform… but it makes more sense then vRA7 did. And it is completely different.
One of the main challenges with automation is, selling it to the organisation… and making it donkey proof… it takes time… take small steps so the changes for succes are bigger. And celebrate them.

TIP: Secure your My VMware account with MFA

I know, it is not the most interesting subject to blog about.
VMware gives you the opportunity for your My VMware account to use MFA.
And I would advise to use MFA.

Why ?

Well, depending on the situation, your my vmworld account can have access to different company accounts. Giving you, depending on roles and permissions, the ability to create support requests, download software, access licenses etc…
And that is great.
But it is a risk when your account gets hacked.
That is where MFA can help. It is an extra line of defense.

MFA

MFA stands for, multi factor authentication. And when MFA is enabled,a person needs to present multiple pieces of evidence to authenticate (see https://en.wikipedia.org/wiki/Multi-factor_authentication )
Most of the times these pieces are a password and a token, generated by a authentication app or device.
So, when you would know my password, you still can’t use my ‘my vmware’ account because I’ve enabled MFA. And the token is generated on my phone, which I have with me, and is locked.

My advise

  1. Use a strong password. Strong doesn’t mean a lot off difficult characters, although there are some rules you have to follow. Study has shown that also the length of your password is a big key in having a strong password.
  2. Enable MFA.
    Check this VMware KB on how to enable MFA for your My VMware account.

Passed

I already blogged about my VCAP deploy experience.( you can find it here ) And as stated…. I passed !!!… Sadly VMware doesn’t let you know the mistakes you made in the exam.
But in the end…. this is what counts !!!

Next step ??

Well… to do the VCAP design exam. To be honest, I don’t look forward to it, because theory questions are not my strong suit.
My strategy is going to study, visit VMware Empower europe 2019, do the VCAP design exam @ Empower (it is part of the conference) so I get a direction of what the exam looks like…. and maybe pass it on the first try.

Issue 2 – bypassing the fingerprint cache message when using PLINK

This article is part of a series of articles about issues I encountered during implementation of a vSphere stretched cluster based on vSphere 6.7 U1.
You can find the introduction article here

Issue 2

For some configuration settings I need SSH access to the host.  I use plink.exe to execute instructions through the SSH session. One issue, the first time when you connect with plink you get a message about storing the fingerprint ID in the cache. Normally you would accept this when using putty. But now this is going to be a challenge.
On some other blogs I found the solution. You echo the ‘Y’ which results in storing the ID in the cache.
In my code I now  call plink two times. The first time to accept the fingerprint, the second time to execute the command.
Why two times ? Well, I can’t assume that the fingerprint ID is already known.
The first plink instruction is a simple exit, we only want to check if we can logon.

$credential=get-credential
$plink="d:\plink.exe $hostname -l "+ $credential.username + " -pw " + $credential.getnetworkcredential().password
$command="ls"
invoke-expression ("echo Y | " + $plink +Ā  " -ssh exit")
invoke-expression ($plink + " "+ $command)

Issue 1 – changing root password

This article is part of a series of articles about issues I encountered during implementation of a vSphere stretched cluster based on vSphere 6.7 U1.
You can find the introduction article here

Issue

All the hosts are delivered with 6.5 U2 pre-installed, and they have their own root password. For the implementation we want to have just one general root account password. So after adding all the hosts to the cluster I want to change the root password with powercli. But I tripped over a bug in get-esxcli (thanks to this thread ). The ‘&’ character is not correctly being interpreted when using get-esxli.
The script I wrote checks if the new password contains that character and will kindly ask to change it. After succesfull validation of the password it will apply it to all selected esxi hosts.
I

#-- select one or more hosts
[array]$esxiHosts=get-vmhost | select name | sort | out-gridview -Title "Select one or more ESXi Hosts"-OutputMode Multiple
if ($esxiHosts.count -eq 0) {
write-host "No host(s) selected, will exit." -foregroundcolor yellow
exit
}
#-- ask for root password and validate it agains known bug
Do {
$newCredential = Get-Credential -Username root -Message "Enter the password for the ESXi root account."
$isValid=$true
if ($newCredential.getNetworkCredential().Password -imatch "[\&]") {
$isValid=$false
write-host"Password contains character & which get-esxcli can't handle (bug)..... please consider a different password." -foregroundcolor yellow
}
}
until ($isValid)

#-- change root password for all selected esxi hosts
foreach ($esxiHost in $esxiHosts) {
$esxiHost=get-vmhost -Name -$esxiHost.name
$esxiCli=get-esxcli -v2 -vmhost $esxiHost
$arguments=$esxcli.system.account.set.createArgs()
$arguments.id=$newCredential.UserName
$arguments.password=$newCredential.GetNetworkCredential().password
$arguments.passwordconfirmation=$arguments.password
try {$esxcli.system.account.set.Invoke($arguments)}
catch{write-host "Setting password failed for " $esxiHost.name -ForegroundColor Yellow}
}

Issues I encountered with a stretched cluster implementation on 6.7 U1

At the moment I’m busy with a stretched cluster implementation based on vSphere 6.7 U1. Most of the configuration is straight forward. But I encounter some snags.
So this post is about these snags, and how I solved them.

For configuring 16 hosts I use a lot of powerCLI. Why ? Well I have some issues with host profiles, and not the time (yet) to figure out what is going on.
Edit: I found out what the issue is, I’ll explain it in Issue 3.

I encountered the following issues

restore data from Synology backup using virtual DSM

It happened to me….Ā 

A crash off all my disks in a Synology DS 412+. Yeah….. For sometime my disks reported problems… but I thought.. just wait a few months more… then I can buy new disks…
But in the end…. the disks crashed all together.
I know what your first response is…. did you make backups ??
Answer: Yes. Yes I did.

How to restore

I used synology Hyper backup to make backups off my photos, documents and music folder to an external USB drive. And I new that these backups where very recent.
But to access this dataĀ  I had to restore it and needed a working Synology DSM. My options were:

  1. buy at least 2 new disks and re-install the DS 412+
  2. Find a friend with a DSM and use his/her DSM to restore my data and to move it into a cloud storage
  3. Looking into running DSM virtual

I choose option 3. Why ?
Well…. I was interested if it could be done.
Technicaly it could be possible. I had VMware workstation running on my laptop and new about xpenology. After some searching on the internet I found thisĀ  https://www.youtube.com/watch?v=a8YUq2QGhks
So it could be done….

Running DSM virtual

After some experiments I got it working, running DSM 6.x in a VM under VMware workstation 12.
I created a virtual DSM according to others blogs. Added a disk (VMDK) 200 GB in size. (large enough to restore a backup-set).
Then I started the DSM, did the usual configuration. Made a no-raid raid config with one disk. Created the volume in the newly created raid set. installed some DSM apps (at least the hyper-backup application).
And restored a backup from my external USB drive.

Moving data in to the cloud

Yes !!!!…. I had found a way to acces my data… but where should I store it ?
I used the Cloud sync app to sync the restored data to a cloud storage provider. If the provider supports webdav then it is possible to sync your local data to it. I needed to keep this running for several days due to the upload bandwith of my internet provider. but in the end I had my data accesible again.
xpenology and virtualization saved the day.

Restore experience vCenter appliance

Until a few months ago I worked with vCenters that were running on a Windows OS. Because that was the common practise. The appliance was promising but not ready for production use. When vSphere 5.5 was introduced it became an interesting discussion,should you use the appliance or still go the windows way.
I think one of the points holding people of from going to the appliance was (and maybe still is) the unknown.
I was mostly familiair with a windows OS.Yes the Windows patching was irritating, and if you wanted to upgrade VMware vCenter ….. that also was annoying. But not anymore… (well I guess for over a year that is…)

The appliance in 6.0 is great. I wrote in another article about my experience of migrating to 6.0 appliance with an update manager deployed. (https://vblog.bartlievers.nl/2017/01/17/update-manager-6-0-with-vcenter-6-0-after-migration-wizard) And I’m loving it. I was impressed by the ease of migrating from vCenter 5.5 running on Windows to 6.0 VCSA.

But there was one, tiny little, issue, called backup and restore.
How should that work when you are running an appliance.
For testing purposes we did a restore of the appliance but ran into a problem. The network adapter changed. The appliance OS didn’t recognize it anymore as eth0 but as eth1.
And our first conclusion was… well…. that is a problem…restoring a VCSA.
Diving into some vmware documentation, I ran into info that restoring a VCSA should work… (see question 29 in https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146439)…. What did we do wrong….

After examening the events of the restored VCSA we saw an error. vCenter was saying that there was a duplicated MAC address…. so it changed the MAC address of the restored VCSA… hence the eth1.

And yes… that was the issue…. in default vCenter will change a MAC address if it already exists. (that is the short version)…. so I tried it again…. But with a small change.

I created a virtual portgroup on a vswitch, not on the dvswitch. Taking care of the duplicated MAC address in a dvSwitch… I changed the MAC address of the restored VCSA back to the original setting. And I started the VM…. voila….
The VM booted…. and after waiting for the boot process to complete…. I had a restored VCSA, fully functional.

Summary:

  • restoring a VCSA does work… just keep in mind if you replace the existing VCSA or restoring it next to it (duplicated MAC address)
  • Updating the VCSA is a breeze . You need connection to the internet, and the appliance will take care of the update. No seperate OS updates and vCenter updates.
  • (side note) the webclient interface is faster than the one in 5.5
%d bloggers like this: