Category: Uncategorized

Resistance is futile – my kubeconEU 2023 experience

As I’m moving from SDDC into the CNA universe, I attended KubeCon 2023 in Amsterdam this year. Being known with the VMware VMworld / Explore events in Barcelone, I found simmelarities and differences.
This blog is (to process) my experiences at Kubecon 2023
(it took a few moments processing… yeah I know… it is december, kubecon was in april )

First Impression

My first impression, no surprise, corresponded with the universe of kubernetes itself. Colourfull, chaotic, daunting.

KubeCon stands for Kubernetes conference…. so the conference circles around all that is kubernetes and related solutions. And what is kubernetes, well in short… it is an orchestrator of containers. Yes in basics it is… but it doesn’t only orchestrate containers, it can be extended with all kind of solutions, so that containers can be consumed to support cloud native apps.
Kubernetes is an open-source project, under the umbrella of the CNCF (Cloud Native link here). Due to the fact that it is open-source, it isn’t owned by a vendor, but ‘owned’ / supported through a community of developers / maintainers. Which is great, and comes with some challenges.

It is great, because anyone can check the code for bugs, security flaws etc… and you can help to fix it. Also adding new features is possible (yes every community has their rules and cultures…. but in basic it is). Try this with a close-sourced software. The only thing you can do, is ask the vendor if it would add a new feature…. and the vendor only adds it, if it adds value to their product and/or bank account.


Chaotic

But why do you think it is chaotic ??
Well… it wasn’t the size, a conference of 10.000+ attendees. It was hard to find out where to start. Sure the sessions where categorized in tracks, and there was a 101 track (which had great sessions)… But there was so much, and so much lingo that was unfamiliar for someone who is used to SDDC.
And the booth / exhibition area, was great… but with a lot of unfamiliar projects.

The universe of kubernetes exists off a lot off projects. Projects that solve / extend certain areas, like networking, security, resource management, green energy, RBAC, logging …..
Which is great. All the projects are on its own, also open-source, and supported by a community. But…. the challenge is to know which project would solve your issue…. and why that project, and the other one that claims to solve the same issue… And how is the support on the project. As an SDDC guy, I can rely on my vendor support (well… okay…. most off the time)…
But in open-source world…. it depends. Because the people in the community, also have their day jobs… And they don’t get paid for their work (well most off them).
So when I visited the projects pavilion…. I had a hard time to understand what they try to solve. and to validate it.

Sessions

Mostly I followed the 101 track sessions. As a kinda noob in CNA land, those where very interesting. The session that stood out for me, (it is december now… and I still remember it) was ….

“How to blow up a kubernetes cluster.”

Yes I loved the title…. who doesn’t want to destroy a k8s cluster…. (at least I want to know how to).
What frightened me was that the room was packed, really packed…. there where even people standing outside, following the session via a live stream…. Why do so many people want to know this ? Especially developers….. scary
And when the presenter started to explain, I got a deja-vu …. it was resource mismanagement…. Overcommitting of CPU and/or memory resources with a orchestrator which basic task was to keep the containers alive. Did a container got ill… kill it (it is cattle, not a pet)… and restart it on another node, if possible (resources would allow it)….
But what if you would configure your pod (container) with resource requests and limits that would get your cluster in danger….
Yes… it was the same problems as with hypervisor and VMs, but than on a playing field that was running in the VMs…. And why are a lot of the attendees in the session looking like they heard some ground breaking stuff….
That was a scary look… this should be common knowledge… right ???
Nope… not right, not for developers….

Questions arose:
Didn’t we solve this issue already as IT … why does CNA try to reinvent the wheel again… why…. … ow wait…. the common denominator is people…. that explains it….
And it dawned on me, as someone who has experiences with SDDC… maybe I could help my fellow CNA-ers with these kind of challenges…. Yes, devOps could be scary…. coding , also… but these kind of challenges, they are not new to the SDDC world. Bring it on….

December

It is december. I wrote this post short after kubecon, but didn’t get to the point of posting it.
Now reading it back, having more experience as a platform-engineer, with the CNA word… I think this post still has worth… cause, yes it is true… there isn’t much realisation about impact of resource requests….
And my switch from SDDC to CNA…. I love it…
I’m still involved with the SDDC stack (to be honest, to much involved)… but the CNA stuff, TAS/TKGi …. no pets / snowflakes (or almost no snowflakes)… but see it as cattle …. loving it….
And what about my blogging… well 2024 is around the corner, and I’m trying to share my experience in moving from SDDC to CNA..

TIP

You made it to the end of this post…
A tip, check this post Kubecon Amsterdam WrapUp

vExpert 2023

Yes, for the fifth year in a row I got the VMware vExpert status.
Being a VMware enthousiast, I blog about my experiences (not as much as I want too) and try to be an active community member of some Facebook VMware expert groups.

For this year I hope to blog more about my experiences transitioning into CNA (Cloud Native Applications), focusing on the platform technology for supporting CNA. Think about VMware Tanzu TAS, software like, Confluence, BOSH, Cloudfoundry, (Kubernetes)…..

Back from Explore EU 2022 – first thoughts

This week I attended VMware Explore Europe. Finally a life event and back in Barcelona.
And for the first time being part of an orange ‘army’ the ITQ workforce.
I had a great time. Getting to know my colleagues, learning some interesting stuff, meeting customers.

Multi cloud

Multi cloud, yeah, you couldn’t have missed it during Explore. It was the buzz word (and even the WIFI password…..) It was no surprise of course. VMware is advertising for years that their vision is to run any application on any infrastructure on any cloud. And even when you think you are not using the bits and bytes from VMware, well there is a good change you still do. Does spring.io ring a bell ??

Eventhough compared to the last VMworld in Barcalona, VMware Explore was a bit tuned down, there was still more then enough to do. It just depends on what your focus is. For me this year, I wanted to focus more on cloud native session. And just some session about new features for vSphere. And just merge into the vExpert en Code communities.
Being part of the vExpert community, you had a treat this year if you signed up for a barebone nuc. The nuc, sponsored by Cohesity is going to be a great addition to my homelab and with make life a bit easier for my esxi host.

I will run pfsense on the nuc, so my esxi is dedicated for homelab, and not running also my pfsense. Which gives me more flexibility when upgrading esxi (impact at the moment would be that internet access is down, which is not appreciated at my home 🙂 ). And I don’t have to reserve CPU and memory resources anymore.
But that is for another time, another blog(story).

key takeaways / highlights

My key takeaways / highlights of Explore 2022 are

  1. 90DaysOfDevops by Michael Cade
    Michael shared his journey of discovering DevOps and documented it on github
    My intention is to start the 90DaysOfDevops journey for myself, but also to check it out if it is interesting for sysadmins I encounter. There is a gap to breach for sysadmins, because the way of working is going to be more dev like. But the dev country can look overwhelming. And what I expect of this journey is to get a better understanding and hands on experiences with the DevOps tooling
  2. Accelerating Your Career: The power of Mentoring (PCB2454EUR)
    A great panel discussion about mentorship in a company, which benefits it will give for the company and for its employees.
  3. Keeping a University Medical Centre Running During a VCF Transition (MCLB1452EUR)
    An honest story about the successes and pitfalls of the transition. Co-presented by a buddy of mine.
  4. Buidling and Running Enterprise-Grade Spring Applications in the Cloud (CNAB1953EUR)
    I’ve heard of the Spring framework, but never knew it was a VMware product, and never knew what it was about. Now I do 🙂
  5. How to Be Successful at Modernizing Apps and Data, and the Adoption of Cloud (CNAB2113EUR).
  6. And off topic …. There is a nice grill restaurant somewhere in Barcelona… I’ve marked the spot …..

What are your key take aways of VMware Explore 2022 , and why ?

dipping my toes in vRA 8

re-visiting an old friend ?

I’ve been quite on my blog for the last half year. That is what moving to a new house, seeing my son growing up, renovating a bathroom, setting up my home automation etc… will do…
So at the last day of 2021, looking to the morning sky from my office at home, I decided to blog about my reintroduction experience with vRA 8

view from my attic

Yeah, reintroduction.

A few years ago I followed the VMware ICM (installation, configuration and management) course for vRA 7. Five days of learing about fabric groups, blueprints, orchestrator, the several VMs needed to build a vRealize Automation platform. By trade I’m a developer / automation engineer. The course was more a meaning to an end. But that end never happened. Yeah I’ve been building a vRA 7 platform. And on another assignment, rescuing a vRA 7 platform ( it was falling apart)… But really developing automation with vRA.. Never happened.
And my warm feelings for vRA 7 went away. A complex platform, a memory game in finding the correct terminology and links in the vRA portal… and that awfull java client for orchestrator. No succes stories for me.

Until

Until this year…
I’m working with a customer who has already a vRealize platform in place, but needs support in developing. In helping their administrators developing a developer mindset. And they have vRA 8

vRealize automation 8

Of course I had seen VMware’s presentation about vRA 8. And to be honest I started to become positive about vRA. No mix of appliances and windows VMs, no MS SQL VM… just one type of appliance, running kubernetes and distributed the several services over kubernetes pods.
vRA8 is underneath the hood a completely different than its predecessor. And no java client. And the principe that everything should be code…

In the last two weeks I’ve (with some help) deployed a new vRA 8 platform, and developed some automation, using the exisiting vRA 7 platform. We decided not to use the migration tool. But to rebuild the functionality. The main reason for this is the different approaches between vRA 7 and 8.
vRA 8 is focused on tag policy driven placement. Meaning you tag resources with several kind of metadata, like enviroment (DMZ,Dev,Production,Test) , OS, storage SLA, backup SLA. And you use constraints in blueprints and projects to guide the deployment.
You use vRO actions as external sources for option values, default values in the vRA request forms to help the end user in making selections.
You don’t develop one big monolithic automation, but need to slice it up in smaller parts. Thinking of ‘can I reuse it’, ‘can I make it more task general and dynamic’, etc.
And in the end you have some simple workflows and blueprints, but build a catalog with dynamic items, helping the administrator and/or developers in deploying VMs

My experience with developing in vRA 8 is a positive one. As for any language or automation platform, the main points to take away are:

  • find out how some programming constructs should be set up, like if…then…else, for..loop, regular expressions etc…
  • learn to code structured. Use readable names for variables, constants, objects etc… Use comments in your code to highlight, explain what the next line of instructions do.
  • The more high level your code is, the more you should use comments to explain what that code, workflow should do.
  • Learn by failing… take one function you are trying to use, build a new workflow around it, and test it… yes it can be time consuming… but you’ll learn
  • work with other developers, learn from their way of structure code. From their approach to automation
  • First try to get a picture of what needs to be automated, describe it in manual actions, how would an administrator solve this task by hand ?
    Try to get a broader sence of how the administrators are working , and ask about why things are as they are, what is the reason / decision for the way of working…..
    This is one of the hard parts of automation. Don’t start right away on the keyboard, but try to understand what is being asked to automate

Final

As you can tell, my experiences with vRA8 are positive. You need to invest time to understand the platform… but it makes more sense then vRA7 did. And it is completely different.
One of the main challenges with automation is, selling it to the organisation… and making it donkey proof… it takes time… take small steps so the changes for succes are bigger. And celebrate them.

Code repository maintenance.

I’ve changed my code repository on github.

And as someone already noticed this breaks some links on my blog.
I’m in the progess of fixing them, but if you find a link that is not working anymore, please leave a comment. I’ll try to fix it as soon as possible.

The new code repository can be found here

New adventure

It has been a while since my last blog post. And there has happened some stuff.
This month I started a new adventure. I started to work at B-Critical in the roll off Senior VMware Consultant.

The goal of this adventure is to work closely with VMware at B-Critical, to be critical at IT infrastructure challenges, to have a lot of fun while answering these challenges and help customers.

Also I want to thank my former colleagues at Conoscenza for the fun and challenges during the projects we worked on.

VCSA 6.5 with subCA Gotcha’s

One of my projects recently was migrating from a vSphere 5.5 enviroment, running on Widows OS, to vSphere 6.5 running on VCSA.
VMware did a good job with the tools for migrating to VCSA 6.5.

In vSphere 5.5 it was a challenge to replace the self-signed certificates, especially if they where expired…. From 6.0 on VMware introduced certificate managed for the vSphere enviroment. And where the PSC would be the CA for the vSphere enviroment.
With this, you can choose to let vCenter be a subCA in your PKI infrastructure.

You can find enough how-to’s for configuring the VCSA as a subCA. Like  here:

I just want to add some Gotcha’s, maybe you think ‘Yeah, of course…. duh….. why don’t you know that…’ well… then that Gotcha’s wasn’t meant for you 🙂

Gotcha’s

  1. file access to VCSA
  2. certificate names
  3. issuing the subCA certificate on a windows offline root CA

Gotcha 1  – File access to VCSA

At some point you need to download the CSR (certificate request) and copy it in your PKI enviroment. The most common method is to use SCP for file access… but the shell of the VCSA doesn’t support this by default.
This is what you need to do:

  1. allow SSH and bash access to VCSA
  2. Login as root into the VCSA with a ssh-client
  3. access the shell
    >shell
  4. change the shell for root to bash shell
    >chsh -s “/bin/bash” root
  5. Run winscp (or any other SCP client) and connect to the VCSA

To change the shell setting back to its original configuration, run
>chsh -s “/bin/appliancesh” root

Gotcha 2 – certificate names

This was a, a-yes-off-course moment…. but it did cost me some headache….

When running the steps to configure the VCSA as a subCA in your PKI, you get a few times the question

Enter proper value for ‘Name’ [Default value : CA] : ……

Make sure you use unique names here. This is the name of a certificate that is going to be generated for you. The certificate manager doesn’t check these settings, and in the end it will make a rollback…. a.k.a. waste of time.

My suggestion is for a naming convention: <hostname>-<file name being configured>
For instance, when your VCSA hostname is: VCSA-01 and your busy with the questions for the vpxd.cfg file, then the value you would enter would be ‘VCSA-01-vpxd’
But you are (off course) free to choose your own. Just make sure that these values are unique.

The certificate manager is asking values for the following files:

  • MACHINE_SSL_CERT.cfg
  • machines.cfg
  • vsphere-webclient.cfg
  • vpxd.cfg
  • vpxd-extensions.cfg
  • certool.cfg (used to creating the subCA CSR)

Gotcha 3 – issuing the subCA on a windows offline root CA

As a good practise, the CA is installed on a windows server that is not part of the AD domain and normally is powered off.

When requestion a certificate you normaly would use a browser and go to the url <hostname subca>/certsrv. But if this isn’t available on a offline root, you can use the cmd-line.

The cmd is:
> certreq -submit -attrib CertificateTemplate:<name of CA template> <filepath of CSR>

Steps to issue a the subCA certificate

  1. create the CSR with the certificate-manager in the VCSA
  2. copy the CSR to the offline root server (see gotcha 1 for scp file access)
  3. request the certificate with the cmd-line command
  4. open the certificate authority
  5. find the requested certificate under ‘pending requests’
  6. issue the certificate
  7. export the certificate to a file (export Binary data -> Binary Certificate )
  8. import the new certificate under local machine in the personal location
  9. export the imported certificate as a base64 certificate
  10. export the rootCA certificate as a base64 certificate
  11. create a chain file by adding first the subCA file content and then the rootCA file content into the chain file
  12. copy the chain file to the VCSA
  13. (shutdown  the offline root server)

build-vmHostImage v0.2

The script has been modified to exclude certain vibs. The reason is that with some configurations we found out that the wrong vib was loaded on a vspherer host. Resulting in a PSOD, due to driver, hardware, ESXi version conflicts.

With this in mind, I added the functionality to exclude VIBs. The excluded VIBs are listed in the parameters.ps1 file.
After the new image profile has been configured, the script will exclude the VIBs, if they are present. VIBs that are excluded are reported in excluded.txt file.

The script creates a parameters.ps1 file in the project folder, containing the names of the Vibs that are excluded. This file is used to re-create an image if necessary.

You can find the script at github, here