Issue 3 – vmk0 removed when applying host profile

This article is part of a series of articles about issues I encountered during implementation of a vSphere stretched cluster based on vSphere 6.7 U1.
You can find the introduction article here

Issue 3

This issue is irritating, to say at least.
You configured your first ESXi host in a cluster, polished it etc… and created a host profile from this host. And then it should be easy to configure the other hosts. But, when you apply the host profile…. the host disconnects (because of a reboot….)… but never comes back….
What happened ??

What happened is that in the wisdom of the vSphere environment the vCenter instructed the host to delete vmk0… yes the kernel interface that vCenter uses to connect to ESXi (assuming the vSphere host is clean installed and has just one kernel port). Delete it to create vmk0 again…..
But it never gets to that part….
Do you recognize this issue ??

I googled for this issue and found blog posts going back to vSphere 5.1. So it looks like a difference in interpretation of how it should work.

Cause

I expect the cause has something to do with the fact that the kernel port is created on installation of ESXi, and not via the vCenter. Assigning it a MAC address that is unknown by vCenter.

Workaround / Solution

The solution is to re-create the kernel port, reboot the host, create the host profile from this host and used it for the others.

The re-creation is done via the server console (iDrac, iLO etc…) due to the fact that by removing the kernel port you also have no connection to manage it.

In short

Below the commands that are used.
assumptions:

  • the original vmk0 has a static ipv4 address
  • vmk0 is connected to a distributed vSwitch
esxcfg-vswitch -l | grep vmk0 | awk '{print $1}'
esxcli system coredump network set --enable=false
esxcli network ip interface remove --interface-name=vmk0
esxcli network ip interface add --interface-name=vmk0 --dvs-name=<name of dvSwitch>  --dvport-id=<port ID for vmk0 on the dvSwitch>
esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=<ipv4 address of kernel port> --netmask < subnetmask of kernel port> --type=static
esxcfg-route -a default <ipv4 gateway address>
esxcli network ip interface tag add --interface-name=vmk0 -t Management
esxcli system coredump network set --enable=true

The steps

0 Put the ESXi host in maintenance mode

We put it in maintenance mode so no VMs are present on the host, and that vCenter knows that it can’t use the host at all.

1 (optional) write down the port ID

Look up on the dvSwitch what the port ID is for this vmk0. You need this info later.
This applies only if you use a dvSwitch and vmk0 is connected to the dvSwitch.

esxcfg-vswitch -l | grep vmk0 | awk '{print $1}'

2 make sure the ESXi shell is running

You can do this in the webclient, with a powerCLI command or via the ESXi shell. The ESXi shell instruction is

3 open a session to the server console

4 Go to the ESXi shell

Normaly the server console will display the DCUI interface of ESXi. Switch over to the shell with ALT+F1. A login prompt should be shown. If not then you forgot step 1.
Logon with the root account.

5 (optional) disable network coredump

If network coredump is enabled and configured to use vmk0 disabled it before removing.

esxcli system coredump network set --enable=false

6 remove vmk0 – the management kernel port

esxcli network ip interface remove --interface-name=vmk0

6 Create the a new kernel port

esxcli network ip interface add --interface-name=vmk0 --dvs-name=<name of dvSwitch> --dvport-id=<saved port ID from step 1>

7 configure the port

Assuming that vmk0 has a static IPv4 address.

esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=<IPv4 Address of kernel port> --netmask=<subnetMask of kernel port> --static

8 update gateway

esxcfg-route -a default <ipv4 address of vmk0>

9 tag management service

esxcli network ip interface tag add --interface-name=vmk0 -t Management

10 (optional) Enable network coredump

If disabled in step 5, enabled it again.

esxcli system coredump network set --enable=true

11 check that vCenter is connected to the ESXi host

If all went well then the ESXi host should show connected in vCenter.

12 create a Host Profile from this host

Abstract a new host profile from this host, edit the host customization.

13 edit the host customization for all hosts

For the hosts where this host profile should be applied on, edit the host customization

14 apply the host profile

Apply the host profile to the other hosts and reboot them.

I hope this post was usefull, use the comments below if needed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.