NSX

AUST IT Information Technology for everyone

NSX-V DB or disk stave issue
This article related with NSX-V upgrade error appearing during image uploading.

We are goinig to discuss quickly 2 error possible could appear:

"Cannot continue upgrade due to errors : Large database table. There are some tables with 5215948 entries, but the recommended table size is 5000000. We recommend running a database full vacuum before proceeding with upgrade. Upgrade aborted.. Please correct before proceeding."
OR
```
"Cannot continue upgrade due to errors : Insufficient disk space. 
Database disk usage is at 87%, but it should be less than 70%. 
We recommend running a database full vacuum before proceeding with upgrade. 
Upgrade aborted.. Please correct before proceeding."
```
If you have NSX-V cross vCenter, the upgade possible will be stalled on both.

During

TRUNCATE TABLE job_instance_task_instances, task_instance_task_data, task_instance_task_output, task_instance, task_task_init_data, task, task_policy, task_target, job_instance_job_output, job_instance, job_data_task_dependency_map, task_dependency_tasks, dependent_task,job_data, job_schedule, task_dependency, housekeeping_module;
NSX-v Apache Log4j voulnerability patching steps

This article will discuss critical vulnerabilities in Apache Log4j identified by CVE-2021-44228 and CVE-2021-45046 in VMware NSX-V.

Overall steps desctibed on the VMware official page, however, I had to search some details separatly to apply and test the patch.

Lets go trough the steps:

PRE-CHANGE ACTIVITIES:

1. Take a backup of your NSX-V Manager. This is always important step. One day it will save your live :), so please take time and do it properly.

2. Download the patch file from official VMware website signed_bsh_fix_log4j.encoded

CONFIRMING VOULNERABILITY IS PRESENT:

To confirm the voulnerability is present, we need to initiate test attack to the NSX-V manager.

Voulnerabiluty code will be sent by POSTMAN and result of the NSX-V behavior will be caprured by TCPDUMP running directly on the NSX-V manager.

1. SSH to the NSX-V to the root level (Engineering mode).

Additional information how to login in Engineering mode could be found here: https://austit.com/faq/324-nsx-engeneering-mode

If you experiencing isssues with SSH connectivity to the NSX-V manager, some troubleshooting steps could be found here: https://austit.com/faq/353-nsx-v-enable-ssh

2. Run next tcpdump command:
tcpdump -i lo -s 1500 -XX port 389

3. Using POSTMAN, post REST API to https://<NSX-Manager-IP>/api/2.0/services/securitygroup/globalroot-0

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body:
<securitygroup>

</securitygroup>

4. My result after th POST request on the POSTMAN:

securitygroup-17

5. My result fot TCPDUMP automatically appeared on the screen (patched system should not capture any traffic):

12:10:01.529110 IP localhost.36158 > localhost.ldap: Flags [S], seq 724932146, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
0x0000: 0000 0000 0000 0000 0000 0000 0800 4500..............E.
0x0010: 0034 551f 4000 4006 e7a2 7f00 0001 7f00 .4U.@.@.........
0x0020: 0001 8d3e 0185 2b35 9632 0000 0000 8002...>..+5.2......
0x0030: aaaa fe28 0000 0204 ffd7 0101 0402 0103...(............
0x0040: 0307 ..
12:10:01.529138 IP localhost.ldap > localhost.36158: Flags [R.], seq 0, ack 724932147, win 0, length 0
0x0000: 0000 0000 0000 0000 0000 0000 0800 4500..............E.
0x0010: 0028 0000 4000 4006 3cce 7f00 0001 7f00 .(..@.@.<.......
0x0020: 0001 0185 8d3e 0000 0000 2b35 9633 5014.....>....+5.3P.
0x0030: 0000 61a2 0000 ..a...

PATCHING SYSTEM:

Patching will be applied over POSTMAN REST API.

1. Using POSTMAN, post REST API to https://<NSX-Manager-IP>/api/1.0/services/debug/script

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body:
Select 'Binary' as body type and attach the file signed_bsh_fix_log4j.encoded

4. My result after th POST request on the POSTMAN is "1".

Starus 200 OK.

CONFIRMING NO VOULNERABILITY anymore:

To confirm the voulnerability patch was installed successfuly, we need to initiate test attack to the NSX-V manager.

Voulnerabiluty code will be sent by POSTMAN and result of the NSX-V behavior will be caprured by TCPDUMP running directly on the NSX-V manager.

1. SSH to the NSX-V to the root level (Engineering mode).

Additional information how to login in Engineering mode could be found here: https://austit.com/faq/324-nsx-engeneering-mode

If you experiencing isssues with SSH connectivity to the NSX-V manager, some troubleshooting steps could be found here: https://austit.com/faq/353-nsx-v-enable-ssh

2. Run next tcpdump command:
tcpdump -i lo -s 1500 -XX port 389

3. Using POSTMAN, post REST API to https://<NSX-Manager-IP>/api/2.0/services/securitygroup/globalroot-0

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body:
<securitygroup>

</securitygroup>

4. My result after th POST request on the POSTMAN:

Status: 404 Not Found

5. My result fot TCPDUMP shows no output.

Hope it helps, HAPPY PATCHING.
NSX-V enable SSH

My two NSX managers suddenly stop responding to the SSH requests.

Quick troubleshooting shows that SSH SERVICE on the both managers was stoped.

How to start the SSH Service on NSX-V:

1. Login to the NSX-V managed under admin account.

2. On the Summary tab, under System-level components, click Start to enable the SSH service.

Hope this quick tip save you time.
NSX-T Password expired issue

Just recently connected to my test NSX-T environment and found that my password is espired.

Actually it is admin credentials. See picture below:

Fix is eazy, just SSH to the host and update the admin passwrd.

You are required to change your password immediately (password aged)
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for admin.
(current) UNIX password:

New password:
Retype new password:

After updating the password over CLI, you can successfuly login over UI.

Password expiration is 90 days by default.

To avoind this issue in the future we can extend expiration dates using next command:

set user admin password-expiration 9999

To check the expiration date:
get user admin password-expiration

Thu Nov 09 2021 UTC 00:36:18.240
Password expires 9970 days after last change. Current password will expire in 9970 days.
NSX-V to NSX-T Migration: universal objects

We are planning to process with NSX to vSphere to NSX-T migration.

This article will be focused on multisite implementation.

If you have Standalone NSX-V implemetation, this article is not for you.

We have two site NSX-V (Cross-vCenters). Unfortunately, when we try to test the migtarion using the migration coordinator, we stuck solution it fails as it is not a supported connection.

It is related to universal objects that not able to be migrated due to miltisite implementation.

Migration of a true Cross-vCenter NSX for vSphere deployment with a primary NSX Manager and multiple secondary NSX Managers is not supported yet. However, if either the primary or secondary NSX Manager is set to a standalone or transit mode, the migration is supported.

We had further issue once we set the site as standalone, and had some issues with universal objects.

Fortunately for us, starting in NSX-T 3.1.1, migration of a single universal objects on the primary site, is supported.

Hope it helps.
NSX-T issue: Some appliance components are not functioning properly

One of the NSX-T nodes reporting "Some appliance components are not functioning properly" and web page is down.

Also could be some errors on the screen like:

Component health: SEARCH:UP, POLICY:UP, MANAGER:UNKNOWN, UI:UP, NODE_MGMT:UP.
Error code: 101

First, CLI to the applience and check if next services status:

get service

If manager services for example is down tray to start it manually and if it fails check errors for manager.log using this command:
get log-file manager.log

Assuming reboot alredy was in place.

If you need to detuch manangement plane:

For NSX-T 2.3 the command it:

detach management-plane

For NSX-T 3.0 it involes:

del nsx
NSX-T node deployment stuck in "Waiting To Register Host"

During the auto deployment installation of additional NSX node using User Interface (UI), the NSX Node displays a status of "Waiting to Register VM" indefinitely.

Issues could be differet and must be checked case by case.

Workaround is do delete the node with issues over API.

First check all your deployments and grab vm_id.

Delete the node using next command:

https://10.3.60.179/api/v1/cluster/nodes/deployments/86f643e9-e722-451b-be68-9a9f08d0b5b7?action=delete

Bonus:

Issue with the deloyment in my case was laydown in NFS storage. Simly moving the delpoyment VM target to anither type of storage, solved issue.
Troubleshooting NSX-V Manager

This article focusing on help to troubleshoot not responding or working incorrectly NSX0V Managed. Also, issue could be that NSX managet os not visiable on the vCenter.

1. Login to the NSX magager over SSH (if webpage is not responding). Otherwise use vCenter to open the consolle.

2. Check system uptime: show system uptime

3. Check NSX version: show version

4. Check tech support information: show tech-support

5. Check current time: show clock

6. Check free space: show filesystem

7. Check manager logs live: show log manager follow

8. Check system logs live: show log system follow

9. Check appmgmt logs live: show log appmgmt follow

10. Restart web-mananer service: (config)# web-manager restart
NSX Edge Unable to Resolve Hostname

This acricle based on VMware KB 68035: https://kb.vmware.com/s/article/68035

So, ESG installed, however having error under log: "Error resolving hostname; host='vrli.domain.local'

As article said, going to API to the NSX:

1. Using Postman, find all your ESGs:

GET https://NSX-Manager/api/4.0/edges

Grab edge ID: in my case "edge-1"

2.

PUT https://NSX-Manager/api/4.0/edges/{edgeId}/dnsclient

Body:

(USE KB ARTICELE)
HTTP Result Codes:
204 NO CONTENT

3. Check settings applied:

GET https://NSX-Manager/api/4.0/edges/{edgeId}/dnsclient
NSX-T v2.4.1 Installation

Why NSX-t v2.4.1: I am currently want to stick to VMware Validated Design VVD 5.1

1. Log in to the vCenter via vSphere client, right click the cluster and click “Deploy OVF Template…” of NSX-T.

2. Answer all the quiestions and deploy the instance.

3. Login to the instance.

4. Add vCenters as compute manager. Go to Fabric -> Computer Managers -> Add

5. Configure NSX on ESXi or KVM. Go to Fabric -> Nodes. Add host to the Nodes.

GLOSSARY:

N-VDS
N-VDS is a new construct which comes with NSX-T. It is like a distributed switch within vSphere, but managed entirely through NSX-T. Because of the fact that NSX-T supports multiple entities, like hypervisors, clouds and so on, it was necessary to use a construct that was not bound to vSphere. So an N-VDS can be created and connected to ESXi hosts, KVM-hosts, bare-metal servers and clouds.

N-VDS’s use uplinks to connect to the physical world. The creation of an N-VDS is done when creating a Transport Zone and the name cannot be changed afterwards (or at least I have not found a way yet).
NSX controller reboot

Sometimes you can see errors that one of the NSX controllers not healthy.

First, try to SSH to the applience and check logs.

Also, good practice to restart applience:

restart system
NSX Firewall Rules investigation

111
NSX Engeneering mode

This is basically a root bash shell on the underlying Linux based appliance. From here, system configuration files and scripts as well as most normal Linux functions can be accessed.

Engineering Mode: The authorized NSX Manager system administrator is requesting a shell which is able to perform lower level unix commands/diagnostics and make changes to the appliance. VMware asks that you do so only in conjunction with a support call to prevent breaking your virtual infrastructure. Please enter the shell diagnostics string before proceeding.Type Exit to return to the NSX shell. Type y to continue:”
VMware recommends to take full backup of the system before performing any changes after logging into the Tech Support Mode.

Login to NSX Manager Engeneering mode:
The NSX Manager contains many tools to help customers in conjunction with Global Support Services to resolve operational issues. The NSX for vSphere 6.x product features a customized command line interface that covers most of the basics that the user interface does and a little bit more.
- Login to NSX Manager shell
- show controller list all
- Go to enable mode
- enter engineering mode by typing st eng
- Type the shell diagnostics string: "IAmOnThePhoneWithTechSupport"

Find NSX controllers engeneering password:
- /home/secureall/secureall/sem/WEB-INF/classes/GetNvpApiPassword.sh controller-4

NSX controller Engineering Mode:
- Login to NSX controller shell
- Go to enable mode
- enter engineering mode by typing : debug os-shell
- use password string from previouse step
Step by Step - Upgrade NSX 6.3.5 to NSX 6.4.1

In the realease notes you will find a description of all the news https://docs.vmware.com/en/VMware-NSX-for-vSphere/6.4/rn/releasenotes_nsx_vsphere_641.html

1. Download from MyVMware NSX 6.4.1 Upgrade Bundle: VMware-NSX-Manager-upgrade-bundle-6.4.1-8599035.tar.gz
Trunt (VLAN Tagging) to VMs on ESXi
Sometimes you need pass through Trunk to VMs on ESXi emvitonment. Ex: Nested ESXi with NSX.

To set a standard vSwitch portgroup to trunk mode:
1. Edit host networking via the Virtual infrastructure Client.
2. Navigate to Host> Configuration> Networking> vSwitch> Properties.
3. Click Ports> Portgroup> Edit.
4. Click the Generaltab.
5. Set the VLAN ID to 4095. A VLAN ID of 4095 represents all trunked VLANs.
6. Click OK.
To set a distributed vSwitch portgroup to trunk mode:
1. Edit host networking via the Virtual infrastructure Client.
2. Navigate to Home > Inventory > Networking.
3. Right-click on the dvPortGroup and select Edit Settings.
4. Within that dvPortGroup, go to Policies > VLAN.
5. Set VLAN type to VLAN Trunking and specify a range of VLANs or specificy a list of VLANs to be passed to the Virtual machines connected to this portgroup.
  Note: To improve security, virtual Distributed Switches allow you to specify a range or selection of VLANs to trunk rather than allowing all VLANS via VLAN 4095.
Install VIB on VMware ESXi manually

If you experience issue with Host Preparation on your LAN NSX enviroment, it could be related with SSL certificate and dificulties to get

https://NSXMGR/bin/vdn/vibs-6.2.5/6.5-4463934/vxlan.zip

https://NSXMGR/bin/vdn/vibs-6.3.1/5.5-5114250/vxlan.zip

https://NSXMGR/bin/vdn/vibs-6.3.1/6.5-5124743/vxlan.zip

Solution:

1. Download vxlan.zip your PC

2. pscp vxlan.zip root@ESXIIP:tmp

3. on ESXi: esxcli software vib install -d /path/to/vxlan.zip

NOTE: if error appear: VIB VMware_bootbank_esx-vxlan_6.5.0-0.0.4463934 requires nsx-api <= 1, but the requirement cannot be satisfied within the ImageProfile.

please force the installation: esxcli software vib install -d /path/to/vxlan.zip --force

4. Check that VIB installed:
vmware -vl; esxcfg-vmknic -l | grep vxlan ; esxcli software vib list | grep esx-v

5. Back to NSX Manager and repair Host preparation.
NSX: Uninstalling stuck in progress
Issue explanation:

I went to prepare a cluster but the hosts never fully installed.
I figured I'd uninstall and give it another try.
Now all the hosts in the cluster report "In Progress" with the cluster status "Uninstalling".
I've rebooted all hosts multiple times and checked that the VIBs have been removed.
```
esxcli software vib list
```
```
esx-vxlan
esx-vsip
esx-dvfilter-switch-security
```
The manager and controllers have been rebooted as well.

Solution:

The vibs had already been uninstalled but the task was still hung.
I was able to work around this issue by disconnecting each host and adding back to a different cluster.
I deleted the cluster that was stuck and was then able to prepare the hosts in a new cluster.
The only think I lost was VM folder structure.
The hosts installed without issue and have now been added into my transport zone.

Note:

1. Do not forget to have DNS server and add there:
- vCenter
- esxi's
- NSX Manager
- Controllers
2. Domain Controller is not compulsory but recommended.

Google AdSence

AUST IT - Computer help out of hours, when you need it most.

Find out why we do it for less.

About

AUST IT will help you resolve any technical support issues you are facing onsite or remotely via remote desktop 24/7. More...

Contacts

Reservoir, Melbourne,
3073, VIC, Australia

Phone: 0422 348 882

This email address is being protected from spambots. You need JavaScript enabled to view it.

Sydney: 0481 837 077

Connect

Join us in social networks to be in touch.

Newsletter