VMware

AUST IT Information Technology for everyone

send e-mail

Renewing NSX-T Local certificates
This article focusing on the main piont to help to renew the local NSX-T sertificates:

Use REST API requests by Postman for example.

use admin credentials to make API requests.

STEP x: Generate self signed certificate:

Make sure that when this certificate was imported, the option Service Certificate was set to No

STEPx: Validate the cetrificate:

- The basic API call for this is: GET https://<nsx-mgr>/api/v1/trust-management/certificates/<cert-id>?action=validate>

Responce shuld be: "status": "OK"
```
POST https://<nsx-local-mgr>/api/v1/trust-management/certificates?action=set_pi_certificate_for_federation
{
    "cert_id": "c5f13ec0-8075-441e-80b5-aeb707f6b87e",
    "service_type": "LOCAL_MANAGER"
}
```
https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.1/administration/GUID-50C36862-A29D-48FA-8CE7-697E64E10E37.html

Main article for more information:

https://dmware.nl/2023/12/renewing-nsx-t-localmanager-certificates/

After implementation test:

https://.../api/v1/trust-management/certificates/

Search the cert ID and section "used_by" should represent where the cert is in use.
NSX-T randomly stopped services

If you experining the SSH unavaliability on the NSX-T, probably one of the reason is that SSH serivece not running,

To start SSH servces on the NSX-V, you have to use vCenter to havedirect access to the NSX applience.
VeloCloud initial CLI setup

CLI Activation procedure (undocumented):

reset_config.sh - reset the SDWAN unit configuration to factory default.

set_wan_config.sh SFP1 STATIC 172.16.0.10 255.255.255.0 172.16.0.254 -V XXX - setup IP address on the SFP1 for vlan XXX.

activate.py -s vcoYYY-syd1.velocloud.net XXX-YYY-ZZZ-RRR - activate velocloud unit.

Addidional useful CLI commands:

debug.py --link - provides status of the interfaces and IP addresses assigned.

debug.py --ha verp - check HA status.

tcpdump.sh -i any host IPADDRESS - capturing traffic on the SDWAN unit based on cryterias source/destination

tcpdump.sh -nNvi sfp1.XXX proto icmp- capturing ICMP traffic on the interface.

grep -i /var/log/mgd.log - check recent logs

ifconfig sfp1.XXX - check interface configuration.

/opt/vc/bin/vc_procmon stop - ???.
NSX-V DB or disk stave issue
This article related with NSX-V upgrade error appearing during image uploading.

We are goinig to discuss quickly 2 error possible could appear:

"Cannot continue upgrade due to errors : Large database table. There are some tables with 5215948 entries, but the recommended table size is 5000000. We recommend running a database full vacuum before proceeding with upgrade. Upgrade aborted.. Please correct before proceeding."
OR
```
"Cannot continue upgrade due to errors : Insufficient disk space. 
Database disk usage is at 87%, but it should be less than 70%. 
We recommend running a database full vacuum before proceeding with upgrade. 
Upgrade aborted.. Please correct before proceeding."
```
If you have NSX-V cross vCenter, the upgade possible will be stalled on both.

During

TRUNCATE TABLE job_instance_task_instances, task_instance_task_data, task_instance_task_output, task_instance, task_task_init_data, task, task_policy, task_target, job_instance_job_output, job_instance, job_data_task_dependency_map, task_dependency_tasks, dependent_task,job_data, job_schedule, task_dependency, housekeeping_module;
NSX-v Apache Log4j voulnerability patching steps

This article will discuss critical vulnerabilities in Apache Log4j identified by CVE-2021-44228 and CVE-2021-45046 in VMware NSX-V.

Overall steps desctibed on the VMware official page, however, I had to search some details separatly to apply and test the patch.

Lets go trough the steps:

PRE-CHANGE ACTIVITIES:

1. Take a backup of your NSX-V Manager. This is always important step. One day it will save your live :), so please take time and do it properly.

2. Download the patch file from official VMware website signed_bsh_fix_log4j.encoded

CONFIRMING VOULNERABILITY IS PRESENT:

To confirm the voulnerability is present, we need to initiate test attack to the NSX-V manager.

Voulnerabiluty code will be sent by POSTMAN and result of the NSX-V behavior will be caprured by TCPDUMP running directly on the NSX-V manager.

1. SSH to the NSX-V to the root level (Engineering mode).

Additional information how to login in Engineering mode could be found here: https://austit.com/faq/324-nsx-engeneering-mode

If you experiencing isssues with SSH connectivity to the NSX-V manager, some troubleshooting steps could be found here: https://austit.com/faq/353-nsx-v-enable-ssh

2. Run next tcpdump command:
tcpdump -i lo -s 1500 -XX port 389

3. Using POSTMAN, post REST API to https://<NSX-Manager-IP>/api/2.0/services/securitygroup/globalroot-0

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body:
<securitygroup>

</securitygroup>

4. My result after th POST request on the POSTMAN:

securitygroup-17

5. My result fot TCPDUMP automatically appeared on the screen (patched system should not capture any traffic):

12:10:01.529110 IP localhost.36158 > localhost.ldap: Flags [S], seq 724932146, win 43690, options [mss 65495,nop,nop,sackOK,nop,wscale 7], length 0
0x0000: 0000 0000 0000 0000 0000 0000 0800 4500..............E.
0x0010: 0034 551f 4000 4006 e7a2 7f00 0001 7f00 .4U.@.@.........
0x0020: 0001 8d3e 0185 2b35 9632 0000 0000 8002...>..+5.2......
0x0030: aaaa fe28 0000 0204 ffd7 0101 0402 0103...(............
0x0040: 0307 ..
12:10:01.529138 IP localhost.ldap > localhost.36158: Flags [R.], seq 0, ack 724932147, win 0, length 0
0x0000: 0000 0000 0000 0000 0000 0000 0800 4500..............E.
0x0010: 0028 0000 4000 4006 3cce 7f00 0001 7f00 .(..@.@.<.......
0x0020: 0001 0185 8d3e 0000 0000 2b35 9633 5014.....>....+5.3P.
0x0030: 0000 61a2 0000 ..a...

PATCHING SYSTEM:

Patching will be applied over POSTMAN REST API.

1. Using POSTMAN, post REST API to https://<NSX-Manager-IP>/api/1.0/services/debug/script

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body:
Select 'Binary' as body type and attach the file signed_bsh_fix_log4j.encoded

4. My result after th POST request on the POSTMAN is "1".

Starus 200 OK.

CONFIRMING NO VOULNERABILITY anymore:

To confirm the voulnerability patch was installed successfuly, we need to initiate test attack to the NSX-V manager.

Voulnerabiluty code will be sent by POSTMAN and result of the NSX-V behavior will be caprured by TCPDUMP running directly on the NSX-V manager.

1. SSH to the NSX-V to the root level (Engineering mode).

Additional information how to login in Engineering mode could be found here: https://austit.com/faq/324-nsx-engeneering-mode

If you experiencing isssues with SSH connectivity to the NSX-V manager, some troubleshooting steps could be found here: https://austit.com/faq/353-nsx-v-enable-ssh

2. Run next tcpdump command:
tcpdump -i lo -s 1500 -XX port 389

3. Using POSTMAN, post REST API to https://<NSX-Manager-IP>/api/2.0/services/securitygroup/globalroot-0

Authentication: Basic Auth (Username: admin)

Headers: Content-Type - application/xml

Body:
<securitygroup>

</securitygroup>

4. My result after th POST request on the POSTMAN:

Status: 404 Not Found

5. My result fot TCPDUMP shows no output.

Hope it helps, HAPPY PATCHING.
VMware NSX-T Avi Networks Load Balancer Installation and initial configuration

This article will discuss VMware NSX-T Avi Networks Load Balancer Installation and initial configuration.

Let's start with simple OVA deployment, assuming you already download the controller-21.1.1-9045.ova file:

1. Start deploying the downloaded image:

2. Select friendly name and appropreate folder.

3. Select compute and image validation will happened.

4.
NSX-T Password expired issue

Just recently connected to my test NSX-T environment and found that my password is espired.

Actually it is admin credentials. See picture below:

Fix is eazy, just SSH to the host and update the admin passwrd.

You are required to change your password immediately (password aged)
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for admin.
(current) UNIX password:

New password:
Retype new password:

After updating the password over CLI, you can successfuly login over UI.

Password expiration is 90 days by default.

To avoind this issue in the future we can extend expiration dates using next command:

set user admin password-expiration 9999

To check the expiration date:
get user admin password-expiration

Thu Nov 09 2021 UTC 00:36:18.240
Password expires 9970 days after last change. Current password will expire in 9970 days.
NSX-V to NSX-T Migration: universal objects

We are planning to process with NSX to vSphere to NSX-T migration.
NSX-T issue: Some appliance components are not functioning properly

One of the NSX-T nodes reporting "Some appliance components are not functioning properly" and web page is down.
Deploy OVF from datastore

Deploying OVF file is straight forward process.

But if you have slow link to the managed environment, how you can deploy OVF from internal storage.

Lets assume you can copy/get the OVF file to the datastore. How we can point the OVF deployment to the file located in the Datastore?

So, just login to the vCenter or ESXi host to the https://HOST/folder and copy the url for file.

Another issue you could face is that OVF deloyment like to have ".ovf" at the end.

Let's put tdo the trick and add "&f.ovf" at the end.

Now we can continue our OVF deployment...
ICMP issue troubleshooting between esxi guests

Issue recently appeared that 2 guests out of 50 not response to ICMP from another vCenter VMs.

Source and destination on the same proadcast domain stratched accross all vCenters.

All VMs in the same vCenter can communicate with no issues.

Just 2 VM experiencing issues.

MAC address appeared on the source VM.

Checked mac address table on all equipements, looks ok.

Next step to do packetcap on vmnic of destination VM.
NSX Edge Unable to Resolve Hostname

This acricle based on VMware KB 68035: https://kb.vmware.com/s/article/68035

So, ESG installed, however having error under log: "Error resolving hostname; host='vrli.domain.local'

As article said, going to API to the NSX:

1. Using Postman, find all your ESGs:

GET https://NSX-Manager/api/4.0/edges

Grab edge ID: in my case "edge-1"

2.

PUT https://NSX-Manager/api/4.0/edges/{edgeId}/dnsclient

Body:

(USE KB ARTICELE)
HTTP Result Codes:
204 NO CONTENT

3. Check settings applied:

GET https://NSX-Manager/api/4.0/edges/{edgeId}/dnsclient
Nested Ubuntu KVM on esxi
Here I will collect main steps to deploy UbuntuKVM on esxi for NSX-T deployment for testing purposes.

1. Install NSX-T version 2.4.1

2. Install Ubuntu 16.04.2 as most compatable for NSX-T. https://docs.vmware.com/en/VMware-NSX-T-Data-Center/2.4/installation/

3. Enable SSH root for Ubuntu:
```
sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
```
sudo service ssh restart4. Change interface to support trunk if required:

auto ens160.22
iface ens160.22 inet static
vlan-raw-device ens160

5. To validate that you have properly enable VHV, you run the following command:

egrep '(vmx|svm)' /proc/cpuinfo

5.1 Another command you can also run to check if the VM has been properly configure is the following:

kvm-ok

Output should be:

INFO: /dev/kvm exists
KVM acceleration can be used

6. Install next packets on the Ubuntu if next error appeared on host adding:

ERROR:

Failed to install software on host. Unresolved dependencies : ['libunwind8', 'libgflags2v5', 'libgoogle-perftools4', 'traceroute', 'python-mako', 'python-simplejson', 'python-unittest2', 'python-yaml', 'python-netaddr', 'libprotobuf9v5', 'libboost-chrono1.58.0', 'libgoogle-glog0v5', 'dkms', 'libboost-date-time1.58.0', 'libleveldb1v5', 'libsnappy1v5', 'python-gevent', 'python-protobuf', 'ieee-data', 'libyaml-0-2', 'python-linecache2', 'python-traceback2', 'libtcmalloc-minimal4', 'python-greenlet', 'python-markupsafe', 'libboost-program-options1.58.0', 'python-openssl']
```
apt-get install qemu-kvm 
apt-get install libvirt-bin 
apt-get install virtinst 
apt-get install virt-manager 
apt-get install virt-viewer
apt-get install ubuntu-vm-builder
apt-get install bridge-utils 
apt-get install libunwind8 libgflags2v5 libgoogle-perftools4 traceroute python-mako python-simplejson python-unittest2 python-yaml python-netaddr libprotobuf9v5 libboost-chrono1.58.0 libgoogle-glog0v5 dkms libboost-date-time1.58.0 libleveldb1v5 libsnappy1v5 python-gevent python-protobuf ieee-data libyaml-0-2 python-linecache2 python-traceback2 libtcmalloc-minimal4 python-greenlet python-markupsafe libboost-program-options1.58.0
```
NSX-T v2.4.1 Installation

Why NSX-t v2.4.1: I am currently want to stick to VMware Validated Design VVD 5.1

1. Log in to the vCenter via vSphere client, right click the cluster and click “Deploy OVF Template…” of NSX-T.

2. Answer all the quiestions and deploy the instance.

3. Login to the instance.

4. Add vCenters as compute manager. Go to Fabric -> Computer Managers -> Add

5. Configure NSX on ESXi or KVM. Go to Fabric -> Nodes. Add host to the Nodes.

GLOSSARY:

N-VDS
N-VDS is a new construct which comes with NSX-T. It is like a distributed switch within vSphere, but managed entirely through NSX-T. Because of the fact that NSX-T supports multiple entities, like hypervisors, clouds and so on, it was necessary to use a construct that was not bound to vSphere. So an N-VDS can be created and connected to ESXi hosts, KVM-hosts, bare-metal servers and clouds.

N-VDS’s use uplinks to connect to the physical world. The creation of an N-VDS is done when creating a Transport Zone and the name cannot be changed afterwards (or at least I have not found a way yet).
NSX controller reboot

Sometimes you can see errors that one of the NSX controllers not healthy.

First, try to SSH to the applience and check logs.

Also, good practice to restart applience:

restart system
NSX Firewall Rules investigation

111
vCenter Enhanced Authentication Plugin Not Working

ISSUE: When navigating to your vCenter Web Client (Flash) page, the browser no longer allows you to check the “Use Windows session authentication” box.

FIX: need to navigate to https://vmware-plugin:8094 and Advanced and then Proceed.

Note: For this who are interested, the reason your machine is able to resolve https://vmware-plugin is because during the installation of the plugin your hosts file is manipulated to point vmware-plugin to 127.0.0.1
NSX controllers /var/log usage - 100%

NSX Manager version: 6.4.1
Appeared in earlier versions too.

Issue: The /var/log is full with 100% utilization for all 3 Controllers.
root@nsx-controller [ /var/log ]# df -h | grep log
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 4.8G 4.8G 0 100% /var/log

Found that syslog.1 file occupied 4 GB of size, as a working scenario syslog file should get rotate the log file, but in our case it didn't happened, this found to be a match of a know issue in NSX 6.4.1

root@nsx-controller [ /var/log ]# ls -lah | grep syslog
-rw-r----- 1 root adm 0 Sep 18 04:03 syslog
-rw-r----- 1 root adm 4.2G Sep 22 02:39 syslog.1
-rw-r----- 1 root adm 3.7M Aug 15 22:22 syslog.10.gz
-rw-r----- 1 root adm 3.7M Aug 15 20:59 syslog.11.gz
-rw-r----- 1 root adm 3.7M Aug 15 19:36 syslog.12.gz
-rw-r----- 1 root adm 3.7M Aug 15 18:13 syslog.13.gz

Solution 1: The permanent fix for this solution is to upgrade to NSX 6.4.2

Solution 2:
- Move the syslog.1 file to datastore for a copy of it and made the file to null.
- Make backup of existing rsyslog file cp /etc/logrotate.d/rsyslog /etc/logrotate.d/rsyslog.backup01092018
- Edit /etc/logrotate.d/rsyslog existing value "usr/bin/systemctl reload syslog.service > /dev/null" to "/usr/bin/systemctl restart syslog.service > /dev/null"
- Restart rsyslog: systemctl restart rsyslog
- Check syslog now not zerro and has content
NSX Engeneering mode

This is basically a root bash shell on the underlying Linux based appliance. From here, system configuration files and scripts as well as most normal Linux functions can be accessed.

Engineering Mode: The authorized NSX Manager system administrator is requesting a shell which is able to perform lower level unix commands/diagnostics and make changes to the appliance. VMware asks that you do so only in conjunction with a support call to prevent breaking your virtual infrastructure. Please enter the shell diagnostics string before proceeding.Type Exit to return to the NSX shell. Type y to continue:”
VMware recommends to take full backup of the system before performing any changes after logging into the Tech Support Mode.

Login to NSX Manager Engeneering mode:
The NSX Manager contains many tools to help customers in conjunction with Global Support Services to resolve operational issues. The NSX for vSphere 6.x product features a customized command line interface that covers most of the basics that the user interface does and a little bit more.
- Login to NSX Manager shell
- show controller list all
- Go to enable mode
- enter engineering mode by typing st eng
- Type the shell diagnostics string: "IAmOnThePhoneWithTechSupport"

Find NSX controllers engeneering password:
- /home/secureall/secureall/sem/WEB-INF/classes/GetNvpApiPassword.sh controller-4

NSX controller Engineering Mode:
- Login to NSX controller shell
- Go to enable mode
- enter engineering mode by typing : debug os-shell
- use password string from previouse step
Enable RSS in the ESXi host

To take full advantage of the Intel® Converged Network Adapter's capabilities, enable Receive Side
Scaling (RSS) in the ESXi host to balance the CPU load across multiple cores. Refer to the VMware
performance paper VXLAN Performance Evaluation from VMware 5.1 for performance results
with Intel® Converged Network Adapter X520 with RSS enabled:

ESXi VSwitch | Firewall Performance Testing

Enable pnic RSS / mq:
in shell:
vmkload_mod -u ixgbe
vmkload_mod ixgbe RSS=”4,4,4,4,4,4″

in .vmx or advanced config:
ethernetX.pnicFeatures = “4”

allow multiple vnic thread for single vm:
ethernetX.ctxPerDev = “1”
sources:
http://blogs.vmware.com/performance/2015/04/network-improvements-vsphere-6-boost-performance-40g-nics.html

Page 1 of 2

Google AdSence

AUST IT - Computer help out of hours, when you need it most.

Find out why we do it for less.

About

AUST IT will help you resolve any technical support issues you are facing onsite or remotely via remote desktop 24/7. More...

Contacts

Reservoir, Melbourne,
3073, VIC, Australia

Phone: 0422 348 882

This email address is being protected from spambots. You need JavaScript enabled to view it.

Sydney: 0481 837 077

Connect

Join us in social networks to be in touch.

Renewing NSX-T Local certificates

NSX-T randomly stopped services

VeloCloud initial CLI setup

NSX-V DB or disk stave issue

NSX-v Apache Log4j voulnerability patching steps

VMware NSX-T Avi Networks Load Balancer Installation and initial configuration

NSX-T Password expired issue

NSX-V to NSX-T Migration: universal objects

NSX-T issue: Some appliance components are not functioning properly

Deploy OVF from datastore

ICMP issue troubleshooting between esxi guests

NSX Edge Unable to Resolve Hostname

Nested Ubuntu KVM on esxi

NSX-T v2.4.1 Installation

NSX controller reboot

NSX Firewall Rules investigation

vCenter Enhanced Authentication Plugin Not Working

NSX controllers /var/log usage - 100%

NSX Engeneering mode

Enable RSS in the ESXi host

Popular tag

Who's Online

Google AdSence

AUST IT - Computer help out of hours, when you need it most.

About

Contacts

Connect

Newsletter