VCF9 – Supervisor with VPC networking and Avi

With the introduction of VCF 9, Broadcom has changed the way we look at Private Cloud. With Containers now being treated as “first class citizens” and VCF Operations centralising the management of entire VCF Fleets, we are now able to create a “hyperscaler-like” user experience.

Part of this new Private Cloud user experience is done by activating vSphere Supervisor, which allows you to then utilise VCF Automation as the main consumption layer for the VCF platform, providing self-service capabilities in a multi-tenant environment.

Another requirement for this cloud experience is the use of VPCs, which provide isolation of the data and infrastructure between tenants.

The steps outlined in this post are the ones I took to successfully enable vSphere Supervisor on my lab, using VPC networking and Avi Load balancer. For this I used VCF version 9.0.1, Avi version 31.1.2-9193, and a Microsoft CA (the SDDC Manager CA option is also supported).

Pre-Install Requirements

vCenter

Configure Centralized Connectivity

On the target vCenter server, navigate to Inventory (network tab), then select the vCenter instance on the side menu.
Navigate to Networks / Network Connectivity and confirm that Centralized Connectivity is configured.

(Optional/Recommended) Create a test VPC and connect to a test VM. Confirm that North-South connectivity is available.

Create a Local Content Library

A Local Content Library is required to store Avi SE images.

Create a vSphere Zone

Navigate to Inventory and select the vCenter instance on the side menu.
Navigate to Configure / vSphere Zones and then click on ADD NEW VSPHERE ZONE.

Follow the wizard to create a new zone.

NSX

Avi SE Management Network

Tier 0 Gateway configured and operational (this is done during the Centralized Connectivity configuration).
Create Tier-1 Gateway, linked to the Tier-0 Gateway.
Create and overlay Segment for the Avi SE Management, connected to the Tier-1 Gateway. Enable DHCP on this segment.

(Optional/Recommended) Connect a test VM to the Avi SE Management segment and confirm that North-South connectivity is available.

Microsoft CA

Configure a Microsoft CA and integrate it with your VCF Instance on VCF Operations. Make sure all components are using an Enterprise CA issued SSL certificate.

Configure a Certificate Authority for VMware Cloud Foundation

Avi Deployment

To deploy Avi on a VI Workload Domain, first follow the Avi Bundle Upload procedure to upload the Avi installer into SDDC Manager (first section only):

Deploying Avi Load Balancer in VCF – Planning and Preparation

After the upload is complete, navigate to the target Workload Domain on SDDC Manager and then select Actions / Deploy Avi Load Balancer.

Follow the wizard to deploy Avi.

Configure an SSL Certificate on Avi

Use the following procedure to configure Avi with a Certificate from the Microsoft CA (steps 3 to 7):

Deploying Avi Load Balancer in VCF – Creating and Deploying Avi Load Balancer Portal Certificate

The procedure documented by Broadcom utilises OpenSSL as the Certificate Authority, however the same steps apply for a Microsoft CA.
When both Avi and NSX are using certificates from the Enterprise CA, the Root Certificate should be already on both, so there is no need to export the Avi root certificate and import it into NSX (step 8).

Prepare Avi for the vSphere Supervisor activation

When opening the Avi portal for the first time, follow the wizard to complete the initial setup.

Navigate to Administration / Licensing and make sure that a valid Enterprise Tier licence is assigned.

Create temporary NSX Credentials

The deployment of Avi through SDDC Manager automatically creates a Service Account to integrate Avi with NSX. However, during my testing it appears that those credentials do not have sufficient permissions for the Supervisor activation. A workaround for this is to configure temporary NSX Admin credentials on Avi and then reconfigure the integration with the Service Account after the activation is finished.

On the Avi portal, navigate to Administration / User Credentials and then click Create.

Enter the NSX Admin account details and then click Save.

Create a new Cloud

On the Avi portal, navigate to Infrastructure / Clouds and then click Create>NSX Cloud.

Follow the form to create a new NSX Cloud.

Under NSX, make sure to use the NSX Admin credentials.

Under Management Network, select the environment’s Overlay Transport Zone, as well as the Tier-1 Gateway and Avi SE Management segment that were created as pre-install requirements.
Make sure that Enable VPC Mode is selected.

Under vCenter Servers, use the system-created Service Account and select the Local Content Library that was created as a pre-install requirement.

After configuring the NSX Cloud, confirm that its status shows a green dot.

(Recommended) Test Avi Load Balancer

At this point, confirm that Avi works within the environment.

Connect a test VM on a test Private VPC.
On NSX, edit the VPC, expand Advanced Settings and select Enable for Avi Load Balancer.

Confirm on Avi that a new VCF Context appears under Infrastructure / Cloud Resources / VRF Context (this might take a few minutes).

Using this VRF Context on Avi:

Configure a new test Pool.
Configure a new test VIP allocated in the PUBLIC network. That’s the “VPC External IP Blocks” defining when first configuring VPC Network Connectivity on vCenter.

Create a new Virtual Service to access the test VM.
To simplify my testing, I deployed a simple web server on the test VM, allowing for a HTTP (port 80) Virtual Service that can be accessed via a browser.

When the Virtual Service is first deployed, it will deploy 2 Avi Service Engines by default. This process might take a few minutes.

Check that the test VM is accessible through Avi.
Clean up / Delete any test objects before proceeding.

Register an Enforcement Point

Register an Enforcement Point on NSX using the following command, on a terminal that is accessible by NSX (make sure to replace all < > variables):

curl -k -u 'admin:<NSX MANAGER PASSWORD>' --location --request PUT 'https://<NSX MANAGER>/policy/api/v1/infra/alb-onboarding-workflow' --header 'X-Allow-Overwrite: True' --header 'Content-Type: application/json' --data-raw '{"owned_by": "LCM", "cluster_ip": "<AVI CONTROLLER IP>", "infra_admin_username" : "admin", "infra_admin_password" : "<AVI PASSWORD>", "default_cert": false}'

Enable Supervisor

On vCenter navigate to Menu / Supervisor Management then click the Get Started button.

Select VCF Networking with VPC and then click NEXT (ignore the warning about assigned Content Library. A new Subscribed Content Library is automatically created during this process).

Follow the Wizard to configure the Supervisor and click FINISH in the end.

Notes:

Place the Supervisor on the vSphere Zone that was created as a pre-install requirement.
Place the Supervisor on a VLAN that is routable to the VCF Management components and assign at least 5 IP addresses to it.