XG-7100 Switch Overview

High Availability

Switched Ethernet ports can be used for High Availability (HA), but there is one limitation when configuring switchports for HA. Because the uplinks from the switch to the SoC are always up, failover is only effective in scenarios where a system completely dies. If a single switch interface goes down, CARP will not be able to detect this properly so the PRIMARY will remain PRIMARY on any switch interfaces that drop link.

The SECONDARY will also consider itself PRIMARY of the network associated to the switch link that dropped. In this situation, LAN clients will likely go through the SECONDARY but will not be able to get online if NAT is utilized with a WAN CARP IP. It’s possible to NAT to the WAN interface IP to get around this but it can cause state issues during failover.

For best results, use the ports on a Network Interface expansion card. When configured correctly, the discrete ports of the add-in NIC will provide full redundancy and failover in the event of a network outage or scheduled maintenance.

For HA configuration instructions, visit the High Availability page.

Switch LAGG

ix2 and ix3 (switch uplink ports 9 and 10), are configured as a load-balanced LAGG. This provides an aggregate uplink capable of 5Gbps for ethernet switchports ETH1-8. This is further demonstrated in the diagram below:

../_images/xg-7100-1u-switch_2.png

When data is received on ETH1-8, the switch is capable of utilizing LAGG to determine whether that data should be sent out of PORT 9 or PORT 10. That data then passes over one of two 2.5Gbps switch links (PORT 9/10) to the SoC. Data coming from PORT 9 has a direct line to ix2 and data from PORT 10 has a direct line to ix3.

pfSense® LAGG will then take in traffic from both ix2 and ix3 as though it came in on a single interface, lagg0. The same concept applies to traffic sourcing from the pfSense LAGG to the switch LAGG.

Switch VLANs

By default, ETH1 on the the switch is configured as a WAN interface and ETH2-8 are configured as the LAN interface. These eight switchports are customizable and each can be configured to act as an independent interface. For example, all of these configurations are possible:

  • ETH1-8 dedicated as a LAN switch

  • ETH1-4 configured as a switch for LAN network A and ETH5-8 configured as a switch for LAN network B

  • ETH1-8 configured as individual network interfaces

  • ETH1 configured for WAN A, ETH2 configured for WAN B, ETH3 configured for LAN network A, ETH4-6 configured as a switch for LAN network B, and ETH8 configured as a H/A sync port.

These scenarios are possible by utilizing VLANs. Each of the switchports (ETH1-8 and PORT9-10) are VLAN aware interfaces. They are capable of functioning like a standard access or trunk port:

Access Port:

Adds a VLAN tag to inbound untagged traffic

Trunk Port:

Allows tagged traffic containing specified VLAN IDs

In the default configuration, two VLANs are used to create the ETH1 WAN interface and ETH2-8 LAN interface:

WAN

VLAN 4090

LAN

VLAN 4091

ETH1-8 are configured to act as Access ports.

  • When data comes into the ETH1 interface, a VLAN tag of 4090 is added to the ethernet frame.

  • When data comes into interfaces ETH2-8, a VLAN tag of 4091 is added to the ethernet frame.

PORT9-10 are configured to act as Trunk ports.

  • By default, only ethernet frames containing a VLAN tag of 4090 or 4091 are allowed over the trunk.

Each VLAN configured on the switch uses the LAGG interface as its parent interface. For example, the default interface assignment for WAN and LAN:

WAN

lagg0.4090

LAN

lagg0.4091

This means vlan4090 and vlan4091, as well as any other VLANs created for the switch, all share the same 5Gbps LAGG uplink across two 2.5Gbps links. The visual below demonstrates how the VLAN tagging works along with the traffic flow:

../_images/xg-7100-1u-switch_3.png

Note that traffic leaving and entering the ETH1-3 interfaces in the visual above are untagged. Devices sending/receiving traffic over these ports do not need to be VLAN aware. The VLAN tagging that occurs within the switch is completely transparent to clients. It’s used solely for segmenting switch traffic internally.

Aside from being able to specify whether a switchport should act as an access or trunk port, it’s also possible to disable 802.1q VLAN mode. When this is done, a third mode called Port VLAN Mode is enabled. In this mode, any and all VLAN tags are allowed on all ports. No VLAN tags are added or removed. Think of it as a dummy switch that retains VLAN tags on frames, if present. This mode is useful when you have numerous VLANs on your network and want to physically segment the switch, while allowing the same VLANs on all segments of the switch.

In Port VLAN Mode, rather than specifying which interfaces are associated to a VLAN, you can specify which physical ports form a switch. For example, if I want to create two physical switches that act as individual dummy switches - allowing tagged or untagged traffic, I could configure Port VLAN Mode like so:

// UPLINKS
VLAN group 9, Port 9, Members 1,2,3,4,10
VLAN group 10, Port 10, Members 1,2,3,4,9

// SWITCH-A
VLAN group 1, Port 1, Members 2,3,4,9,10
VLAN group 2, Port 2, Members 1,3,4,9,10
VLAN group 3, Port 3, Members 1,2,4,9,10
VLAN group 4, Port 4, Members 1,2,3,9,10

// SWITCH-B
VLAN group 5, Port 5, Members 6,7,8
VLAN group 6, Port 6, Members 5,7,8
VLAN group 7, Port 7, Members 5,6,8
VLAN group 8, Port 8, Members 5,6,7

With this configuration in place, ETH1-8 now function like so:

// SWITCH-A
PORT 1 = ETH1
PORT 2 = ETH2
PORT 3 = ETH3
PORT 4 = ETH4
PORT 9 = UPLINK 1
PORT 10 = UPLINK 2

// SWITCH-B
PORT 5 = ETH5
PORT 6 = ETH6
PORT 7 = ETH7
PORT 8 = ETH8

SWITCH-A

ETH1-4 can talk to each other and to the LAGG uplink. PORT9-10 are members of this switch…this is required for this switch to have uplink to pfSense.

SWITCH-B

ETH5-8 can talk to each other but because PORT9-10 are not included as members, clients connecting to ETH5-8 can only talk to other clients on ETH5-8. They will not be able to reach the SoC where ix2 and ix3 are defined, so they never reach pfSense. This can be useful if you want a device other than pfSense to act as the primary uplink for those connected clients.

Since WAN and LAN are assigned to lagg0.4090 and lagg0.4091, if Port VLAN Mode is enabled, be sure to update the LAN and WAN interface assignment to reference the appropriate VLAN. Also remember to create the new VLANs with lagg0 as the parent interface.

If Port VLAN Mode is being used to handle untagged traffic, the LAGG0 interface should be added, enabled, and configured under Interface Assignments.

Configuring the Switch

Switch Section

From the pfSense webGUI, there is a menu option called Switches under the Interfaces drop-down. This section contains switch specific configuration options.

../_images/xg-7100-1u-switch_4.png

Selecting Switches from the drop-down will bring up the Switch page with four sections:

System

../_images/xg-7100-1u-switch_5.png

Information on the Marvell 6000 switch

LAGGs

../_images/xg-7100-1u-switch_6.png

Information on members of the switch LAG

Ports

Information on switchport status and port names. If 802.1q is enabled, this section can also be used to specify the native VLAN ID for each port. The Port VID defined will be used to tag inbound untagged traffic.

../_images/xg-7100-1u-switch_7.png

802.1q enabled (default)

../_images/xg-7100-1u-switch_8.png

Port VLAN Mode

VLANs

Enable/Disable 802.1q VLAN mode. Configure VLAN access/trunk interfaces with 802.1q or configure port groups with Port VLAN Mode.

../_images/xg-7100-1u-switch_9.png

802.1q enabled (default)

../_images/xg-7100-1u-switch_10.png

Port VLAN Mode

Interfaces Section

There is also relevant configurations under Interfaces -> Assignments.

Interface Assignments

Under Interface Assignments, notice LAGG0 (UPLINK) is displayed as an available port but is not enabled in the list of interfaces. This is because the default configuration is only expecting VLAN tagged traffic so the VLAN child interface 4090 and 4091 are enabled instead.

../_images/xg-7100-1u-switch_11.png

VLANs

Under VLANs, the default WAN and LAN VLAN can be seen. Additional VLAN networks that will be used by the switch should be defined here with lagg0 as the parent interface.

Any additional VLAN interface added to the switch should also be added, enabled, and configured under Interface Assignments. Firewall rules will also be needed for new interfaces added.

../_images/xg-7100-1u-switch_12.png

LAGGs

Under LAGGs, the default lagg0 containing ix2 and ix3 can be seen. The lagg0 interface should not be modified.

../_images/xg-7100-1u-switch_13.png

Switch Configuration Examples

Dedicated LAN switch

In this scenario, SFP+ port ix0 will be configured as the WAN interface. ETH1-8 will be configured as a LAN switch.

For this specific example, I’ll perform the WAN interface reassignment over console. Re-assigning the WAN can be done from the webGUI as well.

This is what the default interface assignments look like on a XG-7100 without an addon NIC:

../_images/xg-7100-1u-switch_14.png

In this example, ix0 will be WAN, so select option 1 to re-assign WAN from lagg0.4090 to ix0:

../_images/xg-7100-1u-switch_15.png

No additional VLANs are needed for this, so enter n to continue.

Input ix0 as the new WAN interface name:

../_images/xg-7100-1u-switch_16.png

Input the same default LAN interface of lagg0.4091 for the LAN interface name and press Enter to complete the interface reassignment:

../_images/xg-7100-1u-switch_17.png

The interface assignments should show like this now:

../_images/xg-7100-1u-switch_18.png

At this point SFP+ port ix0 is now configured as the WAN interface. The LAN interface is still configured the same as the default. Next, the switch will need to be updated so that ETH1 (previously WAN) acts the same as ETH2-8. This will be done from the webGUI.

From the webGUI, pull up the Switch VLAN configuration under Interfaces -> Switches -> VLANs:

../_images/xg-7100-1u-switch_19.png

VLAN 4090 is no longer needed since WAN is dedicated to ix0 now. You can either select fa-trash on the row containing 4090 to delete this entry, or click fa-pencil to remove port 1 as a member:

../_images/xg-7100-1u-switch_20.png

For this example, I simply removed VLAN 4090 from the switch with fa-trash. Now edit the VLAN 4091 entry to include Member 1 as shown below:

../_images/xg-7100-1u-switch_21.png

Next, update the PVID for ETH1 so that it uses VLAN 4091 rather than the old VLAN 4090. To do this, click on the Ports tab and click on the 4090 Port VID to modify it:

../_images/xg-7100-1u-switch_22.png

Then click on Save:

../_images/xg-7100-1u-switch_23.png

At this point, everything should be configured properly. ETH1-8 will act as a single LAN switch. One final step that should be performed is to remove the old VLAN 4090 from pfSense. So far VLAN 4090 was only removed from the switch. To remove the old VLAN, go to Interfaces -> Assignments -> VLANs and use fa-trash on the 4090 row to remove this VLAN interface:

../_images/xg-7100-1u-switch_24.png

Two LAN switches

In this scenario, the LAN switch from the previous example will be split into two LAN switches.

A new LAN network should be created in pfSense first. Similar to the existing LAN interface, another VLAN interface should be used so the switch can segment traffic appropriately.

Create a new VLAN with lagg0 as the parent under Interfaces -> Assignments -> VLANs:

../_images/xg-7100-1u-switch_25.png

Once the VLAN has been created, it should look something like this:

../_images/xg-7100-1u-switch_26.png

Add, enable, and configure the VLAN interface under Interfaces Assignments:

../_images/xg-7100-1u-switch_27.png ../_images/xg-7100-1u-switch_28.png

Also create any necessary firewall rules under Firewall -> Rules.

Now that pfSense knows of this new VLAN network, configure the switch so that ETH1-4 use the new network. To do this, go to Interfaces -> Switches -> VLANs and click the Add Tag button. Input the VLAN tag for the new network (same as the VLAN ID configured in the previous steps) and add ETH1-4 and PORT9-10 (uplinks) as members. Be sure 9 and 10 are marked as tagged:

../_images/xg-7100-1u-switch_29.png

Once this is done, click the Save button. The final result should look like this:

../_images/xg-7100-1u-switch_30.png

Lastly, update the Port VIDs to use the new 4081 VLAN rather than 4091 on ETH1-4 and click Save:

../_images/xg-7100-1u-switch_31.png

Now ETH1-4 act as a switch for the VLAN 4081 LAN and ETH5-8 act as a switch for the VLAN 4091 LAN.

Trunking VLAN tagged traffic

For expanding on the previous example, let’s assume there is a management VLAN of 4000 where devices are already tagged on this VLAN prior to hitting pfSense. Devices on this VLAN may come through on ETH8 but there may also be untagged client traffic.

First, create the management VLAN of 4000 in pfSense using the same steps in the previous example (up to the switch configuration part). Next, add the VLAN to the switch under Interfaces -> Switches -> VLANs. ETH8 and PORT9-10 should be added as members and all three will be marked as tagged:

../_images/xg-7100-1u-switch_32.png

Once it’s added, the final result should look like this:

../_images/xg-7100-1u-switch_33.png

Untagged traffic on ETH8 will be assigned a VLAN ID of 4091. ETH8 and the uplinks will also accept traffic that has already been tagged with a VLAN ID of 4000 as well.