Switch Ports Overview¶
Switched Ethernet ports can be used for High Availability (HA), but there is one limitation when configuring switchports for HA. Because the uplinks from the switch to the SoC are always up, failover is only effective in scenarios where a system completely dies. If a single switch interface goes down, CARP will not be able to detect this properly so the PRIMARY will remain PRIMARY on any switch interfaces that drop link.
The SECONDARY will also consider itself PRIMARY of the network associated to the switch link that dropped. In this situation, LAN clients will likely go through the SECONDARY but will not be able to get online if NAT is utilized with a WAN CARP IP. It’s possible to NAT to the WAN interface IP to get around this but it can cause state issues during failover.
For best results, use the ports on a Network Interface expansion card. When configured correctly, the discrete ports of the add-in NIC will provide full redundancy and failover in the event of a network outage or scheduled maintenance.
For HA configuration instructions, visit the High Availability page.
ix2 and ix3 (switch uplink ports 9 and 10), are configured as a load-balanced LAGG. This provides an aggregate uplink capable of 5Gbps for ethernet switchports ETH1-8. This is further demonstrated in the diagram below:
When data is received on ETH1-8, the switch is capable of utilizing LAGG to determine whether that data should be sent out of PORT 9 or PORT 10. That data then passes over one of two 2.5Gbps switch links (PORT 9/10) to the SoC. Data coming from PORT 9 has a direct line to ix2 and data from PORT 10 has a direct line to ix3.
pfSense® Plus LAGG will then take in traffic from both ix2 and ix3 as though it came in on a single interface, lagg0. The same concept applies to traffic sourcing from the pfSense® Plus LAGG to the switch LAGG.
By default, ETH1 on the the switch is configured as a WAN interface and ETH2-8 are configured as the LAN interface. These eight switchports are customizable and each can be configured to act as an independent interface. For example, all of these configurations are possible:
ETH1-8 dedicated as a LAN switch
ETH1-4 configured as a switch for LAN network A and ETH5-8 configured as a switch for LAN network B
ETH1-8 configured as individual network interfaces
ETH1 configured for WAN A, ETH2 configured for WAN B, ETH3 configured for LAN network A, ETH4-6 configured as a switch for LAN network B, and ETH8 configured as a H/A sync port.
These scenarios are possible by utilizing VLANs. Each of the switchports (ETH1-8 and PORT9-10) are VLAN aware interfaces. They are capable of functioning like a standard access or trunk port:
- Access Port:
Adds a VLAN tag to inbound untagged traffic
- Trunk Port:
Allows tagged traffic containing specified VLAN IDs
In the default configuration, two VLANs are used to create the ETH1 WAN interface and ETH2-8 LAN interface:
ETH1-8 are configured to act as Access ports.
When data comes into the ETH1 interface, a VLAN tag of 4090 is added to the ethernet frame.
When data comes into interfaces ETH2-8, a VLAN tag of 4091 is added to the ethernet frame.
PORT9-10 are configured to act as Trunk ports.
By default, only ethernet frames containing a VLAN tag of 4090 or 4091 are allowed over the trunk.
Each VLAN configured on the switch uses the LAGG interface as its parent interface. For example, the default interface assignment for WAN and LAN:
This means vlan4090 and vlan4091, as well as any other VLANs created for the switch, all share the same 5Gbps LAGG uplink across two 2.5Gbps links. The visual below demonstrates how the VLAN tagging works along with the traffic flow:
Traffic leaving and entering the ETH1-3 interfaces in the visual above are untagged. Devices sending/receiving traffic over these ports do not need to be VLAN aware. The VLAN tagging that occurs within the switch is completely transparent to clients. It’s used solely for segmenting switch traffic internally.
Aside from being able to specify whether a switchport should act as an access or trunk port, it’s also possible to disable 802.1q VLAN mode. When this is done, a third mode called Port VLAN Mode is enabled. In this mode, any and all VLAN tags are allowed on all ports. No VLAN tags are added or removed. Think of it as a dummy switch that retains VLAN tags on frames, if present. This mode is useful when you have numerous VLANs on your network and want to physically segment the switch, while allowing the same VLANs on all segments of the switch.
In Port VLAN Mode, rather than specifying which interfaces are associated to a VLAN, you can specify which physical ports form a switch. For example, if I want to create two physical switches that act as individual dummy switches - allowing tagged or untagged traffic, I could configure Port VLAN Mode like so:
// UPLINKS VLAN group 9, Port 9, Members 1,2,3,4,10 VLAN group 10, Port 10, Members 1,2,3,4,9 // SWITCH-A VLAN group 1, Port 1, Members 2,3,4,9,10 VLAN group 2, Port 2, Members 1,3,4,9,10 VLAN group 3, Port 3, Members 1,2,4,9,10 VLAN group 4, Port 4, Members 1,2,3,9,10 // SWITCH-B VLAN group 5, Port 5, Members 6,7,8 VLAN group 6, Port 6, Members 5,7,8 VLAN group 7, Port 7, Members 5,6,8 VLAN group 8, Port 8, Members 5,6,7
With this configuration in place, ETH1-8 now function like so:
// SWITCH-A PORT 1 = ETH1 PORT 2 = ETH2 PORT 3 = ETH3 PORT 4 = ETH4 PORT 9 = UPLINK 1 PORT 10 = UPLINK 2 // SWITCH-B PORT 5 = ETH5 PORT 6 = ETH6 PORT 7 = ETH7 PORT 8 = ETH8
ETH1-4 can talk to each other and to the LAGG uplink. PORT9-10 are members of this switch…this is required for this switch to have uplink to pfSense® Plus.
ETH5-8 can talk to each other but because PORT9-10 are not included as members, clients connecting to ETH5-8 can only talk to other clients on ETH5-8. They will not be able to reach the SoC where ix2 and ix3 are defined, so they never reach the pfSense® Plus softawre. This can be useful if you want a device other than pfSense® Plus to act as the primary uplink for those connected clients.
Since WAN and LAN are assigned to lagg0.4090 and lagg0.4091, if Port VLAN Mode is enabled, be sure to update the LAN and WAN interface assignment to reference the appropriate VLAN. Also remember to create the new VLANs with lagg0 as the parent interface.
If Port VLAN Mode is being used to handle untagged traffic, the LAGG0 interface should be added, enabled, and configured under Interface Assignments.
For more information on how to configure the switch ports, see Configuring the Switch Ports.