MC-LAG
Jan 06
MC-Lag
Everyone mostly know what MC-Lag does, for the benefit its a variant of LAG where the Down stream Devices share LAG interface on two Physical devices instead of One, I know its confusing let see a sample topology
Vqfx1 will see the upstream as 1 ae1 instead of two different VMX devices, this has its own advantage and the entire discussion is something out of scope of this blog post.
Blog Post Goal – Demonstrate MC-Lag on VMX and Quickly highlight the options Common and different in MC-LAG, a ready reference for someone going for an exam or a implementation.
Take-Away’s
-> ICCP is the protocol between the nodes
- Uses TCP/IP to replicates control plane and forwarding traffic between peers
- One stand-by and one active (active responsible status control)
- ICCP messages exchange configuration information to ensure both peers use correct LACP Parameters
- ICL-PL (interface between Mx1 and Mx2 ex) supports Multiple MC-Lag Between the peers so its recommended to be a AE.
-> ICL-PL HA
- A Keep-alive message is exchanged between MC-LAG peers which is recommended to be the management connection
- If ICL-PL fails, keep alive is still through the management connection and in that scenario, the stand-by brings down its local member link to avoid split-brain scenarios
-> MC-LAG Modes
- Active/Stand-by & Active-Active (QFX Only supports Active/Active MC-LAG Mode)
- A/A supports traffic distribution and if one link fails, the traffic will be through ICL-PL connection as ICL-PL will have all the VLANS allowed
- All Mac-Addresses are shared among MC-LAG Peers using ICCP
- Mcast Flooding happens on all the links if MC-LAG peers are members of VLANS associated with traffic
- IRB routing is possible
Phase-1 - Setting up ICCP (ICL Peers)
Two things here
-> If you are operating at VMX level, if you want this to work
– Configure Static ARP for IRB else ICCP will not establish
– Configure LACP Session-ID else LACP will not form
– I have seen LACP session getting initiated with different ID’s at different flaps which resulted in AE0 Flaps.
When Observed from vQFX, since its Active-standby, one interface will be attached, while the other is in distributing
Values common on both the Peers
Mc-ae-id <1>
Mode active-active
lacp active
lacp admin-key <1>
lacp system-id <01:01:01:01:01:01>
Different on Both the peers
Chassis-id 0
Status-control active
Regards Rakesh
Mar 14, 2018 @ 11:06:06
Hi Rakesh,
Really nicely articulated articles and simple to understand.
One question is that i have just installed EVE-NG and manage to import vqfx Res/PFEs, i am unable to configure xe-0/0/0 to xe-0/0/7, i am only able to configure em interfaces. can you throw some light that how to use xe-0/0/0….. interfaces on vqfx?
Thanks in advance
Regards
Badar
Mar 16, 2018 @ 17:08:29
Hi Badar,
Try with increasing the number of VCPU’s and total Ram, if it does not work then i would suggest you should start with Esxi and Direct Ova images instead of EVE. Eve on the whole is a good tool but if you are deploying vqfx and other VM images, you should consider Esxi and ova Deployment, i switched from Eve to Esxi for the same reason
May 20, 2018 @ 08:46:00
Hi Rakesh,
Nice work!! and my apologies for this verbose comment.
I am working on some what a similar setup but need to understand the underlaying infrastructure that you are using. What bridges are you using for MC-AE??.
As my understanding LACP does not work in virtio mode due to the fact that linux bridges are used and the work around would be to use OVS instead of linux bridges.
My ae21 bundle will fall on a OVS bridge for MC-AE but struggling to bring the links up. I am no expert on OVS but this lab exercise is a good way to know more about it.
ge-0/0/1.2732 up up aenet –> ae21.2732
ae21 up down
ae21.2732 up down bridge
My ICCP peer’s are established and the interfaces fall on a linux bridge. My initial attempt was to use LACP/OSPF for ICCP interfaces, but interfaces won’t come up if LACP is configured on Linux bridges and works like a charm if I remove LACP and put IP’s on the interfaces.
unit 0 {
family inet {
address 169.254.254.253/30;
}
}
But I am unable t bring LACP on OVS too and I would appreciate any help on this.
Config
vMX1
ge-0/0/1 {
gigether-options {
802.3ad ae21;
}
ae21 {
flexible-vlan-tagging;
multi-chassis-protection 169.254.254.254 {
interface ge-0/0/0;
}
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
system-id 00:00:00:00:00:02;
admin-key 1;
}
mc-ae {
mc-ae-id 2;
redundancy-group 1;
chassis-id 0;
mode active-active;
status-control active;
}
}
unit 2732 {
encapsulation vlan-bridge;
vlan-id 2732;
family bridge;
}
}
vMX2
ge-0/0/1 {
gigether-options {
802.3ad ae21;
}
}
ae21 {
flexible-vlan-tagging;
multi-chassis-protection 169.254.254.253 {
interface ge-0/0/0;
}
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
system-id 00:00:00:00:00:02;
admin-key 1;
}
mc-ae {
mc-ae-id 2;
redundancy-group 1;
chassis-id 1;
mode active-active;
status-control standby;
}
}
unit 2732 {
encapsulation vlan-bridge;
vlan-id 2732;
family bridge;
}
}
My underlaying infrastructure
* OS: Ubuntu 14.04.5 LTS
* kernel : 3.13.0-32-generic
* libvirtd (libvirt) 1.2.19
OVSBridge
Bridge “ovsbr0”
Port “ge-0.0.1-vmx2”
Interface “ge-0.0.1-vmx2”
Port “ge-0.0.1-vmx1”
Interface “ge-0.0.1-vmx1″
ovs_version: “2.0.2”
Sample Dump of my XML
vMX1
vMX2
May 21, 2018 @ 03:42:18
Found the solution with reference to the below article
https://blog.scottlowe.org/2012/10/04/some-insight-into-open-vswitch-configuration/
I have to set my ports to lacp.
ovs-vsctl set port ge-0.0.2-vmx1 lacp=active
ae2 up up
ae2.0 up up inet 10.10.10.1/30
multiservice
run show iccp
Redundancy Group Information for peer 10.10.10.2
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: None
Client Application: lacpd
Redundancy Group IDs Joined: None
May 21, 2018 @ 11:28:02
Nice to know , I used esxi and relied on esxi networking than ovs, the most important part for me was that not to mix internal coomunication with fpc and RE with other networks doing so immediately reset the connectivity and was never ending loop
Rakesh
May 28, 2018 @ 17:12:08
root@R2# commit
error: IRB interface(irb.200) and l2-interface(ae0.0) do not belong to the same routing instance
error: configuration check-out failed
May 28, 2018 @ 17:35:02
You caught it right, the commit complains that its not the part of same RTi
May 29, 2018 @ 10:36:05
Hi Pakesh M,
I’d like to get vqfx on gns3 or eve where can i get it.
Note !!! I have not juniper account.
May 29, 2018 @ 12:46:22
Hi San, https://github.com/Juniper/vqfx10k-vagrant, you can use official juniper qfx gitpage and use vagrant to get vqfx images
May 28, 2018 @ 17:13:16
I found irb do not belong to the same routing instance
Jul 14, 2018 @ 16:40:07
Dear Rakesh M
I found as below error my lab
Current State Machine’s State: mcae config exchange error
Jul 18, 2018 @ 04:51:33
Looks like reachability issue mate, is your iccp up ?
May 13, 2020 @ 22:56:03
Hi Rakesh, In my environment we have spine 1&2 and mc-lag is configured.
in one switch RSTP is configured for all the ports and in another switch no RSTP configured. Is it correct or not.
May 16, 2020 @ 03:02:23
Hi Sandeep, is there a particular reason you would leave out RSTP on the other switch ? RSTP configuration would only matter on how down stream devices communicate with each other, Spines should never have a problem with or without having its configuration.