SDWAN is one of the most appealing use cases. Manages the connectivity between branches thru an overlay VPN Network. Automated policies that bring agility for users and happiness to business managers (i.e. Get revenue in advance avoiding shop opening delays. New branches can be connected on matter of days/hours.)
Even better if you can give it a try starting today in a very affordable way. I am confident we’ll have this SDWAN option soon into our Nuage Networks eXperience. Until then, you can use a single 32G memory server. How? well, simulating networks and instances thru the power of the virtualization.
Following notes brings details how to simulate a SDWAN solution into only one bare metal server. This server plays as the edge router between DC and branches. Networks are simulated thru linux bridges and instances with libvirt domains.
Prepare your environment
You will need to prepare your management and control plane. Use my previous post as reference to build this up: “#Nuage #docker demo in a box”. Take in consideration I am changing the domain name to sdwan.lab on this case. My next notes have been built based on this setup. I left a copy of all my config and xml files at https://github.com/p1nrojas/sdwan-in-a-box
You’ll need QCOW2 images for all Nuage componentes. ping me for those anytime!
Create at least a couple of other bridges (i.e. brctl addbr int-bridge and use “ifconfig up” to initiate them). Check out my post Multiple dummies interfaces and bridges to simulate your #SDWAN for further reference.
Also, you’ll have to add a DHCP server for NSGs an their uplink connection. Simulating the same thing will happen when you connect a gateway to a Internet or WAN connection. That way, all NSGs will reach out the util (proxy) server to get the certificates and start with the activation process. We call this process automated bootstrapping. Check “Secure, Automated Zero-Touch Provisioning in Nuage Networks VNS” by Mostafa Mansour for further details. Use this post as reference to build your own dhcpd service: DHCP Server in a few steps (CentOS)
VSC – BGP Peering
You need at least two VSCs. Every VSC takes care of every network – one for WAN and other for Internet in our case. Every branch gateway is identified thru its unique uplink IP at every VSC. You can have a gateway using two uplinks IP over just one VSC.
This case I used just one IP for control access. The peering is done thru the control interface. Check out the vsc files to see the commands regarding that configuration. And don’t forget to create your VSC with two interfaces (Special Thanks to Mostafa Mansour for that heads up).
Installing your Util server
Util Server – Step ONE: Creating the instance
When you had finished installing VSD and both VSCs, then you can proceed with the the util server.
The util server in charge to send the notifications. Sends emails and SMS messages as part of the TWO factor authentication for NSGs’ bootstrapping process. Also, it brings certificates in behalf of VSD to every NSG (VSD is kept behind and protected from any external thread).
Get the image and move it to /var/lib/libvirt/images/util.sdwan.lab/util.qcow2
Define the domain thru the following xml file. Here you have the commands in case you forget them.
# you have to create your bridges in advance # and copy your qcow2 images # then execute: virsh define util.sdwan.lab.xml virsh start util.sdwan.lab
Util Server – Step TWO: Configure VM
Disable NetworkManager. We’ll use network.service to manage the interfaces.
service NetworkManager stop systemctl disable NetworkManager
Add nameserver and domain in /etc/resolv.conf
nameserver 10.10.10.2 search sdwan.lab
Configure interface eth0 as follow:
TYPE=Ethernet BOOTPROTO=static DEFROUTE=yes NAME=eth0 ONBOOT=yes DEVICE=eth0 NM_CONTROLLED=no IPADDR=10.10.10.161 PREFIX=24 GATEWAY=10.10.10.1
Restart network services: service network restart
Don’t forget to add your server settings also to /etc/hosts and set your FQDN.
Util Server – Step THREE: Install and configure Util App
Execute the following:
./rpms/install.sh -x vsd.swan.lab -u util.sdwan.lab
Edit your /etc/ntp.conf file and replace the time server with your local ntp one. Restart services. use “ntpstat” to see if it’s synchronized correctly.
Copy certificates from VSD to your util server. Run this command from VSD (ensure your ssh connection to util from VSD is working. It doesn’t work if it’s gonna be your first time and you have to accept to add util to your .ssh/known_hosts list):
/opt/vsd/ejbca/deploy/certMgmt.sh -a generate -u proxy -c proxy -o csp \ -d util.sdwan.lab -f pem -t server -s email@example.com:/opt/SDVPNHAProxy/config/keys/
VSD post install tasks
VSD – Step ONE: Prepare VSD to accept encrypted connections
Change “Ejabberd Encryption mode” from claired text to ‘allow’. More details annexed.
“jboss” process will take a while.
Check out the configuration as follow:
[root@vsd ~]# /opt/vsd/ejbca/bin/ejbca.sh ra listendentities -S 00 SETTING: -S as 00 End Entity: proxy, "CN=proxy,O=csp", "dNSName=util.sdwan.lab, firstname.lastname@example.org", null, 40, 1, 4, 0 End Entity: keyserver, "CN=keyserver,O=csp", "dNSName=vsd.sdwan.lab, email@example.com", null, 40, 1, 3, 0 End Entity: ocspsigner, "CN=ocspsigner", "dNSName=vsd.sdwan.lab", null, 40, 1, 1, 0 End Entity: vsd.sdwan.lab, "CN=vsd.sdwan.lab,O=csp", "dNSName=vsd.sdwan.lab", null, 40, 1, 3, 0 End Entity: admin, "CN=admin", "null", null, 40, 1, 2, 0 [
Util Server: Starting Services
Switch over util server and do the following:
chkconfig haproxy on chkconfig supervisord on service haproxy restart service supervisord restart
Check out services as follow:
[root@util ~]# ps -ef |grep supervisord root 23153 1 0 21:30 ? 00:00:00 /usr/bin/python /usr//bin/supervisord --pidfile /var/run/supervisord.pid root 23162 22899 0 21:30 pts/0 00:00:00 grep --color=auto supervisord [root@util ~]# ps -ef |grep proxy root 23125 1 0 21:30 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid haproxy 23126 23125 0 21:30 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds haproxy 23127 23126 0 21:30 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds root 23164 22899 0 21:30 pts/0 00:00:00 grep --color=auto proxy [root@util ~]# ps -ef |grep node root 23155 23153 38 21:30 ? 00:00:02 /usr/bin/node app root 23166 22899 0 21:30 pts/0 00:00:00 grep --color=auto node
VSC: Set up VSC certificates
Use the following command from VSD to copy certificates to both VSCs (use different users):
/opt/vsd/ejbca/deploy/certMgmt.sh -a generate -u vsc-nsg -c vsc-nsg -d vsd.sdwan.lab -f pem -t server -o csp -s firstname.lastname@example.org:/ /opt/vsd/ejbca/deploy/certMgmt.sh -a generate -u vsc-nsg2 -c vsc-nsg2 -d vsd.sdwan.lab -f pem -t server -o csp -s email@example.com:/
Set your TLS profile in both VSC as follow:
*A:vsc01# configure *A:vsc01>config# system *A:vsc01>config>system# security *A:vsc01>config>system>security# tls-profile "ex-tls-profile" create *A:vsc01>config>sys>sec>tls-profile>$ own-key "cf1:\vscnsg-Key.pem" *A:vsc01>config>sys>sec>tls-profile>$ own-certificate "cf1:\vscnsg.pem" *A:vsc01>config>sys>sec>tls-profile>$ ca-certificate "cf1:\vscnsg-CA.pem" *A:vsc01>config>sys>sec>tls-profile>$ no shutdown *A:vsc01>config>sys>sec>tls-profile>$ exit *A:vsc01>config>system>security# exit *A:vsc01>config>system# exit *A:vsc01>config# exit *A:vsc01# configure vswitch-controller open-flow tls-profile "ex-tls-profile" *A:vsc01# configure vswitch-controller xmpp tls-profile "ex-tls-profile" *A:vsc01# admin save Writing configuration to cf1:\config.cfg Saving configuration ... OK Completed.
NSG – Step ONE: Create your overlay VPN network
Create the virsh domain for your dummies NSGs
My NSG’s XML file content is annexed. You can use it under your discretion.
Then, you need to copy the image and turn up that server. Don’t forget to connect port1 to the bridge “wan” as uplink to the WAN as follow:
<interface type='bridge'> <mac address='52:54:00:ae:6c:72'/> <source bridge='wan'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
port2 in connected to ‘inet’ bridge and port 3 “MUST” me connected to ‘branch01’.
Look up the wan interface into the DHCP server lease.
Do the same for your “head” NSGs connecting it them to ‘wan’, ‘inet’ and ‘core’ respectively.
NSG – Step TWO: Create your profile and templates
The following picture shows how you have to set the “Infrastructure Gateway Profile” in VSD as follow:
I haven’t used “two factor authentication”. That way I will only get an email over the bootstapping process.
Create the “Infrastructure VSC profiles” in VSD also as follow:
And create the templates. Define your ‘network’ and ‘access’. All my ‘network’ connections have been defined as uplinks. All ones need at least a “VSC profile” assigned. ‘Access’ ports must be define starting from port3.
Define type of uplink. I will use ‘primary’ on both, wan and inet ports, to have a balanced uplink.
NSG – Step THREE: Bootstrap your NSGs
Now, create your branch user in the organization to enable your gateways. The user must to have a valid email address. Then, create a NSG and assign a user. Util server will send a notification to that user to activate the branch NSG.
Start up your NSG (i.e. mex01.sdwan.lab) and the dhcp will assign it an address to ‘wan’ and ‘inet’ ports. the NSG will serve as dhcp server thru port 3 with a temporary IP segment. connect client01 (check domain xml definitions)
You got an email, something like this:
The NSG will show this status after being activated:
Finally, check out this video to understand how create your Layer 3 domain between branches and DC:
————- ANNEXED Details
Configure proxy server output
The following is the output that I got when I setup the util server:
[root@util ~]# ./rpms/install.sh -x vsd.sdwan.lab -u util.sdwan.lab IPv4 Enabled Installing ... Updating cron Done Note: Forwarding request to 'systemctl enable httpd.service'. ln -s '/usr/lib/systemd/system/httpd.service' '/etc/systemd/system/multi-user.target.wants/httpd.service' Redirecting to /bin/systemctl restart httpd.service Restarting supervisord (via systemctl): [ OK ] Note: Forwarding request to 'systemctl enable rsyslog.service'. Redirecting to /bin/systemctl restart rsyslog.service Note: Forwarding request to 'systemctl enable ntpd.service'. ln -s '/usr/lib/systemd/system/ntpd.service' '/etc/systemd/system/multi-user.target.wants/ntpd.service' Redirecting to /bin/systemctl restart ntpd.service
Changing VSD to ‘allow’ mode output
[root@vsd ~]# /opt/vsd/bin/ejmode allow [Tue Nov 15 15:18:45 UTC 2016]: Set Ejabberd Encryption mode *********************************************************** * This command configures ejabberd to allow encryption mode. * It requires restart vsd. Please confirm to continue. *********************************************************** Continue (yes/no)?yes Please type yes again to continue Continue (yes/no)?yes Continue to switch on allow mode... [Tue Nov 15 15:18:48 UTC 2016]: Enabling Ejabberd ... ************************************************************** * Successfully changed ejabbrerd to allow tls config * Please wait for the prompt when vsd is fully restarted. ************************************************************** monit (pid 15219) is running... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 1 0 1 0 0 76 0 --:--:-- --:--:-- --:--:-- 166 0waiting for ejabberd (10348) to go away... waiting for ejabberd (10348) to go away... Starting ejabberd... done. Stopping jboss: [ OK ] Starting jboss: Standalone...... [ OK ] monit (pid 15219) is running... [root@vsd ~]# monit -g check summary The Monit daemon 5.17.1 uptime: 38m Program 'zookeeper-status' Status ok Program 'ntp-status' Status ok Program 'mysql-status' Status ok Program 'mediator-status' Status ok Program 'keyserver-status' Status ok Program 'jboss-status' Status ok Program 'ejbca-status' Status ok Program 'ejabberd-status' Status ok
Util server Troubleshooting
I got some issues with my connection to reach out SMTP servers outside. How I knew it, just checked the /var/log/maillog as follow:
[root@util ~]# tail -f /var/log/maillog Nov 10 13:57:46 util postfix/smtp: connect to gmail-smtp-in.l.google.com[188.8.131.52]:25: Connection timed out Nov 10 13:57:46 util postfix/smtp: connect to gmail-smtp-in.l.google.com[2607:f8b0:4003:c14::1b]:25: Network is unreachable Nov 10 13:58:16 util postfix/smtp: connect to alt1.aspmx.l.google.com[184.108.40.206]:25: Connection timed out Nov 10 13:58:16 util postfix/smtp: connect to alt1.gmail-smtp-in.l.google.com[220.127.116.11]:25: Connection timed out Nov 10 13:58:16 util postfix/smtp: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4002:c03::1b]:25: Network is unreachable Nov 10 13:58:16 util postfix/smtp: connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:400d:c01::1b]:25: Network is unreachable Nov 10 13:58:16 util postfix/smtp: 8A9D710E99AA: to=<firstname.lastname@example.org>, relay=none, delay=492, delays=432/0.07/60/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:400d:c01::1b]:25: Network is unreachable) Nov 10 13:58:46 util postfix/smtp: connect to alt2.aspmx.l.google.com[18.104.22.168]:25: Connection timed out Nov 10 13:58:46 util postfix/smtp: connect to alt1.aspmx.l.google.com[2607:f8b0:4002:c03::1b]:25: Network is unreachable Nov 10 13:58:46 util postfix/smtp: 2E945108A788: to=<email@example.com>, relay=none, delay=612, delays=521/0.06/90/0, dsn=4.4.1, status=deferred (connect to alt1.aspmx.l.google.com[2607:f8b0:4002:c03::1b]:25: Network is unreachable)
We are using postfix in our util server. Then just execute the following to get the mail queue and get the email and open it from your laptop.
[root@util ~]# postqueue -p -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient------- 2E945108A788 29347 Thu Nov 10 13:48:35 firstname.lastname@example.org (connect to alt2.aspmx.l.google.com[22.214.171.124]:25: Connection timed out) email@example.com 8A9D710E99AA 29397 Thu Nov 10 13:50:04 firstname.lastname@example.org (connect to alt2.gmail-smtp-in.l.google.com[126.96.36.199]:25: Connection timed out) email@example.com -- 58 Kbytes in 2 Requests. [root@util ~]# postcat -q 8A9D710E99AA > /root/notif01.eml
Just copy the file notif01.eml to your laptop and open it with any email app client.