No more excuses! Get into #SDWAN experience effortlessly

SDWAN is one of the most appealing use cases. Manages the connectivity between branches thru an overlay VPN Network. Automated policies that bring agility for users and happiness to business managers (i.e. Get revenue in advance avoiding shop opening delays. New branches can be connected on matter of days/hours.)

Even better if you can give it a try starting today in a very affordable way. I am confident we’ll have this SDWAN option soon into our Nuage Networks eXperience. Until then, you can use a single 32G memory server. How? well, simulating networks and instances thru the power of the virtualization.

Following notes brings details how to simulate a SDWAN solution into only one bare metal server. This server plays as the edge router between DC and branches. Networks are simulated thru linux bridges and instances with libvirt domains.

The intended demo topology is the following:

Prepare your environment

You will need to prepare your management and control plane. Use my previous post as reference to build this up: “#Nuage #docker demo in a box”. Take in consideration I am changing the domain name to sdwan.lab on this case. My next notes have been built based on this setup. I left a copy of all my config and xml files at

You’ll need QCOW2 images for all Nuage componentes. ping me for those anytime!

Create at least a couple of other bridges (i.e. brctl addbr int-bridge and use “ifconfig up” to initiate them). Check out my post Multiple dummies interfaces and bridges to simulate your #SDWAN for further reference.

Also, you’ll have to add a DHCP server for NSGs an their uplink connection. Simulating the same thing will happen when you connect a gateway to a Internet or WAN connection. That way, all NSGs will reach out the util (proxy) server to get the certificates and start with the activation process. We call this process automated bootstrapping. Check “Secure, Automated Zero-Touch Provisioning in Nuage Networks VNS” by Mostafa Mansour for further details. Use this post as reference to build your own dhcpd service: DHCP Server in a few steps (CentOS)

VSC – BGP Peering

You need at least two VSCs. Every VSC takes care of every network – one for WAN and other for Internet in our case. Every branch gateway is identified thru its unique uplink IP at every VSC. You can have a gateway using two uplinks IP over just one VSC.

This case I used just one IP for control access. The peering is done thru the control interface. Check out the vsc files to see the commands regarding that configuration. And don’t forget to create your VSC with two interfaces (Special Thanks to Mostafa Mansour for that heads up).

Installing your Util server

Util Server – Step ONE: Creating the instance

When you had finished installing VSD and both VSCs, then you can proceed with the the util server.
The util server in charge to send the notifications. Sends emails and SMS messages as part of the TWO factor authentication for NSGs’ bootstrapping process. Also, it brings certificates in behalf of VSD to every NSG (VSD is kept behind and protected from any external thread).
Get the image and move it to /var/lib/libvirt/images/util.sdwan.lab/util.qcow2
Define the domain thru the following xml file. Here you have the commands in case you forget them.

# you have to create your bridges in advance
# and copy your qcow2 images
# then execute:
virsh define util.sdwan.lab.xml
virsh start util.sdwan.lab

Util Server – Step TWO: Configure VM

Disable NetworkManager. We’ll use network.service to manage the interfaces.

service NetworkManager stop
systemctl disable NetworkManager

Add nameserver and domain in /etc/resolv.conf

search sdwan.lab

Configure interface eth0 as follow:


Restart network services: service network restart
Don’t forget to add your server settings also to /etc/hosts and set your FQDN.

Util Server – Step THREE: Install and configure Util App

Execute the following:

./rpms/ -x vsd.swan.lab -u util.sdwan.lab 

Edit your /etc/ntp.conf file and replace the time server with your local ntp one. Restart services. use “ntpstat” to see if it’s synchronized correctly.

Copy certificates from VSD to your util server. Run this command from VSD (ensure your ssh connection to util from VSD is working. It doesn’t work if it’s gonna be your first time and you have to accept to add util to your .ssh/known_hosts list):

/opt/vsd/ejbca/deploy/ -a generate -u proxy -c proxy -o csp \
-d util.sdwan.lab -f pem -t server -s root@

VSD post install tasks

VSD – Step ONE: Prepare VSD to accept encrypted connections

Change “Ejabberd Encryption mode” from claired text to ‘allow’. More details annexed.

/opt/vsd/bin/ejmode allow

“jboss” process will take a while.

Check out the configuration as follow:

[root@vsd ~]# /opt/vsd/ejbca/bin/ ra listendentities -S 00
SETTING: -S as 00
End Entity: proxy, "CN=proxy,O=csp", "dNSName=util.sdwan.lab,", null, 40, 1, 4, 0
End Entity: keyserver, "CN=keyserver,O=csp", "dNSName=vsd.sdwan.lab,", null, 40, 1, 3, 0
End Entity: ocspsigner, "CN=ocspsigner", "dNSName=vsd.sdwan.lab", null, 40, 1, 1, 0
End Entity: vsd.sdwan.lab, "CN=vsd.sdwan.lab,O=csp", "dNSName=vsd.sdwan.lab", null, 40, 1, 3, 0
End Entity: admin, "CN=admin", "null", null, 40, 1, 2, 0

Util Server: Starting Services

Switch over util server and do the following:

chkconfig haproxy on
chkconfig supervisord on
service haproxy restart
service supervisord restart

Check out services as follow:

[root@util ~]# ps -ef |grep supervisord
root     23153     1  0 21:30 ?        00:00:00 /usr/bin/python /usr//bin/supervisord --pidfile /var/run/
root     23162 22899  0 21:30 pts/0    00:00:00 grep --color=auto supervisord
[root@util ~]# ps -ef |grep proxy
root     23125     1  0 21:30 ?        00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/
haproxy  23126 23125  0 21:30 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/ -Ds
haproxy  23127 23126  0 21:30 ?        00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/ -Ds
root     23164 22899  0 21:30 pts/0    00:00:00 grep --color=auto proxy
[root@util ~]# ps -ef |grep node
root     23155 23153 38 21:30 ?        00:00:02 /usr/bin/node app
root     23166 22899  0 21:30 pts/0    00:00:00 grep --color=auto node

It’s important to create the user “proxy” under “csp” and add it to administrator group as follow:

VSC: Set up VSC certificates

Use the following command from VSD to copy certificates to both VSCs (use different users):

/opt/vsd/ejbca/deploy/ -a generate -u vsc-nsg -c vsc-nsg -d vsd.sdwan.lab  -f pem -t server -o csp -s admin@
/opt/vsd/ejbca/deploy/ -a generate -u vsc-nsg2 -c vsc-nsg2 -d vsd.sdwan.lab  -f pem -t server -o csp -s admin@

Set your TLS profile in both VSC as follow:

*A:vsc01# configure 
*A:vsc01>config# system 
*A:vsc01>config>system# security 
*A:vsc01>config>system>security# tls-profile "ex-tls-profile" create 
*A:vsc01>config>sys>sec>tls-profile>$ own-key "cf1:\vscnsg-Key.pem" 
*A:vsc01>config>sys>sec>tls-profile>$ own-certificate "cf1:\vscnsg.pem" 
*A:vsc01>config>sys>sec>tls-profile>$ ca-certificate "cf1:\vscnsg-CA.pem" 
*A:vsc01>config>sys>sec>tls-profile>$ no shutdown 
*A:vsc01>config>sys>sec>tls-profile>$ exit 
*A:vsc01>config>system>security# exit 
*A:vsc01>config>system# exit 
*A:vsc01>config# exit 
*A:vsc01# configure vswitch-controller open-flow tls-profile "ex-tls-profile" 
*A:vsc01# configure vswitch-controller xmpp tls-profile "ex-tls-profile" 
*A:vsc01# admin save 
Writing configuration to cf1:\config.cfg
Saving configuration ... OK

NSG – Step ONE: Create your overlay VPN network

Create the virsh domain for your dummies NSGs

My NSG’s XML file content is annexed. You can use it under your discretion.
Then, you need to copy the image and turn up that server. Don’t forget to connect port1 to the bridge “wan” as uplink to the WAN as follow:

    <interface type='bridge'>
      <mac address='52:54:00:ae:6c:72'/>
      <source bridge='wan'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

port2 in connected to ‘inet’ bridge and port 3 “MUST” me connected to ‘branch01’.

Look up the wan interface into the DHCP server lease.
Do the same for your “head” NSGs connecting it them to ‘wan’, ‘inet’ and ‘core’ respectively.

NSG – Step TWO: Create your profile and templates

The following picture shows how you have to set the “Infrastructure Gateway Profile” in VSD as follow:


I haven’t used “two factor authentication”. That way I will only get an email over the bootstapping process.

Create the “Infrastructure VSC profiles” in VSD also as follow:


And create the templates. Define your ‘network’ and ‘access’. All my ‘network’ connections have been defined as uplinks. All ones need at least a “VSC profile” assigned. ‘Access’ ports must be define starting from port3.


Define type of uplink. I will use ‘primary’ on both, wan and inet ports, to have a balanced uplink.


NSG – Step THREE: Bootstrap your NSGs

Now, create your branch user in the organization to enable your gateways. The user must to have a valid email address. Then, create a NSG and assign a user. Util server will send a notification to that user to activate the branch NSG.


Start up your NSG (i.e. mex01.sdwan.lab) and the dhcp will assign it an address to ‘wan’ and ‘inet’ ports. the NSG will serve as dhcp server thru port 3 with a temporary IP segment. connect client01 (check domain xml definitions)

You got an email, something like this:


The NSG will show this status after being activated:


Finally, check out this video to understand how create your Layer 3 domain between branches and DC:

See ya!

————- ANNEXED Details

Configure proxy server output

The following is the output that I got when I setup the util server:

[root@util ~]# ./rpms/ -x vsd.sdwan.lab -u util.sdwan.lab
IPv4 Enabled
Installing ...
Updating cron
Note: Forwarding request to 'systemctl enable httpd.service'.
ln -s '/usr/lib/systemd/system/httpd.service' '/etc/systemd/system/'
Redirecting to /bin/systemctl restart  httpd.service
Restarting supervisord (via systemctl):                    [  OK  ]
Note: Forwarding request to 'systemctl enable rsyslog.service'.
Redirecting to /bin/systemctl restart  rsyslog.service
Note: Forwarding request to 'systemctl enable ntpd.service'.
ln -s '/usr/lib/systemd/system/ntpd.service' '/etc/systemd/system/'
Redirecting to /bin/systemctl restart  ntpd.service

Changing VSD to ‘allow’ mode output

[root@vsd ~]# /opt/vsd/bin/ejmode allow
[Tue Nov 15 15:18:45 UTC 2016]: Set Ejabberd Encryption mode
* This command configures ejabberd to allow encryption mode.  
* It requires restart vsd. Please confirm to continue.     
Continue (yes/no)?yes
Please type yes again to continue
Continue (yes/no)?yes
Continue to switch on allow mode...
[Tue Nov 15 15:18:48 UTC 2016]: Enabling Ejabberd ...
* Successfully changed ejabbrerd to allow tls config         
* Please wait for the prompt when vsd is fully restarted.     
monit (pid  15219) is running...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     1    0     1    0     0     76      0 --:--:-- --:--:-- --:--:--   166
0waiting for ejabberd (10348) to go away...
waiting for ejabberd (10348) to go away...
Starting ejabberd...
Stopping jboss:                                            [  OK  ]
Starting jboss:  Standalone......
                                                           [  OK  ]
monit (pid  15219) is running...
[root@vsd ~]# monit -g check summary
The Monit daemon 5.17.1 uptime: 38m 

Program 'zookeeper-status'          Status ok
Program 'ntp-status'                Status ok
Program 'mysql-status'              Status ok
Program 'mediator-status'           Status ok
Program 'keyserver-status'          Status ok
Program 'jboss-status'              Status ok
Program 'ejbca-status'              Status ok
Program 'ejabberd-status'           Status ok

Util server Troubleshooting

I got some issues with my connection to reach out SMTP servers outside. How I knew it, just checked the /var/log/maillog as follow:

[root@util ~]# tail -f /var/log/maillog
Nov 10 13:57:46 util postfix/smtp[4538]: connect to[]:25: Connection timed out
Nov 10 13:57:46 util postfix/smtp[4538]: connect to[2607:f8b0:4003:c14::1b]:25: Network is unreachable
Nov 10 13:58:16 util postfix/smtp[4537]: connect to[]:25: Connection timed out
Nov 10 13:58:16 util postfix/smtp[4538]: connect to[]:25: Connection timed out
Nov 10 13:58:16 util postfix/smtp[4538]: connect to[2607:f8b0:4002:c03::1b]:25: Network is unreachable
Nov 10 13:58:16 util postfix/smtp[4538]: connect to[2607:f8b0:400d:c01::1b]:25: Network is unreachable
Nov 10 13:58:16 util postfix/smtp[4538]: 8A9D710E99AA: to=<>, relay=none, delay=492, delays=432/0.07/60/0, dsn=4.4.1, status=deferred (connect to[2607:f8b0:400d:c01::1b]:25: Network is unreachable)
Nov 10 13:58:46 util postfix/smtp[4537]: connect to[]:25: Connection timed out
Nov 10 13:58:46 util postfix/smtp[4537]: connect to[2607:f8b0:4002:c03::1b]:25: Network is unreachable
Nov 10 13:58:46 util postfix/smtp[4537]: 2E945108A788: to=<>, relay=none, delay=612, delays=521/0.06/90/0, dsn=4.4.1, status=deferred (connect to[2607:f8b0:4002:c03::1b]:25: Network is unreachable)

We are using postfix in our util server. Then just execute the following to get the mail queue and get the email and open it from your laptop.

[root@util ~]# postqueue -p
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
2E945108A788    29347 Thu Nov 10 13:48:35
  (connect to[]:25: Connection timed out)

8A9D710E99AA    29397 Thu Nov 10 13:50:04
(connect to[]:25: Connection timed out)

-- 58 Kbytes in 2 Requests.
[root@util ~]# postcat -q  8A9D710E99AA > /root/notif01.eml

Just copy the file notif01.eml to your laptop and open it with any email app client.

Categories: network, sdn

Tagged as: , , , , , , ,

8 replies »

  1. Hi Mauricio,
    I am very interested in deploying this in my lab, just took a course on this and been itching to get this going in my personal lab….. could you please share with me images and all the needed directions as to how to get this started.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: