Scale out #Kubernetes with #Nuage: Installation notes


Kubernetes is the dreamed orchestration tool for any developer. However, it could be a nightmare for security guys (get online images make them cringe). After all, K8s manages containers with apps and databases connected to an overlay network. Containers could have sensible information. They could be exploded and used to attack internal/external assets.

Nuage will help to microsegment containers and even put security instances in the middle to analyze traffic. A overall network management and their settings. Manage forwarding and control rules. Manage Public/WAN segment to publish your apps and to control bandwidth utilization.

The best thing! All those settings can be managed thru automated policies. You do not lose any agility.

Scalability is also a known issue. Customers managing 10000s of instances using a unique VSD instance as Global console. The usage of a lightweight resource as openflow is the evidence (Most “SDN” solutions use XMPP to program vswicthes or vrouters).

Containers scales in seconds. Does your security and network do so?

Next you will find a step-by-step to install your Kubernetes cluster with Nuage

Nuage Core solution

Nuage VSD and VSC have to be installed in advance. VSD is being serving at and VSC is installed at in my case.

Also, I have a dns/ntp server for my lab. Even the K8s nodes have their CNAMEs and reverses. I am using my dns as web server to publish the rpm files.

Prepare your installation

I’ve done a fresh install of my two servers (check my previous post)
Then, we need to publish Nuage rpm files into any http server. And the prepare our ansible/git server (I would be the same k8s master in this case)

Don’t forget both servers as to be accessed thru ssh with no password at all (you will have to add ansible node’s public keys into authorized_server files into both servers)

rpm -iUvh
yum -y update
yum -y install ansible
yum -y install git
yum -y install python-netaddr

Synchronize to your ntp server and change the localtime zone to your current
Normalize your hostname vars

[root@k8scluster ansible]# HOSTNAME=k8scluster.nuage.lab
[root@k8scluster ansible]# cat /etc/hosts   localhost localhost.localdomain localhost4 localhost4.localdomain4 k8scluster.nuage.lab k8scluster
[root@k8scluster ansible]# cat /etc/hostname
[root@k8scluster ansible]# hostname k8scluster.nuage.lab

Clone Nuage files:

git clone
cd contrib
git checkout origin/nuage -b nuage

You will get something like this:

[root@k8snode01 ~]# git clone
Cloning into 'contrib'...
remote: Counting objects: 23650, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 23650 (delta 0), reused 0 (delta 0), pack-reused 23647
Receiving objects: 100% (23650/23650), 27.79 MiB | 12.87 MiB/s, done.
Resolving deltas: 100% (10400/10400), done.
Checking out files:  81% (10289/12702)   
Checking out files: 100% (12702/12702), done.
[root@k8snode01 ~]# cd contrib
[root@k8snode01 contrib]# git checkout origin/nuage -b nuage
Branch nuage set up to track remote branch nuage from origin.
Switched to a new branch 'nuage'

Installing K8s with Nuage in a few steps

Create ansible.cfg inside ~/contrib/ansible

# this ~/contrib/ansible/ansible.cfg file

# Set the log_path
log_path = /var/log/ansible.log

pipelining = True

Create your inventory nodes file ~/contrib/ansible/nodes

# Inventory file ~/contrib/ansible/nodes
# Create an k8s group that contains the masters and nodes groups





# host group for masters


# host group for nodes, includes region info

Execute playbook

cd contrib/ansible
ansible-playbook -vvvv -i nodes cluster.yml

This is the final result I’ve got:

PLAY RECAP *********************************************************************
k8scluster.nuage.lab       : ok=303  changed=51   unreachable=0    failed=0   
k8snode01.nuage.lab        : ok=102  changed=36   unreachable=0    failed=0 

Check K8s cluster settings:

[root@k8scluster ~]# kubectl cluster-info
Kubernetes master is running at http://localhost:8080
Elasticsearch is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
Grafana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
InfluxDB is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

Start your Nuage Kubernetes Monitor service:

service nuagekubemon start

================== Begin update Sept 22, 2016 ================
If you want to use services thru kube-proxy (like service type=loadbalancer). We have to change /etc/kubernetes/proxy settings in all nodes as follow. The reason? Instead iptables, Nuage is managing security settings now.

KUBE_PROXY_ARGS="--kubeconfig=/etc/kubernetes/proxy.kubeconfig --proxy-mode=userspace"

================== End update Sept 22, 2016 ==================

Launching an App

Finally, Let’s launch soemthing:

[root@k8scluster ansible]# kubectl run my-nginx --image=nginx --replicas=2 --port=80
deployment "my-nginx" created
[root@k8scluster ansible]# kubectl get po
NAME                        READY     STATUS              RESTARTS   AGE
my-nginx-3800858182-r0om7   0/1       ContainerCreating   0          2m
my-nginx-3800858182-xtffj   0/1       ContainerCreating   0          2m
[root@k8scluster ansible]# kubectl get po
NAME                        READY     STATUS    RESTARTS   AGE
my-nginx-3800858182-r0om7   1/1       Running   0          6m
my-nginx-3800858182-xtffj   1/1       Running   0          6m

What we have in our VSD GUI: You will see containers from K8s cluster and nginx

kubernetes nuage 01

Now, you’ll see Security¬†policies in a more human interface.

kubernetes nuage 02

See ya!

2 replies »

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: