CNF/VNF Under-the-hood: Quality of Service (QoS) in SDN

We are moving out! An updated post can be found here:


Relevant technologies like DPDK or SmartNICs are starting to come up lately to bring better performance options to CNF/VNF instances. Sharing a Linux kernel among other applications, is not good enough for some network functions. However, there are still other options before considering replacing your hardware.

Quality of Service (QoS) could be the cheapest and quick way to solve some scale issues. Or unless it will give you a better understanding of what sort of traffic you’re dealing with, or help you to avoid any fight over resources with traffic managed from IP Stack for KVM, management or Storage. 

QoS sounds like the right path to follow for a network professional, but unnatural for IT guys or DevOps. If you combine those skills, that would bring an easy and fast way to deploy QoS policies at big scale, and the power to adapt them much faster to changing conditions (i.e. market, covid, weather…)

CNF/VNF vs Network Infrastructure

OpenStack is being popular in Telco to orchestrate network functions. And along with that, different sorts of requirements have come up from the network. A VNF can be a bunch of VMs with different demands. Network traffic to/from those VMs can be managed through an OVS at the kernel, fighting for resources with other applications between kernel interruptions.
VNF traffic could be diverse, and some packets should be treated and forwarded faster. But if you don’t do anything, small critical packets (i.e. network control) will be treated as any sort of traffic, and they could be waiting for their turn after some big non-critical file transfer that maybe doesn’t have anyone urgently waiting on the other side.
There are hardware options like DPDK, off-load NICs or SR-IOV. They are more expensive and require specific pieces of software to manage them. SRIOV works passing-through OVS in the kernel and that would bring a higher challenger for orchestration to keep the so demanded agility. This won’t avoid you the fuss of designing appropriate QoS policies. Resources are limited, no matter what magic you do.
5G and CNF won’t make things easier. The density of end-points per server has grown exponentially. And if VMs could still be managed thru VLANs, command-line and QoS policies over a traditional fabric. CNF will force you to automate and orchestrate those policies through APIs and group policies based on metadata. Containers are state-less and change their IP address on every reboot. And sorry, you can’t skip QoS, best effort or FIFO approach from kernel modules is not good enough anymore.

Let’s check some QoS concepts in the following section

Common QoS Mechanisms

Quality service is more a collection of tool that just one thing. And the following are the categories of those tools.

  • Classification and Marking (i.e. DSCP): Before entering the network, some packets will be classified and marked to make their identification easier just reading the header for any later switch/router or virtual router involved on the forwarding process. That will make things more efficient in time and resources. There’s a point where you don’t know if the packet will stay in the DC or will be sent outside through MPLS to another DC. Apps will be able to flow anywhere.
  • Queueing: every router will manage different “buckets” for different packets that have been identified using the marks we told before, in the buffer for queuing and wait to be forwarded. Of course, packets classified with higher priority will be forwarded accordingly to their priority. SDN usually relies on queues which are defined in the NIC and mapped back to the tap interface of VM where the packets are marked before to be forwarded anywhere.
  • Congestion avoidance: If that buffer gets full at the network card, then packets will start to get dropped. Tools like RED will avoid that sort of issue by dropping random packets.
  • Policing (drop) and shaping (hold): Traffic conditioning, set a speed limit, maximum of bandwidth. Policing drops the traffic and shaping hold the traffic until packets can  be sent staying inside the limits of speed.

Classification and Marking (DSCP)

Class of Service (CoS)

There are bits you can use to define a CoS in every frame header. Then you can have 8 values (0 to 7). Vendors like Cisco leave the value 6 and 7 for other purposes. Then, It’s better to use values between 0-5. Cisco mark some of the packets for their VoIP products by default with the value 5.

Class of Service - CoS

Class of Service – CoS

Because it’s a Ethernet datagram, then you will run into issues with CoS when they go through a router, cause they will re-write the headers and then you might lose what was define.

Type of service (ToS) Byte

A byte in the IP packet header will carry the Type of Service (IPv6 call it Type of Class). You may use the three left bits in this section to identify your type of service. Those bits are used for the IP precedence marking. You may not use the values 6 and 7 like we said before, because those are still used to identify the CoS like we showed for datagrams.

If we add the next three bits of the byte, we expand the amount of values for a classification from 8 to 64. A more realistic number for nowadays, don’t you think? Those 6 left bits in the byte is what we call DSCP (Differentiated Services Code Point).

IPv4 Type of Service or IPv6 Type of Class Byte

IPv4 Type of Service or IPv6 Type of Class Byte

We have values between 0-63 (64 possible values like we said). To make things easier IETF set a standard of which values you could use and stay in sync with the rest of the world. They set 21 values. IETF gave them names and they call that Per hub behaviors (PHB). The two most important values to remember are:

  • Default: 000000
  • Expedited Forwarding: 101110

Next table will show the Assured Forwarding (AF) values. It’s important to mention that the dropping priority is independent of the class. Then, in case the buffer gets full and some packets need to be dropped, AF22 will be dropped before AF11. it doesn’t matter, it has higher priority on forwarding. Be careful selecting your values.

Table I from Pervasive Differentiated Services: A QoS Model for ...

DSCP Assured Forwarding Table

Random Early Detection (RED)

Now that we are talking about packet dropping. Then it’s important to talk about RED. RED is the industry standard for dropping packets. Eventually your router will do it and the question is how that will be managed. RED defines a Maximum threshold that when it is hit, it’s a 100% probability that queue will drop that traffic.

RED packet dropping

RED packet dropping

A better way to see it’s using the next picture. As an example we have a queue with 45 packets as average queue depth. before to reach 25 packets, there’s no dropping. After 25 the line is smoothly introducing the probability of discard packets until reach the 45, where the probability goes from 20% to 100%.

You can set those thresholds and MPD (mark probability denominator). MPD will define the probability at where the random dropping will work before to reach the Maximum threshold. And for your information, none ever try to touch those values. Unless you have a very specific use case.

RED Drop probability

Now, Cisco varies that probability depending on the Class of Service, defining RED profiles for each AF value.

WRED drop profile depending on Class of service

WRED drop profile depending on Class of service

Explicit congestion Notification (ECN)

If you remember we talked about ToS and the bits regarding DSCP. Ok, and what about the last 2 bits i the ToS Byte? Those bits, basically, are use to notify the origin router that the queue is full or congested and ask it to slow down. Then, if the router send 11, means it’s experiencing congestion and other route should be considered.

This is more informative. I didn’t want to let that gap i this overview. We won’t talk of this in CNF/VNF use cases.

Explicit Congestion Notification (ECN) RFC 3168 Justin Yackoski ...

Traffic Policing vs. Traffic Shaping

Well, a decent SDN solution should also offer Traffic Policing and Traffic Shaping. The idea is to understand the difference between both.

Some SDN software applies it only for the traffic which goes from VM to Network, around the VM’s tap interface. Others must be enforced directly in an external switch/router hardware piece thru the identification which VM is sending that packet thru some way. but. again, depends on the technology/vendor.

Traffic Policing

Traffic policing is more regarding a limitation in Bytes over a certain period of time. We have two important values:

  • Committed Information Rate (CIR) which is the average speed that the Network guarantees the VM’s port or container’s port
  • Committed Burst (Bc): which defines the amount of packets (usually in bytes) that can be sent as a group without causing any violations or exceeding the CIR.

Let me illustrate this with the following example:

Let’s say the average speed or CIR will be 200Mbps and we allow a Bc of 20MB. Then, the VM sends an initial bunch of packets for a total of 9.6MB. We have 10.4MB or 10.4M tokens left in the bucket. If we sent another burst 0.1s (100 msecs ) later. What would be the limit we have to avoid any packet delay or drop? The answer is 12.9MB maximum. Why? because we have 10.4 left plus the 2.5MB added after 0.1s based on the following formula (0.1s*200Mbps/8bits):

traffic policing

Traffic Shaping

Now, let’s do the same with traffic shaping:

  • Now we have other value call Traffic Interval (Tc)

We have 200Mbps on a 1Gbps virtual interface. and we want to set a Tc of 125ms. means we can send 25Mbps every 125ms. In this case, the enforcement is you can fill a bucket up to 25Mbits every 125ms, the rest of the packets could be dropped or delayed. However, those 25Mbits of data will be sent at “line speed” and the bucket could be emptied long before to reach those 125ms limit. Then, you can have a behavior like is shown in the next picture..

traffic shaping

QoS in a CNF/VNF Telco Cloud Setup

Instances like VNF and CNF commonly run on KVM and Docker Engine respectively. Either way, KVM or Docker will use Linux IP Stack to send traffic and let the kernel decide in what network resources are used to ensure priority in their common tasks. For example,  KVM uses specific queues in the NIC for some traffic like storage or management. The configuration commands on how to configure the egress queue scheduling as well as kernel mappings, etc. are highly dependent on the distribution of Linux used on the hypervisor as well as the specific physical NICs and the driver version.

Screen Shot 2020-07-13 at 5.33.44 PM.png

Nuage QoS Classification, NIC queues, Forwarding Class and Kernel

Nuage uses a management plane to manage multiple control planes with XMPP. VSD can be accessed through a GUI or APIs to define QoS policies at vport, subnet, zone or domains (i.e. VRF).

Nuage forwarding class and traffic shaping in VSD

Nuage forwarding class and traffic shaping in VSD


Nuage, for the VMs that use OVS with a VirtIO driver, uses 8 classes and binds every class to a different queue in the server NIC. You can mark traffic going into the network. However, traffic arriving to the VM is not affected, basically it is not worthy, the packet is already on the server.

Also, in the previous image you can see you can define Traffic Shaping policies also at different levels in the domain.

Also, depending on the SDN software, you will find different ways to deal with DSCP marking or traffic shaping. For example, you can define a specific policy based on TCP or UDP traffic.

DSCP nuage QoS policy TCP UDP marking remarking

DSCP marking in Nuage through a QoS policy based in specific TCP/UDP traffic or former DSCP mark (only for Traffic from VMs to Network)


Hope that explanation worked for you. You can check the references for more details though.

See ya!


We use info from Traffic Policing vs. Traffic Shaping

This post took notes from this tutorial from Kevin Wallace:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: