innercoder.com

A blog for Linux programming enthusiasts

Installing and Configuring OpenvSwitch With QEMU-KVM

| Comments

Intro

The maturity reached by Software Defined Networking (SDN) in the lastest years is a very important one. We are at a point where we could see this architecture /framework become a standard. And what makes it so amazing is its fundamental idea of openness, just like Linux did it around 20 years ago. OpenvSwitch (OVS) is a tool used to create virtual bridges which can then connect virtual or physical hosts. It implements both traditional L2 switching protocols or Openflow.

The goal of this post is to use our QEMU-KVM hypervisor to bring up a virtual machine configured as a OpenvSwitch (OVS) server which will then virtualize other machines which will be chained together with it. In other words, it could be also considered as a OVS virtual lab, but these concepts could easily be made for production environments. The procedure show in this post contains no automation scripts, the reason is that to really understand what’s going on, the user must do each steps and check the output. Once the procedure starts to make sense, you will begin to think of ways to automate all this.

What is OpenvSwitch?

OVS is a switching application used for both virtualized environment and actual switch hardware. Many current solutions use it for both cases.

Procedure

What Linux distro to use?

Qemu and OVS are available for almost all distributions out there. For the sake of prototyping this solution quickly, I chose Ubuntu. We are going to use the OVS controller provided in the package manager but compile from source the actual switching application, so as to use the latest version. I tested this on Ubuntu 12.04 and 14.04.

Check for the right hardware

This solution will work at its best if the host OS supports virtualization extensions and nested virtualization.

To check if your hardware supports virtualization, there are several options:

1
2
3
4
5
6
7
8
9
10
11
//A value > 0 means that hardware virtualization is supported, 0 means it doesn't.
egrep -c '(vmx|svm)' /proc/cpuinfo 

kvm-ok
// If supported
INFO: /dev/kvm exists 
KVM acceleration can be used 

// If not supported
INFO: Your CPU does not support KVM extensions
KVM acceleration can NOT be used

To check for support of nested virtualization, look for the value in /sys/module/kvm_intel/parameters/nested.

1
2
cat /sys/module/kvm_intel/parameters/nested
Y

AMD processors come with this value enabled by default. Certain Intel processors might not have it enabled by default, like mine. A Intel Core i7 5600U. To enable it, append the following parameter to kernel boot command “GRUB_CMDLINE_LINUX” in /etc/default/grub

1
2
3
4
5
// For Intel processirs
kvm-intel.nested=1

// For AMD processors
kvm-amd.nested=1

If nested virtualization is supported then we will be able to have virtual machines that perform virtualization (A Matrix within a Matrix) with the use of hardware extensions. But in the case you are using old hardware, don’t fret. The images we plan to use for nested virtualization are very small, so CPU emulation will also work.

Run the following for the changes to take effect.

1
grub2-mkconfig -o /boot/grub2/grub.cfg

Let’s take care of the main image

Create the image

1
2
qemu-img create -f raw virt2.img 10G
Formatting 'virt2.img', fmt=raw size=10737418240 

Install the Ubuntu iso

1
kvm -m 2048 -smp 2 -cpu host -cdrom ./ubuntu-12.04.5-desktop-amd64.iso -vga vmware -hda virt2.img

NOTE: The parameters above are self-explanatory except for the “-cpu”. If you run the virtual OS without this parameter, it is going to use the QEMU cpu which will not handle hardware virtualization in the guest OS.

Shutdown and restart

1
2
// We no longer need the "-cdrom" parameter
kvm -m 2048 -smp 2 -cpu host -vga vmware -hda virt2.img

Perform basic configuration

1
2
3
4
5
apt-get install -y kvm qemu-kvm git python-simplejson python-twisted-conh \
  automake autoconf gcc uml-utilities \
  libtool build-essential pkg-config \
  libssl-dev libvirt-bin bridge-utils \
  linux-headers-$(uname -r)

Now, we have the option to install the latest version of openvswitch from github or the latest stable version from a tar ball. We will go for the stable one. We are going to compile the openvswitch module with our current kernel headers.

NOTE: this is pretty much the part that brings almost all the errors. If you get any errors while compiling or loading the module. Check the file INSTALL.md at the top of the openvswitch directory. It is a long file but very thorough and clear. Check also the log files with “tail /var/log/kern.log” or “dmesg | tail”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// Download
wget http://openvswitch.org/releases/openvswitch-2.4.0.tar.gz

// Extract
tar xf openvswitch-2.4.0.tar.gz 

// Configure and compile the openvswitch module
cd openvswitch-2.4.0
./configure --with-linux=/lib/modules/$(uname -r)/build
make && make install

// Install the module
modprobe -i openvswitch

// Check if it was installed
lsmod |grep openvswitch
openvswitch      71972  0

To make sure that it survives a reboot. Append it to /etc/modules

1
2
3
4
5
6
7
8
// /etc/modules: kernel modules to load at boot time.
//
// This file contains the names of kernel modules that should be loaded
// at boot time, one per line. Lines beginning with "#" are ignored.

lp
rtc
openvswitch

Now to initialize and configure the ovs configuration database

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// Initialize the configuration database
mkdir -p /usr/local/etc/openvswitch
ovsdb-tool create /usr/local/etc/openvswitch/conf.db ./vswitchd/vswitch.ovsschema

// Configure and start configuration database
ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
                 --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
                 --private-key=db:Open_vSwitch,SSL,private_key \
                 --certificate=db:Open_vSwitch,SSL,certificate \
                 --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
                 --pidfile --detach

// Initialize the database 
ovs-vsctl --no-wait init

Start the Open vSwitch daemon

1
2
3
4
5
6
7
8
9
10
// Start the OVS daemon
ovs-vswitchd --pidfile --detach

2016-02-27T02:08:12Z|00001|ovs_numa|INFO|Discovered 2 CPU cores on NUMA node 0
2016-02-27T02:08:12Z|00002|ovs_numa|INFO|Discovered 1 NUMA nodes and 2 CPU cores
2016-02-27T02:08:12Z|00003|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
connecting...
2016-02-27T02:08:12Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock:
connected

Lab Setup

Once OVS is finally running, it is time to start the fun part of this tutorial. Which is to create the bridge, fire up the guest images, and add their ports to the virtual bridge.

Add a virtual bridge. The bridge is what is what we could call a virtual switch where our virtual ports will connect to to create a broadcast domain. We will call it bridge1.

1
2
3
4
5
6
7
8
ovs-vsctl add-br bridge1

//Outputi from "ovs-vsctl show"
0e92445c-3b4e-41e2-a719-1d1f294d1cbc
    Bridge "bridge1"
        Port "bridge1"
            Interface "bridge1"
                type: internal

Download the small guest images.

1
2
wget http://wiki.qemu.org/download/linux-0.2.img.bz2
bzip2 -d linux-0.2.img.bz2 

Now we are going to start the small virtual images. But first copy them into the amount of virtual host you are planning on using. In this case we are going to use two.

1
2
cp linux-0.2.img host1.img
cp linux-0.2.img host2.img

Now start the images with kvm. Following the QEMU manual, all virtual machines should always start with 52:54 on their MAC addresses.

1
2
sudo kvm -m 128 -name host1 -net nic,macaddr=52:54:00:00:01:00 -net tap,ifname=r1-eth0,script=no,downscript=no -hda ./host1.img &
sudo kvm -m 128 -name host2 -net nic,macaddr=52:54:00:00:02:00 -net tap,ifname=r2-eth0,script=no,downscript=no -hda ./host2.img &

Once the images are up, we are going to see to interfaces. Eth0 and Lo. Let’s configure Eth0.

1
2
3
4
5
// guest VM host 1
ifconfig eth0 192.168.10.1 netmask 255.255.255.0 broadcast 192.168.10.255

// guest VM host 2
ifconfig eth0 192.168.10.2 netmask 255.255.255.0 broadcast 192.168.10.255

They are not able to ping each other yet. Now if you look at the output of “ip link” on the controller machine, you will see the two interface names we gave to our guest VMs. They should look similar to this. Note that the state of the bridge and these two newly created interfaces are down.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
ip link

// output
...

bridge1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group
default 
    link/ether ae:5f:00:c9:ce:4c brd ff:ff:ff:ff:ff:ff
r1-eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group
default qlen 500
    link/ether 9a:68:e7:f5:f7:0d brd ff:ff:ff:ff:ff:ff
r2-eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group
default qlen 500
    link/ether ee:64:e6:ba:51:36 brd ff:ff:ff:ff:ff:ff

...

With the guest VMs interfaces in place, we can now add them to the bridge.

1
2
ovs-vsctl add-port bridge1 r1-eth0
ovs-vsctl add-port bridge1 r2-eth0

Check the output of “ovs-vsctl show”, you should now be able to see the bridge and the two interfaces from the guest VMs.

1
2
3
4
5
6
7
8
9
10
11
ovs-vsctl show

0e92445c-3b4e-41e2-a719-1d1f294d1cbc 
  Bridge "bridge1"
      Port "r1-eth0"
          Interface "r1-eth0"
      Port "r2-eth0"
          Interface "r2-eth0"
      Port "bridge1"
          Interface "bridge1"
              type: internal

We are ready now, simply enable all the guest VM interfaces and the bridge. On the controller machine.

1
2
3
ip link set bridge1 up
ip link set r1-eth0 up
ip link set r2-eth0 up

That’s it. The host should be able to ping each other now. images

Conclusions

Again, we had no automation in place. We could start by creating scripts for the guest images to auto configure their ip addresses. And another to start the OVS database and daemon. There are several possibilites, I will venture into them and come back for another post. One idea will be to find a way to benchmark the performance of OVS compared to Linux bridges.

Hope this post was helpful so please let me know your thoughts and/or feedback in the comments.

Thanks for visiting.

Comments