Open vSwitch in Openstack

A multitude of Openstack users eventually run into Open vSwitch. Not because they want to, but because they have to. So here is something to deconstruct what Open vSwitch really is, and what it does.

What is Open vSwitch?

In the words of the Open vSwitch website:

“Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license.  It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag).”

Openstack uses Open vSwitch as its default virtual switch. There are various reasons for this. The primary reason is VLAN support. Open vSwitch is inherently built with VLAN support, and Openstack uses VLANs extensively to differentiate between its virtual networks. Also, Open vSwitch is works very well in a distributed environment. This is extremely important in a platform like Openstack which works with a range of distributed compute nodes.

Open vSwitch Architecture

Open vSwitch architecture

This is Open vSwitch on an Openstack compute node at the simplest level. The entire setup is present within a single compute node, which can be connected to other compute nodes via the physical port ethX.

If VM01 were to talk to VM02, the packet is first sent to its Linux bridge qBrX. qBrX then forwards the packet to br-int. br-int is the internal Open vSwitch bridge. As far as VM01 is concerned, it behaves exactly as a real switch would. It looks up the port to which VM02 is connected to in its forwarding table and forwards the packet on that port. The packet then hits qBrY, which is VM02’s Linux bridge, after which it reaches VM02.

Now you might wonder, why do we need those qBr* Linux bridges?

Well, an Openstack compute is something that can be shared by multiple tenants. Each of these tenants most probably have different security requirements for their VMs. This is something that would be impossible to enforce at the br-int level, where multiple tenants have their VMs connected. The next logical thing to do would be to have another checkpoint between the VM and  br-int where this can be done. That is exactly what the Linux bridge is. It enforces the security groups applied to the VM by Openstack while keeping br-int reusable for other tenants.

So, how does a VM send data to a VM on another compute node?

From what we saw above, br-int is what does the switching between VMs. When VM01, say, sends a packet to VM05, which is a VM on another compute node, the packet first reaches br-int. br-int sees that VM05 is not something that it is directly connected to, therefore send it out through the virtual port int-br-ethX. This port is connected to the other vSwitch br-ethX via the port phy-br-ethX.

br-ethX is connected to the network driver ethX. Here, ethX is the data link of the compute node. Once the packet reaches ethX, it is sent out onto the wire to the compute node.

Once the packet reaches the destination compute node, it traces back the flow we saw above. It goes up from ethX, to br-ethX, then to br-int, which sends it out to the respective Linux bridge and finally reaches the destination VM.


So that was what I had. Please feel free to reach out to me if there is something that you don’t understand or if there is something that you think I should talk about. The Openstack network administration guide has some typical use-cases and their appropriate configurations for Open vSwitch.


OpenStack developer: Beginner’s guide (Part 2)

In my previous post, I went over a few basics of OpenStack and Neutron in particular. I shall now pick up from where I left off, and give you a little background about how extensions work with Neutron.


Extensions are a great way to extend the features of neutron without having to meddle with the existing plugin or the core neutron component.

You might wonder why we need extensions. Why can’t the plugin incorporate these features directly?

Imagine a scenario where you are a large cloud network provider with thousands of users. Not surprisingly, each user has different needs. It would be close to impossible have a plugin that satisfies the requirements of all these users without compromising on its core functionality. A simpler (and much more efficient) solution would be to have a core plugin that does very specific tasks, and push all the other tasks to external modules. Extensions do exactly that.

You can have an extension that does pretty much anything. Imagine you need to create networks that require credentials before a host is able to connect to them. Doing this with regular networks is almost impossible and with software defined networks, a nightmare.

But extensions for neutron on OpenStack make this really easy. Create a class that is a subclass of the ExtensionDescriptor class, define a table to store the data for this extension and refer the extension in your plugin. And that’s it. You’re good to go.

Let’s take a look at how Cisco makes the credential extension.

If you look at the file, the first thing you will notice is the RESOURCE_ATTRIBUTE_MAP.

    'credentials': {
        'credential_id': {'allow_post': False, 'allow_put': False,
                          'validate': {'type:regex': attributes.UUID_PATTERN},
                          'is_visible': True},
        'credential_name': {'allow_post': True, 'allow_put': True,
                            'is_visible': True, 'default': ''},
        'tenant_id': {'allow_post': True, 'allow_put': False,
                      'is_visible': False, 'default': ''},
        'type': {'allow_post': True, 'allow_put': True,
                 'is_visible': True, 'default': ''},
        'user_name': {'allow_post': True, 'allow_put': True,
                      'is_visible': True, 'default': ''},
        'password': {'allow_post': True, 'allow_put': True,
                     'is_visible': True, 'default': ''},

This defines what attributes the extension expects from the client. The attributes also act as columns for the credential table in the database and the properties defined (allow_post, is_visible, …) act like restrictions on the columns.

Moving down, we see the actual Credential class defined. It inherits from the ExtensionDescriptor class that specifies certain abstract methods that the extension class must override. This ensures all extensions conform to a specific format.

The method that we are interested in here is the get_resources() method. This puts together all the data that the client sends over as a dictionary and sends it over to the plugin to provision the required resources.

class Credential(extensions.ExtensionDescriptor):

    def get_name(cls):
        """Returns Extended Resource Name."""
        return "Cisco Credential"

    def get_alias(cls):
        """Returns Extended Resource Alias."""
        return "credential"

    def get_description(cls):
        """Returns Extended Resource Description."""
        return "Credential include username and password"

    def get_namespace(cls):
        """Returns Extended Resource Namespace."""
        return ""

    def get_updated(cls):
        """Returns Extended Resource Update Time."""
        return "2011-07-25T13:25:27-06:00"

    def get_resources(cls):
        """Returns Extended Resources."""
        resource_name = "credential"
        collection_name = resource_name + "s"
        plugin = manager.NeutronManager.get_plugin()
        params = RESOURCE_ATTRIBUTE_MAP.get(collection_name, dict())
        controller = base.create_resource(collection_name,
                                          plugin, params)
        return [extensions.ResourceExtension(collection_name,

And that’s it! Your extension is ready.

Now, lets head over to the plugin to see how it uses an extension.

Firstly, you need to specify a list of extension aliases that the plugin supports. This is a list of commands that comes over from the client that the plugin identifies as one of its own. Any command for an extension that isn’t on this list will be ignored by the plugin.

supported_extension_aliases = ["credential", "Cisco qos"]

Here, credential is the extension alias that the plugin will be looking for.

Further down in the class, we see the methods that the plugin uses to invoke the operations on the credential extension.

    def get_all_credentials(self):
        """Get all credentials."""
        LOG.debug(_("get_all_credentials() called"))
        credential_list = cdb.get_all_credentials()
        return credential_list

    def get_credential_details(self, credential_id):
        """Get a particular credential."""
        LOG.debug(_("get_credential_details() called"))
        return cdb.get_credential(credential_id)

    def rename_credential(self, credential_id, new_name, new_password):
        """Rename the particular credential resource."""
        LOG.debug(_("rename_credential() called"))
        return cdb.update_credential(credential_id, new_name,

You see a lot of cdb calls here to get, update and so on. These methods are defined in the network_db_v2 file.

def get_all_credentials():
    """Lists all the creds for a tenant."""
    session = db.get_session()
    return (session.query(network_models_v2.Credential).all())

def get_credential(credential_id):
    """Lists the creds for given a cred_id."""
    session = db.get_session()
        return (session.query(network_models_v2.Credential).
    except exc.NoResultFound:
        raise c_exc.CredentialNotFound(credential_id=credential_id)

def get_credential_name(credential_name):
    """Lists the creds for given a cred_name."""
    session = db.get_session()
        return (session.query(network_models_v2.Credential).
    except exc.NoResultFound:
        raise c_exc.CredentialNameNotFound(credential_name=credential_name)

def add_credential(credential_name, user_name, password, type):
    """Create a credential."""
    session = db.get_session()
        cred = (session.query(network_models_v2.Credential).
        raise c_exc.CredentialAlreadyExists(credential_name=credential_name)
    except exc.NoResultFound:
        cred = network_models_v2.Credential(
        return cred

def remove_credential(credential_id):
    """Removes a credential."""
    session = db.get_session()
        cred = (session.query(network_models_v2.Credential).
        return cred
    except exc.NoResultFound:

def update_credential(credential_id,
                      new_user_name=None, new_password=None):
    """Updates a credential for a tenant."""
    session = db.get_session()
        cred = (session.query(network_models_v2.Credential).
        if new_user_name:
            cred["user_name"] = new_user_name
        if new_password:
            cred["password"] = new_password
        return cred
    except exc.NoResultFound:
        raise c_exc.CredentialNotFound(credential_id=credential_id)

This is where all the DB action happens with each of these methods working to create, fetch, delete and update entries to and from the Credential table in the database.

And that was neutron extensions in a nutshell. I’ve tried to be as generic as possible here and this is in no way exhaustive. These are the basic ideas that you would need to follow too to create your own extensions.

Let me know if you don’t get something I spoke about here, or if you think I’ve written something that is incorrect.

In my next post, I’ll talk about how Horizon is structured and what you need to do create your own custom commands on the CLI or have your own fields on the dashboard.

And as always, the OpenStack Q&A Site is your best friend.

OpenStack developer: Beginner’s guide (Part 1)

Let me begin with what this is not.

What I intend to do here is something much more primitive. When I began Openstack development, I found these awesome links that describe Openstack in theory, and how to install it, but there was hardly anything about where to find what code, and how to begin development. What I intend to do here is precisely that. Enable people like me to locate what points of code is hit when operations on OpenStack are run.

OpenStack enables cloud admins to interact with the cloud in 2 ways:

  • The dashboard (also called Horizon)
  • The command line interface (CLI)

The dashboard gives you a great GUI for various operations you might need to perform. For the adventurous lot, the CLI gives you the same dashboard like functionality from your beloved terminal.

OpenStack is made up of seven basic components:

  • Nova (Compute)
  • Keystone (Authentication)
  • Neutron (Networking)
  • Cinder (Block Storage)
  • Glance (Image Storage)
  • Swift (Object Storage)
  • Horizon (GUI)

In addition to this, it also has a host of other components:

  • Heat (Orchestration)
  • Tempest (Test Suite)
  • Ceilometer (Telemetry)

Since OpenStack is far too vast to explain each component in detail, I’ll pick Horizon and Neutron as examples. The rest should have a structure similar to these two.


Neutron is OpenStack’s networking component.


Neutron follows a plugin model where anyone can attach a plugin to work with the general neutron architecture, but offers something unique. Typically, organizations and vendors release their own plugins to customize neutron to work with their hardware. A vanilla OpenStack installation comes with the OpenVSwitch plugin enabled.

So lets get to the point. The code.

Imagine you are a network admin and would like to create a virtual network. You either follow the (very intuitive) steps on the dashboard, or you run the following command on the CLI.

# neutron net-create net1

What does this do internally?

The client (Dashboard/CLI) sends a HTTP request to the neutron-server. The web service catches this request and forwards it to the plugin.The plugin then maps it to the appropriate method. I’m using the OpenVSwitch plugin here as an example, but it would work the same way with any plugin.

def create_network(self, context, network):
        (network_type, physical_network,
         segmentation_id) = self._process_provider_create(context,

        session = context.session
        #set up default security groups
        tenant_id = self._get_tenant_id_for_create(
            context, network['network'])
        self._ensure_default_security_group(context, tenant_id)

        with session.begin(subtransactions=True):
            if not network_type:
                # tenant network
                network_type = self.tenant_network_type
                if network_type == constants.TYPE_NONE:
                    raise q_exc.TenantNetworksDisabled()
                elif network_type == constants.TYPE_VLAN:
                     segmentation_id) = ovs_db_v2.reserve_vlan(session)
                elif network_type in constants.TUNNEL_NETWORK_TYPES:
                    segmentation_id = ovs_db_v2.reserve_tunnel(session)
                # no reservation needed for TYPE_LOCAL
                # provider network
                if network_type in [constants.TYPE_VLAN, constants.TYPE_FLAT]:
                    ovs_db_v2.reserve_specific_vlan(session, physical_network,
                elif network_type in constants.TUNNEL_NETWORK_TYPES:
                    ovs_db_v2.reserve_specific_tunnel(session, segmentation_id)
                # no reservation needed for TYPE_LOCAL
            net = super(OVSNeutronPluginV2, self).create_network(context,
            ovs_db_v2.add_network_binding(session, net['id'], network_type,
                                          physical_network, segmentation_id)

            self._process_l3_create(context, net, network['network'])
            self._extend_network_dict_provider(context, net)
            # note - exception will rollback entire transaction
        LOG.debug(_("Created network: %s"), net['id'])
        return net

create_network() is what does the final network creation. I won’t go into the details here, but if you were to build your own plugin, this is what your network creation code should look like, apart from all the extra functionality that you might want to provide.

All the other operations (subnet-create, router-create….) follow a similar model.

In my next post, I’ll tell you a bit about how extensions work on neutron.

This is meant to just give you pointers about where to start digging when you want to develop with OpenStack, and in no way exhaustive.

If you find yourself lost, feel free to give me a shout, or you could head over to OpenStack’s Q&A Site. It is a great community and there are always people around to help.