基于 Open vSwitch 的 OpenFlow 实践

Sep 172014

今天看到朋友推荐的一篇IBM的文章http://www.ibm.com/developerworks/cn/cloud/library/1401_zhaoyi_openswitch/index.html

文章内容很好，如果我只是收藏起来，那就实在是太浪费，还是动手练习一次，好好补一下我的网络。还是用UnitedStack的UOS，创建一个虚拟机来完成全部的试验。

文章就一个小笔误，把ns2写成ns3。外面很多转载的文章，都没注明出处，搞的以为是原创。如果错误都一样，就有点不好意思。

和IBM的文档不一样的地方是：我使用的是ubuntu 14.04，减少很多没必要的麻烦。

创建一个ubuntu 14.04的最小配置虚拟机，ssh到虚拟机上。

查看ubuntu版本

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:        14.04
Codename:       trusty

Contents

1 OVS
2 OpenFlow 命令
3 修改数据包
4 修改vlan tag
5 Floodlight

OVS

Ubuntu 14.04的OVS版本，已经是2.02，所以默认安装就可以。不过不同的发行版，ovs的名字会有点不同。

apt-cache search openvswitch

开始安装

apt-get install openvswitch-switch

查看OVS运行情况

# ps -ea | grep ovs
 3007 ?        00:00:00 ovsdb-server
 3017 ?        00:00:00 ovs-vswitchd

查看OVS版本

# ovs-appctl --version
ovs-appctl (Open vSwitch) 2.0.2
Compiled Aug 15 2014 14:31:01

查看 OVS 支持的 OpenFlow 协议的版本

# ovs-ofctl --version
ovs-ofctl (Open vSwitch) 2.0.2
Compiled Aug 15 2014 14:31:02
OpenFlow versions 0x1:0x4

OpenFlow 命令

创建一个OVS交换机

ovs-vsctl add-br ovs-switch

创建一个端口 p0，设置端口 p0 的 OpenFlow 端口编号为 100

ovs-vsctl add-port ovs-switch p0 -- set Interface p0 ofport_request=100

设置网络接口设备类型为”internal”,

ovs-vsctl set Interface p0 type=internal

查看设置结果

# ethtool -i p0
driver: openvswitch
version: 
firmware-version: 
bus-info: 
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

创建一个name space：ns0，把p0端口接入到ns0里，并且配置ip地址 192.168.1.100/24

ip netns add ns0
ip link set p0 netns ns0
ip netns exec ns0 ip addr add 192.168.1.100/24 dev p0
ip netns exec ns0 ifconfig p0 promisc up

查看创建结果

# ovs-vsctl show
6507c214-0c7a-4159-9813-977074f73aa1
    Bridge ovs-switch
        Port "p0"
            Interface "p0"
                type: internal
        Port ovs-switch
            Interface ovs-switch
                type: internal
    ovs_version: "2.0.2"

重复步骤，创建p1和p2端口

ovs-vsctl add-port ovs-switch p1 -- set Interface p1 ofport_request=101
ovs-vsctl set Interface p1 type=internal
ip netns add ns1
ip link set p1 netns ns1
ip netns exec ns1 ip addr add 192.168.1.101/24 dev p1
ip netns exec ns1 ifconfig p1 promisc up

查看创建结果

# ovs-vsctl show
6507c214-0c7a-4159-9813-977074f73aa1
    Bridge ovs-switch
        Port "p1"
            Interface "p1"
                type: internal
        Port "p0"
            Interface "p0"
                type: internal
        Port ovs-switch
            Interface ovs-switch
                type: internal
    ovs_version: "2.0.2"

创建p2

ovs-vsctl add-port ovs-switch p2 -- set Interface p2 ofport_request=102
ovs-vsctl set Interface p2 type=internal
ip netns add ns2
ip link set p2 netns ns2
ip netns exec ns2 ip addr add 192.168.1.102/24 dev p2
ip netns exec ns2 ifconfig p2 promisc up

查看

# ovs-vsctl show
6507c214-0c7a-4159-9813-977074f73aa1
    Bridge ovs-switch
        Port "p1"
            Interface "p1"
                type: internal
        Port "p2"
            Interface "p2"
                type: internal
        Port "p0"
            Interface "p0"
                type: internal
        Port ovs-switch
            Interface ovs-switch
                type: internal
    ovs_version: "2.0.2"

查看创建的交换机信息，获得dpid，端口openflow端口编号

# ovs-ofctl show ovs-switch
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000d23b94ce4146
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN 
SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_
DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE
 100(p0): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 101(p1): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 102(p2): addr:00:00:00:00:00:00
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(ovs-switch): addr:2a:be:0c:72:40:45
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

获取openflow端口编号

# ovs-vsctl get Interface p0 ofport
100
# ovs-vsctl get Interface p1 ofport
101
# ovs-vsctl get Interface p2 ofport
102

查看 datapath 的信息

# ovs-dpctl show
system@ovs-system:
        lookups: hit:34 missed:21 lost:0
        flows: 0
        port 0: ovs-system (internal)
        port 1: ovs-switch (internal)
        port 2: p0 (internal)
        port 3: p1 (internal)
        port 4: p2 (internal)

查看mac地址

ip netns exec ns0 ping 192.168.1.100
ip netns exec ns0 ping 192.168.1.101
ip netns exec ns0 ping 192.168.1.102

然后运行

# ovs-appctl fdb/show ovs-switch      
 port  VLAN  MAC                Age
  102     0  22:8e:52:36:92:25   17
  100     0  d6:0f:7e:ed:11:e4    4
  101     0  f2:0d:06:ff:79:d7    4

查看交换机所有table

ovs-ofctl dump-tables ovs-switch

查看交换机中的所有流表项

ovs−ofctl dump−flows ovs-switch

删除编号为 100 的端口上的所有流表项

ovs-ofctl del-flows ovs-switch "in_port=100"

查看交换机端口信息

ovs-ofctl show ovs-switch

修改数据包

屏蔽所有进入 OVS 的以太网广播数据包

$ ovs-ofctl add-flow ovs-switch "table=0, dl_src=01:00:00:00:00:00/01:00:00:00:00:00, actions=drop"

屏蔽 STP 协议的广播数据包

$ ovs-ofctl add-flow ovs-switch "table=0, dl_dst=01:80:c2:00:00:00/ff:ff:ff:ff:ff:f0, actions=drop"

修改数据包，添加新的 OpenFlow 条目，修改从端口 p0 收到的数据包的源地址为 9.181.137.1

ovs-ofctl add-flow ovs-switch "priority=1 idle_timeout=0,\
    in_port=100,actions=mod_nw_src:9.181.137.1,normal"

从端口 p0（192.168.1.100）发送测试数据到端口 p1（192.168.1.101），就是没啥响应

# ip netns exec ns0 ping 192.168.1.101
PING 192.168.1.101 (192.168.1.101) 56(84) bytes of data.

再打开一个ssh终端，登录进去，运行tcpdump，需要等待几分钟，才能看到响应

~# ip netns exec ns1 tcpdump -i p1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on p1, link-type EN10MB (Ethernet), capture size 65535 bytes
06:21:23.802308 IP 9.181.137.1 > 192.168.1.101: ICMP echo request, id 4533, seq 19, length 64
06:21:24.802358 IP 9.181.137.1 > 192.168.1.101: ICMP echo request, id 4533, seq 20, length 64

重定向数据包

添加新的 OpenFlow 条目，重定向所有的 ICMP 数据包到端口 p2

ovs-ofctl add-flow ovs-switch idle_timeout=0,dl_type=0x0800,nw_proto=1,actions=output:102

从端口 p0 （192.168.1.100）发送数据到端口 p1（192.168.1.101）

ip netns exec ns0 ping 192.168.1.101

这个时候你从p2里，可以看到

# ip netns exec ns2 tcpdump -i p2 icmp 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on p2, link-type EN10MB (Ethernet), capture size 65535 bytes
06:25:38.252471 IP 192.168.1.100 > 192.168.1.101: ICMP echo request, id 4668, seq 35, length 64
06:25:39.260438 IP 192.168.1.100 > 192.168.1.101: ICMP echo request, id 4668, seq 36, length 64
06:25:40.268419 IP 192.168.1.100 > 192.168.1.101: ICMP echo request, id 4668, seq 37, length 64

修改vlan tag

修改端口 p1 的 VLAN tag 为 101，使端口 p1 成为一个隶属于 VLAN 101 的端口

ovs-vsctl set Port p1 tag=101

现在由于端口 p0 和 p1 属于不同的 VLAN，它们之间无法进行数据交换。我们使用 ovs-appctl ofproto/trace 生成一个从端口 p0 发送到端口 p1 的数据包，这个数据包不包含任何 VLAN tag，并观察 OVS 的处理过程

ovs-appctl ofproto/trace ovs-switch in_port=100,dl_src=d6:0f:7e:ed:11:e4,\
dl_dst=f2:0d:06:ff:79:d7 -generate

注意：上面第一个mac地址，是p0的，第二个mac地址是p1的，你需要替换，上面有获取mac地址的方法。

# ovs-appctl ofproto/trace ovs-switch in_port=100,dl_src=d6:0f:7e:ed:11:e4,\
> dl_dst=f2:0d:06:ff:79:d7 -generate
Flow: metadata=0,in_port=100,vlan_tci=0x0000,dl_src=d6:0f:7e:ed:11:e4,dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000
Rule: table=0 cookie=0 priority=1,in_port=100
OpenFlow actions=mod_nw_src:9.181.137.1,NORMAL
no learned MAC for destination, flooding

Final flow: unchanged
Relevant fields: skb_priority=0,in_port=100,vlan_tci=0x0000/0x1fff,dl_src=d6:0f:7e:ed:11:e4,
dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000,nw_src=0.0.0.0,nw_proto=0,nw_frag=no
Datapath actions: 1,4

在第一行输出中，“Flow:”之后的字段描述了输入的流的信息。由于我们没有指定太多信息，所以多数字段（例如 dl_type 和 vlan_tci）被 OVS 设置为空值。

在第二行的输出中，“Rule:” 之后的字段描述了匹配成功的流表项。

在第三行的输出中，“OpenFlow actions”之后的字段描述了实际执行的操作。

最后一段以”Final flow”开始的字段是整个处理过程的总结，“Datapath actions: 4,1”代表数据包被发送到 datapath 的 4 和 1 号端口。

创建一条新的 Flow

对于从端口 p0 进入交换机的数据包，如果它不包含任何 VLAN tag，则自动为它添加 VLAN tag 101

ovs-ofctl add-flow ovs-switch "priority=3,in_port=100,dl_vlan=0xffff,\
actions=mod_vlan_vid:101,normal"

再次尝试从端口 p0 发送一个不包含任何 VLAN tag 的数据包，发现数据包进入端口 p0 之后, 会被加上 VLAN tag101, 同时转发到端口 p1 上

# ovs-appctl ofproto/trace ovs-switch in_port=100,dl_src=d6:0f:7e:ed:11:e4,\
> dl_dst=f2:0d:06:ff:79:d7 -generate
Flow: metadata=0,in_port=100,vlan_tci=0x0000,dl_src=d6:0f:7e:ed:11:e4,dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000
Rule: table=0 cookie=0 priority=1,in_port=100
OpenFlow actions=mod_nw_src:9.181.137.1,NORMAL
no learned MAC for destination, flooding

Final flow: unchanged
Relevant fields: skb_priority=0,in_port=100,vlan_tci=0x0000/0x1fff,dl_src=d6:0f:7e:ed:11:e4,dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000,nw_src=0.0.0.0,nw_proto=0,nw_frag=no
Datapath actions: 1,4
root@ovs:~# ovs-ofctl add-flow ovs-switch "priority=3,in_port=100,dl_vlan=0xffff,\
> actions=mod_vlan_vid:101,normal"
root@ovs:~# ovs-appctl ofproto/trace ovs-switch in_port=100,dl_src=d6:0f:7e:ed:11:e4,\
> dl_dst=f2:0d:06:ff:79:d7 -generate
Flow: metadata=0,in_port=100,vlan_tci=0x0000,dl_src=d6:0f:7e:ed:11:e4,dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000
Rule: table=0 cookie=0 priority=3,in_port=100,vlan_tci=0x0000
OpenFlow actions=mod_vlan_vid:101,NORMAL
no learned MAC for destination, flooding

Final flow: metadata=0,in_port=100,dl_vlan=101,dl_vlan_pcp=0,dl_src=d6:0f:7e:ed:11:e4,dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000
Relevant fields: skb_priority=0,in_port=100,vlan_tci=0x0000,dl_src=d6:0f:7e:ed:11:e4,dl_dst=f2:0d:06:ff:79:d7,dl_type=0x0000,nw_proto=0,nw_frag=no
Datapath actions: push_vlan(vid=101,pcp=0),1,pop_vlan,3,push_vlan(vid=101,pcp=0),4

反过来从端口 p1 发送数据包，由于 p1 现在是带有 VLAN tag 101 的 Access 类型的端口，所以数据包进入端口 p1 之后，会被 OVS 添加 VLAN tag 101 并发送到端口 p0

# ovs-appctl ofproto/trace ovs-switch in_port=101,dl_src=f2:0d:06:ff:79:d7,\
> dl_dst=d6:0f:7e:ed:11:e4 -generate
Flow: metadata=0,in_port=101,vlan_tci=0x0000,dl_src=f2:0d:06:ff:79:d7,dl_dst=d6:0f:7e:ed:11:e4,dl_type=0x0000
Rule: table=0 cookie=0 priority=0
OpenFlow actions=NORMAL
no learned MAC for destination, flooding

Final flow: unchanged
Relevant fields: skb_priority=0,in_port=101,vlan_tci=0x0000,dl_src=f2:0d:06:ff:79:d7,dl_dst=d6:0f:7e:ed:11:e4,dl_type=0x0000,nw_proto=0,nw_frag=no
Datapath actions: push_vlan(vid=101,pcp=0),1,2,4

Floodlight

新创建一个ubuntu 14.04的虚拟机。

apt-get update
apt-get install git
apt-get install ant
apt-get install openjdk-7-jdk

源码安装

git clone git://github.com/floodlight/floodlight.git
cd floodlight/
ant
java -jar target/floodlight.jar

这个时候floodlight就启动起来，最后一条命令，就是启动floodlight。

登录OVS节点

设置ovs的控制器为floodlight，10.250.3.10，就是floodlight虚拟机的IP。

ovs-vsctl set-controller ovs-switch tcp:10.250.3.10:6633

设置 OVS 的连接模式为 secure 模式

ovs-vsctl set Bridge ovs-switch fail-mode=secure

查看

# ovs-vsctl show
6507c214-0c7a-4159-9813-977074f73aa1
    Bridge ovs-switch
        Controller "tcp:10.250.3.10:6633"
            is_connected: true
        fail_mode: secure
        Port "p1"
            tag: 101
            Interface "p1"
                type: internal
        Port "p2"
            Interface "p2"
                type: internal
        Port "p0"
            Interface "p0"
                type: internal
        Port ovs-switch
            Interface ovs-switch
                type: internal
    ovs_version: "2.0.2"

通过访问 Floodlight 提供的 Web 管理界面 http://<Host Address>:8080/ui/index.html，我们可以查看 Floodlight 控制器的状态以及所有连接到 Floodlight 的交换机列表

通过 Floodlight 的 RESTAPI，添加两条新的规则让端口 p0 和 p1 可以相互通讯。注意：替换命令行中的 switch 的 ID 为交换机的 datapath ID

注意curl命令，尽量别用 / 换行

curl -d '{"switch": "00:00:d2:3b:94:ce:41:46", "name":"my-flow1", "cookie":"0","priority":"32768","ingress-port":"100","active":"true", "actions":"output=flood"}' http://10.250.3.10:8080/wm/staticflowentrypusher/json

curl -d '{"switch": "00:00:d2:3b:94:ce:41:46", "name":"my-flow2", "cookie":"0","priority":"32768","ingress-port":"101","active":"true", "actions":"output=flood"}' http://10.250.3.10:8080/wm/staticflowentrypusher/json

验证是否能从端口 p0 发送数据包到 p1

# ip netns exec ns0 ping -c4 192.168.1.101
PING 192.168.1.101 (192.168.1.101) 56(84) bytes of data.
64 bytes from 192.168.1.101: icmp_seq=1 ttl=64 time=0.625 ms
64 bytes from 192.168.1.101: icmp_seq=2 ttl=64 time=0.088 ms
64 bytes from 192.168.1.101: icmp_seq=3 ttl=64 time=0.082 ms
64 bytes from 192.168.1.101: icmp_seq=4 ttl=64 time=0.048 ms

在 OVS 端也可以看到，流表规则已经被 OVS 同步到本地。

# ovs-ofctl dump-flows ovs-switch
NXST_FLOW reply (xid=0x4):
 cookie=0xa00000626d6af5, duration=111.468s, table=0, n_packets=7, n_bytes=630, idle_age=2, in_port=100 actions=FLOOD
 cookie=0xa00000626d6af6, duration=83.717s, table=0, n_packets=7, n_bytes=630, idle_age=1, in_port=101 actions=FLOOD

通过 Floodlight 的 RestAPI，查看交换机上的流表规则

curl http://10.250.3.10:8080/wm/staticflowentrypusher/list/00:00:d2:3b:94:ce:41:46/json | python -mjson.tool

采用python的输出，好看很多的

# curl http://10.250.3.10:8080/wm/staticflowentrypusher/list/00:00:d2:3b:94:ce:41:46/json | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1435    0  1435    0     0   109k      0 --:--:-- --:--:-- --:--:--  116k
{
    "00:00:d2:3b:94:ce:41:46": {
        "my-flow1": {
            "actions": [
                {
                    "length": 8,
                    "lengthU": 8,
                    "maxLength": 32767,
                    "port": -5,
                    "type": "OUTPUT"
                }
            ],
            "bufferId": -1,
            "command": 0,
            "cookie": 45035997925042933,
            "flags": 0,
            "hardTimeout": 0,
            "idleTimeout": 0,
            "length": 80,
            "lengthU": 80,
            "match": {
                "dataLayerDestination": "00:00:00:00:00:00",
                "dataLayerSource": "00:00:00:00:00:00",
                "dataLayerType": "0x0000",
                "dataLayerVirtualLan": -1,
                "dataLayerVirtualLanPriorityCodePoint": 0,
                "inputPort": 100,
                "networkDestination": "0.0.0.0",
                "networkDestinationMaskLen": 0,
                "networkProtocol": 0,
                "networkSource": "0.0.0.0",
                "networkSourceMaskLen": 0,
                "networkTypeOfService": 0,
                "transportDestination": 0,
                "transportSource": 0,
                "wildcards": 4194302
            },
            "outPort": -1,
            "priority": -32768,
            "type": "FLOW_MOD",
            "version": 1,
            "xid": 0
        },
        "my-flow2": {
            "actions": [
                {
                    "length": 8,
                    "lengthU": 8,
                    "maxLength": 32767,
                    "port": -5,
                    "type": "OUTPUT"
                }
            ],
            "bufferId": -1,
            "command": 0,
            "cookie": 45035997925042934,
            "flags": 0,
            "hardTimeout": 0,
            "idleTimeout": 0,
            "length": 80,
            "lengthU": 80,
            "match": {
                "dataLayerDestination": "00:00:00:00:00:00",
                "dataLayerSource": "00:00:00:00:00:00",
                "dataLayerType": "0x0000",
                "dataLayerVirtualLan": -1,
                "dataLayerVirtualLanPriorityCodePoint": 0,
                "inputPort": 101,
                "networkDestination": "0.0.0.0",
                "networkDestinationMaskLen": 0,
                "networkProtocol": 0,
                "networkSource": "0.0.0.0",
                "networkSourceMaskLen": 0,
                "networkTypeOfService": 0,
                "transportDestination": 0,
                "transportSource": 0,
                "wildcards": 4194302
            },
            "outPort": -1,
            "priority": -32768,
            "type": "FLOW_MOD",
            "version": 1,
            "xid": 0
        }
    }
}

通过 Floodlight 的 RestAPI，删除交换机上的流表规则

curl http://10.250.3.10:8080/wm/staticflowentrypusher/clear/00:00:d2:3b:94:ce:41:46/json

6 Responses to “基于 Open vSwitch 的 OpenFlow 实践”

cyl5753 says:

2014/09/18 at 9:54 AM

陈老师都开始研究openflow了，期待…

Reply
liujun says:

2014/10/08 at 4:29 PM

陈老师你好，按照您和上面链接的方法照着做了一遍，在执行ovs-appctl ofproto/trace 时会提示
Bad flow syntax
ovs-appctl: ovs-vswitchd: server returned reply code 501
我的ovs版本是1.4.6，系统版本是ubuntu12.04 会有关系么？

Reply
- dou says:
  
  2015/07/07 at 4:14 PM
  
  出现这种问题,是因为ovs的版本过低,升级至新版本,就可以解决该问题.
  
  Reply
sherkyoung says:

2014/10/14 at 4:06 PM

陈老师，您的这篇文章中提到的ovs应该不是openstack中的ovs吧，可不可以把openstack中的ovs与floodlight等sdn的控制器进行对接呢？neutron组件中有个bigswitch插件，同时官网上也有介绍通过修改/etc/neutron/plugins/bigswitch/restproxy.ini文件，配置控制器的ip地址。
但是我这么做并没有成功，不知道陈老师有没有做过这方面的研究？

Reply
- 陈沙克 says:
  
  2014/10/14 at 10:55 PM
  
  ovs肯定是一样的，在opentack里的neutron，肯定也是支持，只是文章资料太少，根本不知道如何玩，而且问题很多而已。只能自己google。大家对openflow了解不多，需要逐步深入。
  
  Reply
tsjsdbd says:

2014/10/16 at 5:10 PM

Hi,

I have some problem when use Bridge mode for Linux containers. which I want to connect all containers hosted on different Host. make all those containers in one subnet.

First, I create a bridge on Host, add eth0 into this bridge, and grab the IP from eth0,

everything is OK, but the packet send out form this bridge is dropped by Openstack.

after some investigate, I find Openstack’s computer node have some iptables rule which check packet’s Mac address will drop the packet.

so I add some Top rule in computer node’s iptables:
-A FORWARD -s 10.10.10.0/24 -j RETURN
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD

it can works, but after maybe 1 minute, the rule was refreshed by Openstack:
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD
-A FORWARD -s 10.10.10.0/24 -j RETURN

so it fail again, anyone can help me? or if openstack have some configuration that allow bridge mode in VM

Thanks

Reply

陈沙克日志