Monthly Archives: December 2013

neutron中一种常见网络不通的现象,附修复脚本

neutron中经常发现vm ping不通网关,我发现很多情况是因为ovs-plugin没有将qvb连接到qbr桥上,所以解决办法也很简单,就是将其连上,即将qbr设为master。

具体的neutron、ovs、veth和iptables等就不展开讲了,具体可以看下图

ovs-neutron

当一个vm启动或被硬重启之后,tap接口(当使用gre模式时为tap)和qbr桥会被自动创建,而且tap的master为qbr。理论上qvb的master也应该是qbr,但我经常发现这个link的master没有被设置。那么设置脚本就很简单了:

#!/bin/bash

LINKS=`ip link|grep qvb |awk '{print $2}' |sed s'/.$//'|sed s'/^...//'`
for LINK in $LINKS
do
# a qbr link should appear after hard reboot an instance
ip link set "qbr$LINK" up
echo ip link set "qvb$LINK" master "qbr$LINK"
ip link set "qvb$LINK" master "qbr$LINK"
echo ip link set "tap$LINK" master "qbr$LINK"
ip link set "tap$LINK" master "qbr$LINK"
done

此外在SDN环境中还有几种vm连不上可能,比如

  1. br-tun每次在ovs-agent重启后controller就会消失,此时应该在重启脚本中添加:  ovs-vsctl set-controller br-tun tcp:30.0.0.1
  2. 确定br-int和br-tun没有多余的流,用ovs-ofctl dump-flows br-int/br-tun可看
  3. 确定网络控制器正常运行,有时候监控of控制流发现只有packet_in没有packet_out,此时重启一下网络控制器可能会解决问题

如果还是有问题,不妨对照上面的连接图,然后分别用tcpdump监听compute node的tap、qbr、qvb、qvo、br-int、br-tun、br-tun所对应的eth(如果用gre模式监听要加参数proto gre才能解gre包),再到network node的eth、br-tun、br-int、再到namespace的网关接口qr-xxx。

如果发现br-int有数据包,但br-tun没有数据包,不妨监听一下openflow的控制流:ovs-ofctl snoop br-int/br-tun,看看控制器的响应如

主动查询ovs的信息

openvswitch的flow信息获取途径现在通常是通过命令行CLI的方式(ovs-ofctl),或是部署agent,如neutron-ovs-agent,但通常这两种方法都需要在ovs所在的节点上部署,比较麻烦。

还有一种办法是网络controller通过openflow协议获取ovs的信息(如流信息),不过其负载就加重了。那么能否模拟网络controller的功能去获取流信息呢?当然可以,不过如果将ovs的控制器设置为主动模式连接我们的软件的话,那么这个软件就需要实现controller的功能,并且要与其他controller协作,更加麻烦。

其实openvswitch可以以服务端的形态存在,我们的软件可以连接ovs,从而获取其信息。要点在于

  1. 设置ovs为被动模式
  2. ovs-vsctl set-controller br-tun tcp:30.0.0.1 ptcp:6601
    这样ovs既可以主动连接网络controller(30.0.0.1:6633),又可以监听6601端口,接受客户端请求。此处br-tun与6601是一对,如果获取其他网桥的信息,则需要监听新的端口

  3. 设置ovs-ofctl为远程模式
  4. 在controller和node1两台机器上分别运行下面的命令:

    nsfocus@controller:/$ ovs-ofctl dump-flows tcp:127.0.0.1:6601
    NXST_FLOW reply (xid=0x4):
     cookie=0x20000000000000, duration=25911.957s, table=0, n_packets=25868, n_bytes=2535064, idle_timeout=5, idle_age=2, priority=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:16:3e:aa:30:94,dl_dst=fa:16:3e:be:25:7b,nw_src=100.0.0.19,nw_dst=192.168.1.1,icmp_type=8,icmp_code=0 actions=output:6
     cookie=0x20000000000000, duration=25911.953s, table=0, n_packets=25866, n_bytes=2534868, idle_timeout=5, idle_age=0, priority=0,icmp,in_port=6,vlan_tci=0x0000,dl_src=fa:16:3e:be:25:7b,dl_dst=fa:16:3e:aa:30:94,nw_src=192.168.1.1,nw_dst=100.0.0.19,icmp_type=0,icmp_code=0 actions=output:4
    
    nsfocus@node1:/$  ovs-ofctl dump-flows tcp:192.168.19.1:6601
    NXST_FLOW reply (xid=0x4):
     cookie=0x20000000000000, duration=25943.909s, table=0, n_packets=25900, n_bytes=2538200, idle_timeout=5, idle_age=2, priority=0,icmp,in_port=4,vlan_tci=0x0000,dl_src=fa:16:3e:aa:30:94,dl_dst=fa:16:3e:be:25:7b,nw_src=100.0.0.19,nw_dst=192.168.1.1,icmp_type=8,icmp_code=0 actions=output:6
     cookie=0x20000000000000, duration=25943.905s, table=0, n_packets=25898, n_bytes=2538004, idle_timeout=5, idle_age=0, priority=0,icmp,in_port=6,vlan_tci=0x0000,dl_src=fa:16:3e:be:25:7b,dl_dst=fa:16:3e:aa:30:94,nw_src=192.168.1.1,nw_dst=100.0.0.19,icmp_type=0,icmp_code=0 actions=output:4
    

    可见从不同主机都可以访问controller节点的br-tun桥,信息相同。

虽然ovs-ofctl不是自动化的方法,当时可以使用管道等技术实现程序化的办法。

在此感谢@Outlook的提示

用REST获得openvswitch ovsdb的信息

客户端可以通过ovsdb定义的协议访问openvswitch的数据库,协议在http://tools.ietf.org/html/draft-pfaff-ovsdb-proto-02,看来要成为ietf的标准了?怎么查询这些数据其实有一个样例,但是比较简单,我这里略作扩展,说明如何查询ovs的网桥、所连controller和流信息。

    1. 准备工作
    2. 因为ovs需要认证(公钥)才能访问其数据,我们为了简化直接在ovs所在节点上运行以下命令:

      ovs-appctl -t ovsdb-server ovsdb-server/add-remote ptcp:6632

      然后可以直接通过tcp的方式访问ovsdb了

    3. echo发送存活信息
    4. 客户端可以使用tcp方式与服务器保持长连接,所以可能定时需要发送echo信息与服务器确认存活。可编写以下脚本:


      import socket
      import json

      OVSDB_IP = '127.0.0.1'
      OVSDB_PORT = 6632
      BUFSIZE = 409600

      s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      s.connect((OVSDB_IP, OVSDB_PORT))

      query = {"method":"echo", "params":[], "id": 0}
      s.send(json.dumps(query))
      response = s.recv(BUFSIZE)
      print response

      执行结果为:
      {“id”:0,”result”:[],”error”:null}
      这是最简单的获取信息方式了,我们接下来要看看OVSDB中到底有些什么数据

    5. 获得所有数据库名
    6. 脚本中其他不变,最后三行换为以下内容,以后步骤也是类似:

      query = {"method":"list_dbs", "params":[], "id": 0}
      s.send(json.dumps(query))
      response = s.recv(BUFSIZE)
      print response

      执行后结果为
      {“id”:0,”result”:[“Open_vSwitch”],”error”:null}
      可见现在OVSDB中只有一个数据库Open_vSwitch。接下来我们查询这个数据库有哪些表

    7. 获得数据库的所有表结构
    8. 同样,代码为

      query = {"method":"get_schema", "params":["Open_vSwitch"], "id": 0}
      s.send(json.dumps(query))
      response = s.recv(BUFSIZE)
      print response

      因为表结构字段较多,所以我们将结果格式化,执行./ovs.py |python -m json.tool
      返回的结果将是所有数据库的内容

      {
          "error": null, 
          "id": 0, 
          "result": {
              "cksum": "2180939265 17455", 
              "name": "Open_vSwitch", 
              "tables": {
                  "Bridge": {
                      "columns": {
                          "controller": {
                              "type": {
                                  "key": {
                                      "refTable": "Controller", 
                                      "type": "uuid"
                                  },   
                                  "max": "unlimited", 
                                  "min": 0
                              }    
                          },   
                          "datapath_id": {
                              "ephemeral": true, 
                              "type": {
                                  "key": "string", 
                                  "min": 0
                              }    
                          },   
                          "datapath_type": {
                              "type": "string"
                          },   
                          "external_ids": {
                              "type": {
                                  "key": "string", 
                                  "max": "unlimited", 
                                  "min": 0, 
                                  "value": "string"
                              }    
                          },   
                          "fail_mode": {
      .......................................

      后面还有很多内容,在这里就不显示了。数据库除了Bridge,还有Controller、Flow Table、Interface、Manager、Mirror、Netflow、Open vswitch、Port、QoS、Queue、SSL、sFlow。看来OVS默认的流Flow表应该是openflow,其他还支持netflow和sflow。

    9. 获得所有网桥
    10. 我们查看一下网桥信息

      query = {"method":"transact", "params":["Open_vSwitch", {"op":"select", "table": "Bridge", "where":[]}], "id": 0}
      s.send(json.dumps(query))
      response = s.recv(BUFSIZE)
      print response

      执行./ovs.py |python -m json.tool
      返回的结果将是所有数据库的内容

      {
          "error": null, 
          "id": 0,  
          "result": [
              {   
                  "rows": [
                      {   
                          "_uuid": [
                              "uuid", 
                              "fd924881-6a15-4a1b-a803-aa49efa38179"
                          ],  
                          "_version": [
                              "uuid", 
                              "5306417b-381e-49dd-9ac2-0d95c450919c"
                          ],  
                          "controller": [
                              "set", 
                              []  
                          ],  
                          "datapath_id": "0000e0db551f99b4", 
                          "datapath_type": "", 
                          "external_ids": [
                              "map", 
                              []  
                          ],  
                          "fail_mode": [
                              "set", 
                              []  
                          ],  
                          "flood_vlans": [
                              "set", 
                              []  
                          ],  
                          "flow_tables": [
                              "map", 
                              []  
                          ],  
                          "mirrors": [
       "set",
                              []
                          ],
                          "name": "br-ex",
                          "netflow": [
                              "set",
                              []
                          ],
                          "other_config": [
                              "map",
                              []
                          ],
                          "ports": [
                              "set",
                              [
                                  [
                                      "uuid",
                                      "28b44f02-7ecd-4135-9836-f1059ac1ec10"
                                  ],
                                  [
                                      "uuid",
                                      "596e6d23-ec51-4b56-bddc-5aa0805fe16c"
                                  ],
                                  [
                                      "uuid",
                                      "82cd970c-ac39-46e6-8aa7-b03529cc8916"
                                  ],
                                  [
                                      "uuid",
                                      "be77f10f-1a52-4be2-8b9e-c97b26c9acee"
                                  ],
                                  [
                                      "uuid",
                                      "f8ae4f3c-9d77-4f7e-8707-ee2e36c70e08"
                                  ]
                              ]
                          ],
                          "protocols": [  "set",
                              []
                          ],
                          "sflow": [
                              "set",
                              []
                          ],
                          "status": [
                              "map",
                              []
                          ],
                          "stp_enable": false
                      },
      {
                          "_uuid": [
                              "uuid",
                              "e29eac0c-a9ae-4108-a277-21388a24a2f1"
                          ],
                          "_version": [
                              "uuid",
                              "65e1bd22-6178-4b01-827f-8d64b0840e72"
                          ],
                          "controller": [
                              "uuid",
                              "15846c69-f014-4a5a-bc8d-3cb67cc3cb03"
                          ],
                          "datapath_id": "00000eac9ee20841",
                          "datapath_type": "",
                          "external_ids": [
                              "map",
                              []
                          ],
                          "fail_mode": [
                              "set",
                              []
                          ],
                          "flood_vlans": [
                              "set",
                              []
                          ],
                          "flow_tables": [
                              "map",
                              []
                          ],
                          "mirrors": [
                              "set",
                              []
                          ],
                          "name": "br-tun",
                          "netflow": [
                              "set",[]
       ],
                          "other_config": [
                              "map",
                              []
                          ],
                          "ports": [
                              "set",
                              [
                                  [
                                      "uuid",
                                      "aff50719-bd85-4b72-9283-656844b663aa"
                                  ],
                                  [
                                      "uuid",
                                      "bcd6e8f9-713f-4177-b730-caed455df9b7"
                                  ],
                                  [
                                      "uuid",
                                      "d8e93efc-7ebd-4d89-b7d2-b9b4b60c993a"
                                  ]
                              ]
                          ],
                          "protocols": [
                              "set",
                              []
                          ],
                          "sflow": [
                              "set",
                              []
                          ],
                          "status": [
                              "map",
                              []
                          ],
                          "stp_enable": false
                      },
      .......................................

      篇幅关系,这里展现了两个网桥br-ex和br-tun,其中port字段可以查询网桥上连接的端口,如果controller字段存在,说明该网桥受控制器控制。那我们再看一下其所连控制器的信息。

    11. 获得所有网桥
    12. 我们查看一下网桥信息

      query = {"method":"transact", "params":["Open_vSwitch", {"op":"select", "table": "Controller", "where":[]}], "id": 0}
      s.send(json.dumps(query))
      response = s.recv(BUFSIZE)
      print response

      执行./ovs.py |python -m json.tool
      返回的结果

      {
          "error": null,
          "id": 0,
          "result": [
              {
                  "rows": [
                      {
                          "_uuid": [
                              "uuid",
                              "15846c69-f014-4a5a-bc8d-3cb67cc3cb03"
                          ],
                          "_version": [
                              "uuid",
                              "3a6bf4ed-035b-4e57-beb3-8bd08997826e"
                          ],
                          "connection_mode": [
                              "set",
                              []
                          ],
                          "controller_burst_limit": [
                              "set",
                              []
                          ],
                      ....
       "role": "master",
                          "status": [
                              "map",
                              [
                                  [
                                      "last_error",
                                      "Connection refused"
                                  ],
                                  [
                                      "sec_since_connect",
                                      "423018"
                                  ],
                                  [
                                      "sec_since_disconnect",
                                      "423089"
                                  ],
                                  [
                                      "state",
                                      "ACTIVE"
                                  ]
                              ]
                          ],
                          "target": "tcp:30.0.0.1"
                      },
                      {
                          "_uuid": [
                              "uuid",
                              "77fab68b-c693-4c9e-aea7-ba847809376d"
                          ],
                          "_version": [
                              "uuid",
                              "5e1f25ec-ae35-43fe-9cf4-835ef5980056"
                          ],
                          "connection_mode": [
                              "set",
                              []
                          ],
                       .....
      "role": "master",
                          "status": [
                              "map",
                              [
                                  [
                                      "last_error",
                                      "Connection refused"
                                  ],
                                  [
                                      "sec_since_connect",
                                      "423018"
                                  ],
                                  [
                                      "sec_since_disconnect",
                                      "423089"
                                  ],
                                  [
                                      "state",
                                      "ACTIVE"
                                  ]
                              ]
                          ],
                          "target": "tcp:30.0.0.1"
                      }
                  ]
              }
          ]
      }
      
      这里列出了两个controller项,但目标地址都是30.0.0.1,说明都是同一个controller,但是这两项的uuid不同,如15846c69-f014-4a5a-bc8d-3cb67cc3cb03是br-tun设置的controller,而77fab68b-c693-4c9e-aea7-ba847809376d是br-in设置的controller。
    13. 获得所有openflow信息
    14. 代码为

      query = {"method":"transact", "params":["Open_vSwitch", {"op":"select", "table": "Flow_Table", "where":[]}], "id": 0}
      s.send(json.dumps(query))
      response = s.recv(BUFSIZE)
      print response

      执行./ovs.py |python -m json.tool
      返回的结果result为空,而且静态动态流都为空,开始很疑惑,后来了解ovsdb主要是存放静态信息,所以流信息是没有保留的。获取流应该从vswitchd中获取,具体怎么做看下一篇吧:-)
      ovsdb和vswitchd的关系如该图所示,还有一些ovs的命令。
      ovs关系图

最后说明一下,用REST访问ovsdb可以使用程序化的方法获得ovs的信息,为下一步工作打下基础