Monthly Archives: July 2013

ubuntu 12.04上安装puppet

发现官方的说明问题很多,不过全做一个参考吧

puppet的agent端安装没有问题,但是master涉及到postgresql数据库的配置,简单运行puppet-enterprise-installer会出错

我在安装目录的answers目录下发现很多配置文件,其中full_suite_existing_postgres.sample最符合master的需求,于是将其复制到安装的根目录,修改其中的一些配置项,主要是将其中数据库的host设置为master的域名,对应的数据库设置用户名密码。

然后配置postgreSQL数据库,可以参考这篇文章,设置好postgres密码后,进入
nsfocus@controller:~$ psql -U postgres -W -h 192.168.19.1
Password for user postgres:
psql (9.1.9)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.

postgres=# CREATE USER "pe-puppetdb" PASSWORD 'onemorestrongpass';
CREATE ROLE
postgres=# CREATE DATABASE "pe-puppetdb" OWNER "pe-puppetdb" ENCODING 'utf8' LC_CTYPE 'en_US.utf8' LC_COLLATE 'en_US.utf8' template template0;
CREATE DATABASE
postgres=# CREATE DATABASE "console_auth" OWNER "auth_user" ENCODING 'utf8' LC_CTYPE 'en_US.utf8' LC_COLLATE 'en_US.utf8' template template0;
CREATE DATABASE

然后运行./puppet-enterprise-installer -a full_suite_existing_postgres.sample

如果还有错误,可以查看当前目录下的install_log.lastrun.[hostname]文件,看看最后FATAL的信息

General Design Issues of the Internet

刚刚看到RFC1958上面关于Internet设计的一些原则,觉得对现在的系统设计都很有启发,摘录如下:

3.1 Heterogeneity is inevitable and must be supported by design.
Multiple types of hardware must be allowed for, e.g. transmission
speeds differing by at least 7 orders of magnitude, various computer
word lengths, and hosts ranging from memory-starved microprocessors
up to massively parallel supercomputers. Multiple types of
application protocol must be allowed for, ranging from the simplest
such as remote login up to the most complex such as distributed
databases.

3.2 If there are several ways of doing the same thing, choose one.
If a previous design, in the Internet context or elsewhere, has
successfully solved the same problem, choose the same solution unless
there is a good technical reason not to. Duplication of the same
protocol functionality should be avoided as far as possible, without
of course using this argument to reject improvements.

3.3 All designs must scale readily to very many nodes per site and to
many millions of sites.

3.4 Performance and cost must be considered as well as functionality.

3.5 Keep it simple. When in doubt during design, choose the simplest
solution.

3.6 Modularity is good. If you can keep things separate, do so.

3.7 In many cases it is better to adopt an almost complete solution
now, rather than to wait until a perfect solution can be found.

3.8 Avoid options and parameters whenever possible. Any options and
parameters should be configured or negotiated dynamically rather than
manually.

3.9 Be strict when sending and tolerant when receiving.
Implementations must follow specifications precisely when sending to
the network, and tolerate faulty input from the network. When in
doubt, discard faulty input silently, without returning an error
message unless this is required by the specification.

3.10 Be parsimonious with unsolicited packets, especially multicasts
and broadcasts.

3.11 Circular dependencies must be avoided.

For example, routing must not depend on look-ups in the Domain
Name System (DNS), since the updating of DNS servers depends on
successful routing.

3.12 Objects should be self decribing (include type and size), within
reasonable limits. Only type codes and other magic numbers assigned
by the Internet Assigned Numbers Authority (IANA) may be used.

3.13 All specifications should use the same terminology and notation,
and the same bit- and byte-order convention.

3.14 And perhaps most important: Nothing gets standardised until
there are multiple instances of running code.

nova实例僵死的解决办法

openstack grizzly的nova模块还是有时候不稳定,异常时可能会出现一些实例僵死在soft reboot和hard reboot中,就算过几天还是这个状态,在horizon界面中唯一的办法是terminate这些实例,但是如果有数据怎么办?

这个问题还不能直接从命令行搞定:
# nova reboot –hard 366a2e92-9483-4074-984d-7ec0de130c14
ERROR: Cannot ‘reboot’ while instance is in task_state rebooting_hard (HTTP 409) (Request-ID: req-1bf39445-5bb5-49d5-8dd6-dcf12aeceae1)

解决办法不是没有,下面是我搜索到的:

1 代码入手,分析问题,然后提交patch到mainstream中,例如

但是我发现我的nova版本是最新的,没有解决我的问题

2 手动重置,如

# nova reset-state –active 366a2e92-9483-4074-984d-7ec0de130c14
# nova reboot 366a2e92-9483-4074-984d-7ec0de130c14

如此即可重启实例

在此吐槽一下nova不够稳定、horizon的功能太少,同时期待更多人和组织贡献代码进来

[openstack]NAT gateway和port不一致导致VM不能到外网

当VM设置完floatingip后,VM还是不能连接外网,排查原因,发现是quantum中设置的问题:

quantum中设置外网为192.168.19.129/25,不设网关,allocation_pools为{“start”: “192.168.19.130”, “end”: “192.168.19.254”}。

root@controller:/usr/src/nova# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.19.129 0.0.0.0 UG 0 0 0 qg-29c30020-2e
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-cd728374-d8
10.0.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-f915c799-96
192.168.19.128 0.0.0.0 255.255.255.128 U 0 0 0 qg-29c30020-2e

路由器的网卡却是:

root@controller:/usr/src/nova# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 ifconfig
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1390 (1.3 KB)  TX bytes:1390 (1.3 KB)

qg-29c30020-2e Link encap:Ethernet  HWaddr fa:16:3e:10:18:21  
          inet addr:192.168.19.130  Bcast:192.168.19.255  Mask:255.255.255.128
          inet6 addr: fe80::f816:3eff:fe10:1821/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:87 errors:0 dropped:0 overruns:0 frame:0
          TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:12593 (12.5 KB)  TX bytes:9608 (9.6 KB)

qr-cd728374-d8 Link encap:Ethernet  HWaddr fa:16:3e:d7:5a:2f  
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fed7:5a2f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:64 errors:0 dropped:0 overruns:0 frame:0
          TX packets:89 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:9710 (9.7 KB)  TX bytes:10627 (10.6 KB)

qr-f915c799-96 Link encap:Ethernet  HWaddr fa:16:3e:96:89:3a  
          inet addr:10.0.1.1  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe96:893a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:594 (594.0 B)

这两个值是不同的,本应从192.168.19.130路由的数据包均发往192.168.19.129,导致VM无法出去。其实后者是quantum中与外网连接的port中的fixed_ips值:

+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                             |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+
| 10d13e25-cc01-4edc-aba4-5e2b3a6dff80 |      | fa:16:3e:e6:9e:30 | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.4"}       |
| 29c30020-2e91-4ffa-91e3-a8acef553641 |      | fa:16:3e:10:18:21 | {"subnet_id": "3f53264f-683b-45a8-a7ab-289afd2288b5", "ip_address": "192.168.19.130"} |
| 7e659611-43b3-4f52-b392-28ddd5051bca |      | fa:16:3e:9e:84:c8 | {"subnet_id": "3f53264f-683b-45a8-a7ab-289afd2288b5", "ip_address": "192.168.19.131"} |
| 7f000789-2e36-4aef-8d08-acb700ddde9f |      | fa:16:3e:07:92:81 | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.2"}       |
| 91da98b9-e9df-4a2c-b97d-02299d33fe89 |      | fa:16:3e:f7:42:d9 | {"subnet_id": "3f53264f-683b-45a8-a7ab-289afd2288b5", "ip_address": "192.168.19.132"} |
| a132b58c-238a-4b9f-92ce-c47521cda668 |      | fa:16:3e:31:81:8e | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.3"}       |
| b1a9afa6-6850-4044-a2b6-cca6c12fc6fa |      | fa:16:3e:89:2e:fb | {"subnet_id": "0636c5f2-70ab-4fb9-a7d5-986c92eaf1aa", "ip_address": "10.0.1.2"}       |
| b629349e-ad6e-427a-8aae-291f55ef4b32 |      | fa:16:3e:31:a2:cf | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.5"}       |
| cd728374-d89e-4f64-b437-b3e1580b49e9 |      | fa:16:3e:d7:5a:2f | {"subnet_id": "169ad3b8-c961-4128-b053-2d6d36afbe1f", "ip_address": "10.0.0.1"}       |
| f915c799-96aa-40bf-a3aa-06d43bc1c284 |      | fa:16:3e:96:89:3a | {"subnet_id": "0636c5f2-70ab-4fb9-a7d5-986c92eaf1aa", "ip_address": "10.0.1.1"}       |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------------+

如果设置该网络的网关为130,提示失败:

# quantum subnet-update userA-public --gateway_ip 192.168.19.130
Gateway ip 192.168.19.130 conflicts with allocation pool 192.168.19.130-192.168.19.254

在quantum代码中体现是:
agent/l3_agent.py

        ex_gw_ip = ex_gw_port['fixed_ips'][0]['ip_address']
        if not ip_lib.device_exists(interface_name,
                                    root_helper=self.root_helper,
                                    namespace=ri.ns_name()):
            
......

        gw_ip = ex_gw_port['subnet']['gateway_ip']
        if ex_gw_port['subnet']['gateway_ip']:
            cmd = ['route', 'add', 'default', 'gw', gw_ip]

不知道为什么这里有两个:ex_gw_ip和gw_ip,不一致导致这个问题。

workaround很简单:

# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 route del default gw 192.168.19.129
# ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 route add default gw 192.168.19.130

——————————我是分割线——————————————–
上面的workaround很是麻烦,每次重启l3agent都需要添加,我今天看了一下这个问题,其实还是因为我们对neutron网络不太理解造成的。我前面的方法是先将数据包扔到qg-f103a9f2-d6接口,然后在命名空间外的路由表中进行路由决策:

# ip netns exec qrouter-09be29ea-25f6-4a53-b3ab-8d0e13dc7198 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.19.130  0.0.0.0         UG    0      0        0 qg-f103a9f2-d6
100.0.0.0       0.0.0.0         255.255.255.0   U     0      0        0 qr-d8fcb028-ea
192.168.19.0    0.0.0.0         255.255.255.0   U     0      0        0 qg-f103a9f2-d6
200.0.0.0       0.0.0.0         255.255.255.0   U     0      0        0 qr-b422c431-d8

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.19.254  0.0.0.0         UG    100    0        0 br-ex
20.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
30.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth2
192.168.19.0    0.0.0.0         255.255.255.0   U     0      0        0 br-ex

数据包的流向是qr-d8fcb028-ea->(namespace routing)->qg-f103a9f2-d6->(routing)->br-ex->eth0->router

其实public-net本身就是一个外网,所以应该跟物理机的网络一致,也就是192.168.19.0/24,网关是物理网关192.168.19.254。这样,每次l3agent都会在命名空间中新建默认路由:

# ip netns exec qrouter-09be29ea-25f6-4a53-b3ab-8d0e13dc7198 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.19.254  0.0.0.0         UG    0      0        0 qg-f103a9f2-d6
100.0.0.0       0.0.0.0         255.255.255.0   U     0      0        0 qr-d8fcb028-ea
192.168.19.0    0.0.0.0         255.255.255.0   U     0      0        0 qg-f103a9f2-d6
200.0.0.0       0.0.0.0         255.255.255.0   U     0      0        0 qr-b422c431-d8

这样数据包到了这个命名空间后,直接经过路由决策从qg-f103a9f2-d6经过br-ex到eth0出去了。虽然数据包流向与前面的一样,但是从命名空间到物理网关还是在一个网络中流动。

VM上不了网的一个原因

Openstack中VM上不了网有很多原因,今天遇到一个,其实之前也遇到过,只是不熟了才调试了半天,悲剧。。。

现象:VM联网速度很慢,例如apt-get update能连上主机,但是半天下载不了多少东西

调试:因为VM能上网,所以开始以为是quantum的l3问题,在命名空间下查看iptables和route,均没有问题。。
root@controller:/usr/src/nova# ip netns
qdhcp-eb2fc4cd-d656-4e64-adc2-001d3cfbcebd
qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02
qdhcp-77a8d872-103a-4d8c-9f47-bc6ec34a2ff4
qdhcp-faacf658-dae9-4230-8fbc-7cde47c425b1

然后用抓包:
ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 tcpdump -i qg-29c30020-2e (外网网卡)
15:20:00.465882 IP 192.168.19.131 > likho.canonical.com: ICMP 192.168.19.131 unreachable – need to frag (mtu 1454), length 556
15:20:02.046704 IP likho.canonical.com.http > 192.168.19.131.56147: Flags [.], seq 1:1449, ack 256, win 61, options [nop,nop,TS val 3517237056 ecr 105897], length 1448
15:20:02.046825 IP 192.168.19.131 > likho.canonical.com: ICMP 192.168.19.131 unreachable – need to frag (mtu 1454), length 556
ip netns exec qrouter-b4721d20-9d39-4d4d-9c37-f18ecb460d02 tcpdump -i qr-cd728374-d8 (内网网卡)
15:20:02.046763 IP likho.canonical.com.http > 10.0.0.4.56147: Flags [.], seq 1:1449, ack 256, win 61, options [nop,nop,TS val 3517237056 ecr 105897], length 1448
15:20:02.046800 IP 10.0.0.4 > likho.canonical.com: ICMP 10.0.0.4 unreachable – need to frag (mtu 1454), length 556
15:20:02.541472 IP sudice.canonical.com.http > 10.0.0.4.39679: Flags [.], seq 1:1449, ack 258, win 61, options [nop,nop,TS val 1537132828 ecr 105323], length 1448
15:20:02.541502 IP 10.0.0.4 > sudice.canonical.com: ICMP 10.0.0.4 unreachable – need to frag (mtu 1454), length 556
发现出现很多unreachable的问题,开始以为是内网没经过SNAT出去,后来才发现错误信息重点是need to frag

原因是VM的MTU太小了,需要设置一个大一些的值,那么处理就很简单了,直接在VM中运行:
ifconfig eth0 mtu 1400

DONE

Floodlight REST module

1 download jackson, restlet lib

2 create AntiDDoSResource extending ServerResource

public class AntiDDoSResource extends ServerResource {

	@Get("json")
    public Object handleRequest() {
		ISecurityAppProviderService service = 
                (ISecurityAppProviderService)getContext().getAttributes().
                get(ISecurityAppProviderService.class.getCanonicalName());

        String op = (String) getRequestAttributes().get("op");
        String obj = (String) getRequestAttributes().get("obj");

        // REST API check status
        if (op.equalsIgnoreCase("status")) {
            if (service.isEnabled())
                return "{\"result\" : \"ADS enabled\"}";
            else
                return "{\"result\" : \"ADS disabled\"}";
        }

        // REST API enable firewall
        if (op.equalsIgnoreCase("enable")) {
        	service.run();
            return "{\"status\" : \"success\", \"details\" : \"ADS running\"}";
        } 
        
        // REST API disable firewall
        if (op.equalsIgnoreCase("disable")) {
            service.terminate();
            return "{\"status\" : \"success\", \"details\" : \"ADS stopped\"}";
        } 

        // no known options found
        return "{\"status\" : \"failure\", \"details\" : \"invalid operation: "+op+"/"+obj+"\"}";
    }

3 create a ADSWebRoutable class implementing RestletRoutable

public class ADSWebRoutable implements RestletRoutable {
@Override
public Router getRestlet(Context context) {
Router router = new Router(context);
router.attach("/{op}/json", AntiDDoSResource.class);
router.attach("/{op}/{obj}/json", AntiDDoSResource.class);
return router;
}

/**
* Set the base path for the Firewall
*/
@Override
public String basePath() {
return "/app/ads";
}
}

3 bind the ADSWebRoutable with