Openstack Folsom 安装比较复杂，尤其是Quantum部分，新的内容很多。Quantum的租户网络有两种模式：GRE和VLAN模式，这两种方式配置有很大的区别，一个明显的区别就是控制节点，Vlan模式2块网卡，GRE模式需要3块网卡。这篇文档就是采用GRE模式，控制节点需要3块网卡

英文原文

https://github.com/jedipunkz/openstack_folsom_deploy

https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/stable/GRE/OpenStack_Folsom_Install_Guide_WebVersion.rst

我基本会参考原文做翻译，下面的和原文差异的地方

1. 我的网络会会原文不同, 文档我会在真实的环境下验证，由于每个人的网络基本都是不一样，所以文档和IP相关的地方，我基本采用变量，比较灵活的方式，大家可以用sed命令实现修改。很多朋友安装不成功，基本都是因为更换IP，导致有地方没修改，所以提供sed的命令修改。
2. 对原文提供的两个keystone导入数据的脚本做了细微的修改，主要是采用变量，让他更加灵活。
3. mysql直接采用IP访问，而不是localhost
4. keystone的token采用随机生成，而不是password

 

文档修改记录

• 2012年9月8日，文档还在草稿中。
• 2012年9月11日：完成文档大部分内容，目前quantum的安装包有冲突。等待上游修复。
• 2012年9月21日：基本完成控制节点的安装，登陆dashboard，创建网络。目前dashboard还是需要http:/ip/horizon 访问。
• 2012年10月11日: 把控制节点安装好。目前Folsom的源，算是已经正式发布。已经成功登陆dashbaord。
• 2012年10月12日：加上计算节点，不过目前还是有问题。看不到新加入的计算节点。
• 2012年10月15日: 基本调试通过.成功创建了第一个虚拟机,不过目前网络还是不通,无法访问.vnc还是有问题.
• 2012年10月16日：根据Essex版本的vnc设置进行调整，目前vnc已经可以工作。已经给作者反馈 。目前发现metadata不工作，密钥没有注入到虚拟机里。调整了一下nova.conf 文件。
• 2012年11月2日：目前quantum的网络还是无法正常工作。需要更多的时间去了解和学习。
• 2012年11月27日：终于可以实现访问虚拟机，不过目前虚拟机还无法访问外网。正在调试中，希望这个星期能完成整个文档。
• 2012年11月29日：经过多次重复安装，基本已经实现虚拟机的访问。不过目前虚拟机还是无法访问外部网络，估计还是quantum的bug，今天也是Folsom发布第一个补丁包，希望可以在ubuntu集成补丁包后，修复所有相关的bug。目前文档已经基本可用。
• 2013年1月5日：作者的原文也做了很多调整，我根据调整也校对了一遍文档，改进了一些地方。目前原文已经把控制节点和网络节点分开，这样更有利于理解，不过调整太大，我就不修改。目前就剩下一个主要问题，虚拟机无法访问外部的网络。
• 2013年1月17日:源已经更新到 Folsom 2012.2.1, 发现确实修复了几个明显的bug,也顺便调整了一下文档. 不过虚拟机无法访问外网的问题,还是没有解决.这个确实很郁闷.

 

介绍

	控制节点（3块网卡）	计算节点（2块网卡）
管理网络（eth0）	10.1.199.53/24	10.1.199.6/24
VMs Networks with OVS in tunnel mode	10.0.0.3/24	10.0.0.4/24
Public Bridge	不需要设置IP
hostname	controller	compute1
服务	MySQL RabbitMQ Nova Glance Keystone Quantum	kvm quantum nova-compute

 

要求

1. 控制节点一定需要3块网卡，计算节点2块网卡，如果测试迁移，那么需要2台计算节点
2. 机器支持kvm，可以通过运行命令 kvm-ok 检测是否支持
3. 全部的命令都是在root下运行

 

 

控制节点

操作系统

安装ubuntu 12.04.1 Server版本，最小化安装，只需要安装SSH server就可以。Cinder 需要一个单独的分区或者硬盘。

源

目前Folsom进入ubuntu 12.04的官方的源，不过需要你手工添加。源的官方说明

cat <<EOF >>/etc/apt/sources.list
deb  http://ubuntu-cloud.archive.canonical.com/ubuntu precise-proposed/folsom main
deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/folsom main
EOF

运行下面命令

apt-get install ubuntu-cloud-keyring
apt-get update && apt-get -y dist-upgrade

 

Hostname设置

 

# cat /etc/hostname 
controller

# cat /etc/hosts
127.0.0.1       localhost
10.1.199.53      controller.chenshake.com        controller
10.1.199.6      compute1.chenshake.com  compute1

# hostname
controller

# hostname -f
controller.chenshake.com

 

网络

直接设置 /etc/network/interface

root@node53:~# cat /etc/network/interfaces 
# This file describes network interfaces avaiulable on your system
# and how to activate them. For more information, see interfaces(5).
# Modified by convert_static.sh.
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 10.1.199.53
hwaddress ether 00:25:90:2d:7a:42  
netmask 255.255.255.0
network 10.1.199.0
gateway 10.1.199.1
dns-search chenshake.com
dns-nameservers 8.8.8.8

# VMs Networks with OVS in tunnel mode
auto eth1
    iface eth1 inet static
    address 10.0.0.3
    netmask 255.255.255.0

# Public Bridge
auto eth2
    iface eth2 inet manual
    up ifconfig $IFACE 0.0.0.0 up
    up ip link set $IFACE promisc on 
    down ip link set $IFACE promisc off
    down ifconfig $IFACE down

 

重启服务

/etc/init.d/networking restart

设置IP转发

sed -i -r 's/^\s*#(net\.ipv4\.ip_forward=1.*)/\1/' /etc/sysctl.conf
echo 1 > /proc/sys/net/ipv4/ip_forward

检查修改结果

# sysctl -p
net.ipv4.ip_forward = 1

修改完这些，重启机器

查看当前机器的路由

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.1.199.1      0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth1
10.1.199.0      *               255.255.255.0   U     0      0        0 eth0

NTP服务器

编辑 /etc/ntp.conf ，在 server ntp.ubuntu.com 下添加两行

server ntp.ubuntu.com
server 127.127.1.0
fudge 127.127.1.0 stratum 10

或者直接运行下面命令

sed -i 's/server ntp.ubuntu.com/server ntp.ubuntu.com\nserver 127.127.1.0\nfudge 127.127.1.0 stratum 10/g' /etc/ntp.conf

重启NTP服务

service ntp restart

环境变量

cat >/root/novarc <<EOF
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=password
export MYSQL_PASS=password
export SERVICE_PASSWORD=password
export RABBIT_PASSWORD=password
export FIXED_RANGE=10.0.0.0/24
export FLOATING_RANGE=$(/sbin/ifconfig eth0 | awk '/inet addr/ {print $2}' | cut -f2 -d ":" | awk -F "." '{print $1"."$2"."$3}').224/27
export OS_AUTH_URL="http://localhost:5000/v2.0/"
export SERVICE_ENDPOINT="http://localhost:35357/v2.0"
export SERVICE_TOKEN=$(openssl rand -hex 10)
export MASTER="$(/sbin/ifconfig eth0 | awk '/inet addr/ {print $2}' | cut -f2 -d ":")"
export LOCAL_IP="$(/sbin/ifconfig eth1 | awk '/inet addr/ {print $2}' | cut -f2 -d ":")"
export OS_TEST_TENANT=bank
export OS_TEST_USER=chenshake
export OS_TEST_NET=bank_net
export OS_TEST_ROUTER=bank_router
export OS_TEST_SUBNET=10.10.10.0/24
EOF

 

你可以根据你的需要调整用户的密码。

source novarc
echo "source novarc">>.bashrc

 

 

Mysql

下面是我们需要用到的数据库

数据库	用户	密码
mysql	root	password
nova	nova	password
keystone	keystone	password
glance	glance	password
cinder	cinder	password
quantum	quantum	password

 

安装

设置自动安装，无需输入密码

cat <<MYSQL_PRESEED | debconf-set-selections
mysql-server-5.5 mysql-server/root_password password $MYSQL_PASS
mysql-server-5.5 mysql-server/root_password_again password $MYSQL_PASS
mysql-server-5.5 mysql-server/start_on_boot boolean true
MYSQL_PRESEED

安装mysql

apt-get -y install mysql-server python-mysqldb curl

设置

运行远程访问mysql

sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf

重启服务

service mysql restart

创建数据库

mysql -uroot -p$MYSQL_PASS <<EOF
CREATE DATABASE nova;
GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY '$MYSQL_PASS';
CREATE DATABASE glance;
GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY '$MYSQL_PASS';
CREATE DATABASE keystone;
GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%'IDENTIFIED BY '$MYSQL_PASS';
CREATE DATABASE cinder;
GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%'IDENTIFIED BY '$MYSQL_PASS';
CREATE DATABASE quantum;
GRANT ALL PRIVILEGES ON quantum.* TO 'quantum'@'%'IDENTIFIED BY '$MYSQL_PASS';
FLUSH PRIVILEGES;
EOF

RabbitMQ

安装

apt-get -y install rabbitmq-server

设置

修改默认密码

我们把默认密码 guest，改成password

rabbitmqctl change_password guest $RABBIT_PASSWORD

Keystone

安装

apt-get -y install keystone

配置

编辑 /etc/keystone/keystone.conf

[DEFAULT]
admin_token = d111cf2d97251a9e0422
bind_host = 0.0.0.0
public_port = 5000
admin_port = 35357
compute_port = 8774
verbose = True
debug = True
log_file = keystone.log
log_dir = /var/log/keystone
log_config = /etc/keystone/logging.conf
[sql]
connection = mysql://keystone:password@10.1.199.53:3306/keystone
idle_timeout = 200

 

或者直接运行下面脚本

sed -i -e " s/# admin_token = ADMIN/admin_token = $SERVICE_TOKEN/g; s/# bind_host = 0.0.0.0/bind_host = 0.0.0.0/g; s/# public_port = 5000/public_port = 5000/g; s/# admin_port = 35357/admin_port = 35357/g; s/# compute_port = 8774/compute_port = 8774/g; s/# verbose = True/verbose = True/g; s/# idle_timeout/idle_timeout/g" /etc/keystone/keystone.conf

使用mysql数据库

sed -i '/connection = .*/{s|sqlite:///.*|mysql://'"keystone"':'"$MYSQL_PASS"'@'"$MASTER"'/keystone|g}' /etc/keystone/keystone.conf

重启服务和初始化数据库

service keystone restart
keystone-manage db_sync

导入keystone数据

keystone-data.sh

wget http://www.chenshake.com/wp-content/uploads/2012/09/keystone-data.sh_.txt
mv keystone-data.sh_.txt keystone-data.sh
bash keystone-data.sh

 

导入endpoint

keystone-endpoints.sh

wget http://www.chenshake.com/wp-content/uploads/2012/09/keystone-endpoints.sh_.txt
mv keystone-endpoints.sh_.txt keystone-endpoints.sh
bash keystone-endpoints.sh

测试

使用curl测试

curl -d '{"auth": {"tenantName": "admin", "passwordCredentials":{"username": "admin", "password": "password"}}}' -H "Content-type:application/json" http://$MASTER:35357/v2.0/tokens | python -mjson.tool

 

查看log

grep ERROR /var/log/keystone/keystone.log
ps -ef | grep -i keystone-all

 

Glance

安装

apt-get -y install glance

配置

编辑/etc/glance/glance-api.conf 和 /etc/glance/glance-registry.conf ,两个文件，都是修改4个地方

sql_connection = mysql://glance:password@10.1.199.53/glance
admin_tenant_name = service
admin_user = glance
admin_password = password

或者直接运行下面脚本实现

sed -i -e " s/%SERVICE_TENANT_NAME%/service/g; s/%SERVICE_USER%/glance/g; s/%SERVICE_PASSWORD%/$SERVICE_PASSWORD/g; " /etc/glance/glance-api.conf  /etc/glance/glance-registry.conf
sed -i '/sql_connection = .*/{s|sqlite:///.*|mysql://'"glance"':'"$MYSQL_PASS"'@'"$MASTER"'/glance|g}' /etc/glance/glance-registry.conf /etc/glance/glance-api.conf

编辑 /etc/glance/glance-api.conf

#notifier_strategy = noop
notifier_strategy = rabbit

#rabbit_password = guest
rabbit_password = password

运行下面命令进行修改

sed -i " s/notifier_strategy = noop/notifier_strategy = rabbit/g;s/rabbit_password = guest/rabbit_password = $RABBIT_PASSWORD/g;" /etc/glance/glance-api.conf

 

运行下面命令

cat <<EOF >>/etc/glance/glance-api.conf
flavor = keystone+cachemanagement
EOF
cat <<EOF >>/etc/glance/glance-registry.conf 
flavor = keystone
EOF

重启服务

service glance-api restart && service glance-registry restart

同步数据库

glance-manage db_sync

下载Image

我们下载CirrOS的image作为测试使用，只有10M。如果是ubuntu官方的image，220M，并且ubuntu官方的image，都是需要使用密钥登陆。

CirrOS

下载image

wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img

上传image

glance image-create --name=cirros-0.3.0-x86_64 --public  --container-format=bare \
--disk-format=qcow2 < /root/cirros-0.3.0-x86_64-disk.img

Cirros，是可以使用用户名和密码登陆，也可以使用密钥登陆

user:cirros
password:cubswin:)

 

Ubuntu官方image

下载image

wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

上传image

glance image-create --name="Ubuntu 12.04 cloudimg amd64" --public \
--container-format=ovf --disk-format=qcow2 < /root/precise-server-cloudimg-amd64-disk1.img

user：ubuntu

只能使用密钥登陆。

测试

查看image

glance image-list

 

查看image详细信息

glance image-show 12e2b864-9601-4506-b19d-3f663c0b2e15

Open-vSwitch

安装

apt-get install -y openvswitch-switch

配置

设置网络

ovs-vsctl add-br br-int
ovs-vsctl add-br br-ex
ovs-vsctl add-port br-ex eth2
ip link set up br-ex

 

大家可以通过下面命令来查看你创建的效果, 具体的用途,正在学习中.

ovs-vsct -h
ovs-vsctl list-br
ovs-vsctl show

查看结果

# ovs-vsctl list-br
br-ex
br-int

# ovs-vsctl show
89742cb3-5d15-4150-a278-a4054ab9c219
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth2"
            Interface "eth2"
    ovs_version: "1.4.0+build0"

 

Quantum

安装

apt-get -y install quantum-server python-cliff \
quantum-plugin-openvswitch-agent \
quantum-l3-agent quantum-dhcp-agent python-pyparsing

配置

编辑 /etc/quantum/quantum.conf

auth_strategy = keystone
fake_rabbit = False
rabbit_host = 10.1.199.53
rabbit_password = password

 

或者运行下面命令

sed -i -e " s/# auth_strategy/auth_strategy/g; s/# fake_rabbit/fake_rabbit/g; s/# rabbit_host = localhost/rabbit_host = $MASTER/g; s/# rabbit_password = guest/rabbit_password = $RABBIT_PASSWORD/g" /etc/quantum/quantum.conf

 

编辑 /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

sql_connection = mysql://quantum:password@10.1.199.53:3306/quantum

[OVS]
tenant_network_type = gre 
enable_tunneling = True
tunnel_id_ranges = 1:1000
integration_bridge = br-int
tunnel_bridge = br-tun
local_ip = 10.0.0.3

或者运行下面命令

sed -i -e " s/# Example: tenant_network_type = gre/tenant_network_type = gre/g; s/# Default: enable_tunneling = False/enable_tunneling = True/g; s/# Example: tunnel_id_ranges = 1:1000/tunnel_id_ranges = 1:1000/g; s/# Default: integration_bridge = br-int/integration_bridge = br-int/g; s/# Default: tunnel_bridge = br-tun/tunnel_bridge = br-tun/g; s/# Default: local_ip =/local_ip = $LOCAL_IP/g" /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

 

修改数据库

sed -i '/sql_connection = .*/{s|sqlite:///.*|mysql://'"quantum"':'"password"'@'"$MASTER"'/quantum|g}' /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

编辑 /etc/quantum/l3_agent.ini 和 /etc/quantum/api-paste.ini

[DEFAULT]
admin_tenant_name = service
admin_user = quantum
admin_password = password

或者运行下面命令

sed -i -e " s/%SERVICE_TENANT_NAME%/service/g; s/%SERVICE_USER%/quantum/g; s/%SERVICE_PASSWORD%/$SERVICE_PASSWORD/g; " /etc/quantum/l3_agent.ini
sed -i -e " s/%SERVICE_TENANT_NAME%/service/g; s/%SERVICE_USER%/quantum/g; s/%SERVICE_PASSWORD%/$SERVICE_PASSWORD/g; " /etc/quantum/api-paste.ini

编辑 /etc/quantum/l3_agent.ini

debug = True
use_namespaces = False
metadata_ip = 10.1.199.53

或者运行下面命令

sed -i -e " s/# debug = True/debug = True/g; s/# use_namespaces = True/use_namespaces = False/g; s/# metadata_ip =/metadata_ip = $MASTER/g" /etc/quantum/l3_agent.ini

编辑 /etc/quantum/dhcp_agent.ini

use_namespaces = False

或者运行命令

sed -i -e " s/# use_namespaces = True/use_namespaces = False/g; "  /etc/quantum/dhcp_agent.ini

 

重启服务

service quantum-server restart
service quantum-plugin-openvswitch-agent restart
service quantum-dhcp-agent restart
service quantum-l3-agent restart

对于Openstack别的组件，都是需要创建数据库的过程。不过quantum，你只需要重启quantum-server服务，他就会自动创建相关的表。

为demo用户创建网络

利用提供的脚本，为用户 Demo创建一个网络

wget http://www.chenshake.com/wp-content/uploads/2012/09/quantum-networking.sh_.txt
mv quantum-networking.sh_.txt quantum-networking.sh

需要对脚本进行一些修改。

##############################################################
### Public Network ###########################################
##############################################################

# Provider Router Information - what name should 
# this provider have in Quantum?
PROV_ROUTER_NAME="provider-router"

# Name of External Network (Don't change it!)
EXT_NET_NAME="ext_net"

# External Network addressing - our official 
# Internet IP address space
EXT_NET_CIDR="10.1.199.0/24"
EXT_NET_LEN=${EXT_NET_CIDR#*/}

# External bridge that we have configured 
# into l3_agent.ini (Don't change it!)
EXT_NET_BRIDGE=br-ex

# IP of external bridge (br-ex) - this node's 
# IP in our official Internet IP address space:
EXT_GW_IP="10.1.199.13"

# IP of the Public Network Gateway - The 
# default GW in our official Internet IP address space:
EXT_NET_GATEWAY="10.1.199.1"

# Floating IP range
POOL_FLOATING_START="10.1.199.130"      # First public IP to be used for VMs
POOL_FLOATING_END="10.1.199.150"        # Last public IP to be used for VMs 

###############################################################

上面最让人困惑的设置就是：EXT_GW_IP, 这其实是控制节点的eth2的IP地址，不过这个IP地址，不是通过/etc/network/interface 设置，而是通过这个脚本设置。运行完脚本，你就可以ping通这个IP。

改完后，运行脚本

root@node53:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.1.199.1      0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth1
10.1.199.0      *               255.255.255.0   U     0      0        0 eth0
root@node53:~# bash quantum-networking.sh 
Added interface to router f69ecf3d-d476-433a-82a6-de20614b9d32
Created a new subnet:
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.1.199.130", "end": "10.1.199.150"} |
| cidr             | 10.1.199.0/24                                    |
| dns_nameservers  |                                                  |
| enable_dhcp      | False                                            |
| gateway_ip       | 10.1.199.1                                       |
| host_routes      |                                                  |
| id               | ef65a5bd-39d2-496f-a042-7234b5b8956e             |
| ip_version       | 4                                                |
| name             |                                                  |
| network_id       | 92c05466-6f80-4e5f-bbc3-59987df8d489             |
| tenant_id        | ab38cf34ab0a4a9995c84a53044a2269                 |
+------------------+--------------------------------------------------+
Set gateway for router f69ecf3d-d476-433a-82a6-de20614b9d32
root@node53:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.1.199.1      0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth1
10.1.199.0      *               255.255.255.0   U     0      0        0 eth0
10.1.199.0      *               255.255.255.0   U     0      0        0 br-ex
10.5.5.0        *               255.255.255.0   U     0      0        0 tap218751e0-6d

修改/etc/quantum/l3_agent.ini，修改 router 和 external network. 我直接使用下面命令修改。

router=$(quantum router-list | awk '/provider-router/ {print $2}')
ext_net=$(quantum net-list | awk '/ext_net/ {print $2}')
sed -i -e " s/# router_id =/router_id = $router/g; s/# gateway_external_network_id =/gateway_external_network_id = $ext_net/g;" /etc/quantum/l3_agent.ini 

 

这个时候，需要重启l3agent服务

service quantum-l3-agent restart

这时候你查看路由表

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.1.199.1      0.0.0.0         UG    0      0        0 eth0
default         10.1.199.1      0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth1
10.1.199.0      *               255.255.255.0   U     0      0        0 eth0
10.1.199.0      *               255.255.255.0   U     0      0        0 br-ex
10.1.199.0      *               255.255.255.0   U     0      0        0 qg-fba8f518-45
10.5.5.0        *               255.255.255.0   U     0      0        0 tap218751e0-6d
10.5.5.0        *               255.255.255.0   U     0      0        0 qr-61f55d19-9e

查看IP

可以看到很多信息。

root@node53:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:2d:7a:18 brd ff:ff:ff:ff:ff:ff
    inet 10.1.199.53/24 brd 10.1.199.255 scope global eth0
    inet6 fe80::225:90ff:fe2d:7a18/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:2d:7a:19 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global eth1
    inet6 fe80::225:90ff:fe2d:7a19/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:25:90:3b:23:c8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::225:90ff:fe3b:23c8/64 scope link 
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:25:90:3b:23:c9 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ba:03:3e:6c:9b:42 brd ff:ff:ff:ff:ff:ff
7: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:25:90:3b:23:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.199.13/24 scope global br-ex
8: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether ba:43:40:98:d2:4a brd ff:ff:ff:ff:ff:ff
9: tapd84b2276-bc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:a2:88:0e brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.2/24 brd 10.5.5.255 scope global tapd84b2276-bc
    inet6 fe80::f816:3eff:fea2:880e/64 scope link 
       valid_lft forever preferred_lft forever
10: qr-7afa9a7d-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:8e:9c:48 brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.1/24 brd 10.5.5.255 scope global qr-7afa9a7d-be
    inet6 fe80::f816:3eff:fe8e:9c48/64 scope link 
       valid_lft forever preferred_lft forever
11: qg-2a8f838e-06: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:b9:75:b8 brd ff:ff:ff:ff:ff:ff
    inet 10.1.199.130/24 brd 10.1.199.255 scope global qg-2a8f838e-06
    inet6 fe80::f816:3eff:feb9:75b8/64 scope link 
       valid_lft forever preferred_lft forever

Cinder

安装

apt-get install -y cinder-api cinder-scheduler cinder-volume iscsitarget \
open-iscsi iscsitarget-dkms python-cinderclient

配置

分区

我的硬盘专门一个分区给volume使用

umount /dev/sda5
pvcreate /dev/sda5
vgcreate cinder-volumes /dev/sda5

去掉开机挂载

sed -i '/nova-volume/s/^/#/' /etc/fstab

iscsi

sed -i 's/false/true/g' /etc/default/iscsitarget
service iscsitarget restart
service open-iscsi restart

编辑 /etc/cinder/cinder.conf ,直接运行下面命令就可以.

cat >/etc/cinder/cinder.conf <<EOF
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
sql_connection = mysql://cinder:$MYSQL_PASS@$MASTER:3306/cinder
iscsi_helper = ietadm 
volume_group = cinder-volumes
rabbit_password= $RABBIT_PASSWORD
logdir=/var/log/cinder
verbose=true
auth_strategy = keystone
EOF

编辑 /etc/cinder/api-paste.ini

admin_tenant_name = service
admin_user = cinder 
admin_password = password

或者用下面命令

sed -i -e " s/%SERVICE_TENANT_NAME%/service/g; s/%SERVICE_USER%/cinder/g; s/%SERVICE_PASSWORD%/$SERVICE_PASSWORD/g; " /etc/cinder/api-paste.ini

同步数据库

cinder-manage db sync

重启服务

service cinder-api restart
service cinder-scheduler  restart
service cinder-volume restart

Nova

安装

apt-get -y install nova-api nova-cert nova-common \
nova-scheduler python-nova python-novaclient nova-consoleauth novnc nova-novncproxy

配置

编辑 /etc/nova/api-paste.ini

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
auth_host = 10.1.199.53
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = password
signing_dirname = /tmp/keystone-signing-nova

 

或者直接运行命令

sed -i -e " s/127.0.0.1/$MASTER/g; s/%SERVICE_TENANT_NAME%/service/g; s/%SERVICE_USER%/nova/g; s/%SERVICE_PASSWORD%/$SERVICE_PASSWORD/g; " /etc/nova/api-paste.ini

 

创建 /etc/nova/nova.conf 文件，直接copy下面的命令，运行就可以。

cat >/etc/nova/nova.conf <<EOF
[DEFAULT]

# MySQL Connection #
sql_connection=mysql://nova:$MYSQL_PASS@$MASTER/nova

# nova-scheduler #
rabbit_host=$MASTER
rabbit_password=$RABBIT_PASSWORD
scheduler_driver=nova.scheduler.simple.SimpleScheduler
#compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

# nova-api #
cc_host=$MASTER
auth_strategy=keystone
s3_host=$MASTER
ec2_host=$MASTER
nova_url=http://$MASTER:8774/v1.1/
ec2_url=http://$MASTER:8773/services/Cloud
keystone_ec2_url=http://$MASTER:5000/v2.0/ec2tokens
api_paste_config=/etc/nova/api-paste.ini
allow_admin_api=true
use_deprecated_auth=false
ec2_private_dns_show_ip=True
dmz_cidr=169.254.169.254/32
ec2_dmz_host=169.254.169.254
metadata_host=$MASTER
metadata_listen=0.0.0.0
enabled_apis=ec2,osapi_compute,metadata

# Networking #
network_api_class=nova.network.quantumv2.api.API
quantum_url=http://$MASTER:9696
quantum_auth_strategy=keystone
quantum_admin_tenant_name=service
quantum_admin_username=quantum
quantum_admin_password=$SERVICE_PASSWORD
quantum_admin_auth_url=http://$MASTER:35357/v2.0
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver  
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver

# Compute #
#compute_driver=libvirt.LibvirtDriver

# Cinder #
volume_api_class=nova.volume.cinder.API

# Glance #
glance_api_servers=$MASTER:9292
image_service=nova.image.glance.GlanceImageService

# novnc #
novnc_enable=true
novncproxy_base_url=http://$MASTER:6080/vnc_auto.html
vncserver_proxyclient_address=$MASTER
vncserver_listen=$MASTER

# Misc #
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
#root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
rootwrap_config=/etc/nova/rootwrap.conf
#verbose=true
verbose=false
EOF

 

同步数据库

nova-manage db sync

重启服务

service nova-api restart
service nova-cert restart
service nova-consoleauth restart
service nova-scheduler restart
service nova-novncproxy restart

Horizon

安装

apt-get -y install apache2 libapache2-mod-wsgi openstack-dashboard memcached python-memcache

删除ubuntu自带的theme 编辑 /etc/openstack-dashboard/local_settings.py

#Comment these lines
#Enable the Ubuntu theme if it is present.
#try:
#    from ubuntu_theme import *
#except ImportError:
#    pass

或者运行下面命令

sed -i '150,153s/^/#/' /etc/openstack-dashboard/local_settings.py

 

Reload 服务

service apache2 restart; service memcached restart

 

 

访问

http://10.1.199.53/horizon
user：admin
pass：password
或者
user:demo
pass:password

看一下中文的Dashboard，由于在控制节点没有安装计算服务，所以你是无法创建虚拟机。

 

计算节点

操作系统

操作系统最小化安装，ssh server就可以。

源

添加Folsom源

cat <<EOF >>/etc/apt/sources.list
deb  http://ubuntu-cloud.archive.canonical.com/ubuntu precise-proposed/folsom main
deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/folsom main
EOF

运行下面命令

apt-get install ubuntu-cloud-keyring
apt-get update && apt-get -y dist-upgrade

网络

# cat /etc/network/interfaces 
# This file describes network interfaces avaiulable on your system
# and how to activate them. For more information, see interfaces(5).
# Modified by convert_static.sh.
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 10.1.199.6
hwaddress ether 00:25:90:2d:7a:42  
netmask 255.255.255.0
network 10.1.199.0
gateway 10.1.199.1
dns-search chenshake.com
dns-nameservers 8.8.8.8

# VMs Networks with OVS in tunnel mode
auto eth1
    iface eth1 inet static
    address 10.0.0.4
    netmask 255.255.255.0

 

重启网络

/etc/init.d/networking restart

 

IP转发

sed -i -r 's/^\s*#(net\.ipv4\.ip_forward=1.*)/\1/' /etc/sysctl.conf
echo 1 > /proc/sys/net/ipv4/ip_forward 

 

环境变量

cat >/root/novarc <<EOF
export CONTROLLER_IP=10.1.199.53
export MASTER="$(/sbin/ifconfig eth0 | awk '/inet addr/ {print $2}' | cut -f2 -d ":")"
export LOCAL_IP="$(/sbin/ifconfig eth1 | awk '/inet addr/ {print $2}' | cut -f2 -d ":")"
EOF

你根据你的情况，调整控制节点的IP

source novarc
echo "source novarc">>.bashrc

 

 

NTP

apt-get -y install ntp

设置

编辑 /etc/ntp.conf, 指向控制节点

server 10.1.199.53

 

或者运行命令

sed -i -e " s/server ntp.ubuntu.com/server $CONTROLLER_IP/g" /etc/ntp.conf

 

重启服务

service ntp restart

Hypervisor

apt-get install -y kvm libvirt-bin pm-utils

 

编辑 /etc/libvirt/qemu.conf ，添加下面内容

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc", "/dev/hpet","/dev/net/tun",
]

 

或者运行命令：这个地方用命令修改有点复杂，还没找到太好的办法。

cat <<EOF>>/etc/libvirt/qemu.conf
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc", "/dev/hpet","/dev/net/tun",
]
EOF

 

删除默认 virtual bridge

virsh net-destroy default
virsh net-undefine default

允许迁移

编辑 /etc/libvirt/libvirtd.conf, 去掉这三行的注释

listen_tls = 0
listen_tcp = 1
auth_tcp = "none" 

 

或者运行下面命令

sed -i '/#listen_tls/s/#listen_tls/listen_tls/; /#listen_tcp/s/#listen_tcp/listen_tcp/; /#auth_tcp/s/#auth_tcp/auth_tcp/; /auth_tcp/s/sasl/none/'  /etc/libvirt/libvirtd.conf

 

编辑 /etc/init/libvirt-bin.conf

env libvirtd_opts="-d -l" 

或者使用命令

sed -i '/env libvirtd_opts/s/-d/-d -l/' /etc/init/libvirt-bin.conf

 

编辑 /etc/default/libvirt-bin

libvirtd_opts="-d -l"

 

或者使用命令

sed -i '/libvirtd_opts/s/-d/-d -l/' /etc/default/libvirt-bin

 

重启服务

service libvirt-bin restart

 

Open-vSwitch

apt-get install -y openvswitch-switch

 

配置bridge

ovs-vsctl add-br br-int

Quantum

apt-get -y install quantum-plugin-openvswitch-agent

 

编辑 /etc/quantum/quantum.conf , 修改和控制节点一样，直接从控制直接复制过来

scp root@$CONTROLLER_IP:/etc/quantum/quantum.conf /etc/quantum/quantum.conf

编辑 /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini，

可以从控制节点copy过来,只需要修改local_IP就可以.

scp root@$CONTROLLER_IP:/etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

 

修改 local_ip

sed -i 's/^local_ip.*$/local_ip = '$LOCAL_IP'/g' /etc/quantum/plugins/openvswitch//ovs_quantum_plugin.ini 

 

重启服务

service openvswitch-switch restart
service quantum-plugin-openvswitch-agent restart

 

Nova

apt-get -y install nova-compute-kvm novnc nova-novncproxy nova-api-metadata

 

编辑 /etc/nova/api-paste.ini

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
auth_host = 10.1.199.53
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = nova
admin_password = password
signing_dirname = /tmp/keystone-signing-nova

 

或者运行下面命令,直接从控制节点复制过来就可以。

scp root@$CONTROLLER_IP:/etc/nova/api-paste.ini /etc/nova/

 

编辑 /etc/nova/nova-compute.conf

[DEFAULT]
libvirt_type=kvm
libvirt_ovs_bridge=br-int
libvirt_vif_type=ethernet
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
libvirt_use_virtio_for_bridges=True

 

或者运行下面命令

cat > /etc/nova/nova-compute.conf <<EOF
[DEFAULT]
libvirt_type=kvm
libvirt_ovs_bridge=br-int
libvirt_vif_type=ethernet
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
libvirt_use_virtio_for_bridges=True
EOF

 

编辑  /etc/nova/nova.conf。我们可以从控制节点copy过来修改

scp root@$CONTROLLER_IP:/etc/nova/nova.conf /etc/nova/nova.conf

修改下面内容

metadata_host=10.1.199.6
enabled_apis=metadata

# Compute #
compute_driver=libvirt.LibvirtDriver

# novnc #
novnc_enable=true
novncproxy_base_url=http://10.1.199.53:6080/vnc_auto.html
vncserver_proxyclient_address=10.1.199.6
vncserver_listen=10.1.199.6

 

可以使用命令

sed -i "/metadata_host/s/$CONTROLLER_IP/$MASTER/; s/^enabled_apis.*$/enabled_apis=metadata/g; s/#compute_driver/compute_driver/g; /vncserver_proxyclient_address/s/$CONTROLLER_IP/$MASTER/; /vncserver_listen/s/$CONTROLLER_IP/$MASTER/" /etc/nova/nova.conf

 

重启服务

service nova-novncproxy restart 
service nova-compute restart
service nova-api-metadata restart

这个时候,你就可以通过命令看到计算节点和控制节点。

创建VM

这个操作是控制节点运行, 由于quantum的脚本，已经为demo的用户创建了一个网络，所以我们就直接用demo的用户操作。

cat > /root/demo << EOF 
export OS_USERNAME=admin
export OS_TENANT_NAME=demo
export OS_PASSWORD=password
export OS_AUTH_URL=http://127.0.0.1:35357/v2.0/
export PS1="[\u@\h \W(demo)]\$ "
EOF

运行

. demo

创建虚拟机，你需要根据情况调整image id

nova keypair-add oskey > oskey.priv
chmod 600 oskey.priv
nova flavor-list
nova image-list
nova boot --flavor 2 --key_name oskey --image ea3ffba1-065e-483f-bfe2-c84184ee76be test1
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

这个时候虚拟机就创建好了。查看虚拟机的Ip

nova list
+--------------------------------------+-------+--------+-------------------+
| ID                                   | Name  | Status | Networks          |
+--------------------------------------+-------+--------+-------------------+
| e1425c3a-9930-4ff7-b8f8-fdb4de4e96d9 | test1 | ACTIVE | demo-net=10.5.5.3 |
+--------------------------------------+-------+--------+-------------------+

这个时候就可以ssh到虚拟机

ssh -i oskey.priv ubuntu@10.5.5.3

设置floating IP

$ quantum floatingip-create ext_net
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 10.1.199.131                         |
| floating_network_id | 92c05466-6f80-4e5f-bbc3-59987df8d489 |
| id                  | afb220f3-ecfc-4919-b6de-eb636b796933 |
| port_id             |                                      |
| router_id           |                                      |
| tenant_id           | c336b7f576c842b48471e1cc6072ddcb     |
+---------------------+--------------------------------------+
[root@node53 ~(demo)]$ quantum port-list
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                           |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| 218751e0-6d36-4a43-8f9b-0dd07e5f6964 |      | fa:16:3e:c1:b8:f6 | {"subnet_id": "7e158f08-f24d-4fe0-bf5f-570c3c4ac5c7", "ip_address": "10.5.5.2"}     |
| 61f55d19-9eed-46af-ad9f-c071fd432cc8 |      | fa:16:3e:a1:6d:d2 | {"subnet_id": "7e158f08-f24d-4fe0-bf5f-570c3c4ac5c7", "ip_address": "10.5.5.1"}     |
| 73fd2420-b759-48c1-8c82-d8d9aeb3f757 |      | fa:16:3e:1b:70:2a | {"subnet_id": "7e158f08-f24d-4fe0-bf5f-570c3c4ac5c7", "ip_address": "10.5.5.3"}     |
| 8051fc93-d6fa-458e-bc05-5f10da2f5eb0 |      | fa:16:3e:b4:c5:dc | {"subnet_id": "ef65a5bd-39d2-496f-a042-7234b5b8956e", "ip_address": "10.1.199.131"} |
| fba8f518-45ad-4b81-a520-8a739eddb848 |      | fa:16:3e:6f:9b:fb | {"subnet_id": "ef65a5bd-39d2-496f-a042-7234b5b8956e", "ip_address": "10.1.199.130"} |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
[root@node53 ~(demo)]$ quantum floatingip-associate afb220f3-ecfc-4919-b6de-eb636b796933 73fd2420-b759-48c1-8c82-d8d9aeb3f757
Associated floatingip afb220f3-ecfc-4919-b6de-eb636b796933

 

目前存在的问题

ssh到虚拟机里，还是无法访问外网。

 

附录

由于上面已经使用quantum的脚本创建网络，所以这个时候，你就可以使用demo的账号创建虚拟机。下面是通过手工的方式创建网络。

网络

这也是难点，在控制节点执行下面的命令，我尽可能把命令变得简单。

创建一个租户，用户

keystone tenant-create --name $OS_TEST_TENANT
tenant=$(keystone tenant-list | awk '/'"$OS_TEST_TENANT"'/ {print $2}')
role=$(keystone role-list | awk '/Member/ {print $2}')
keystone user-create --name=$OS_TEST_USER --pass=$OS_PASSWORD --tenant-id $tenant
user=$(keystone user-list | awk '/'"$OS_TEST_USER"'/ {print $2}')
keystone user-role-add --tenant-id $tenant --user-id $user --role-id $role

 

为租户创建网络

quantum net-create --tenant-id $tenant $OS_TEST_NET

为租户网络创建子网

quantum subnet-create --tenant-id $tenant $OS_TEST_NET $OS_TEST_SUBNET

 

为租户创建路由

quantum router-create --tenant-id $tenant $OS_TEST_ROUTER

 

为子网添加路由

subnet=$(quantum net-list | awk '/'"$OS_TEST_NET"'/ {print $6}')
router=$(quantum router-list | awk '/'"$OS_TEST_ROUTER"'/ {print $2}')
quantum router-interface-add $router $subnet

为service租户创建一个外部网络

service=$(keystone tenant-list | awk '/service/ {print $2}') 
quantum net-create --tenant-id $service ext_net --router:external=True

 

编辑 etc/quantum/l3_agent.ini

gateway_external_net_id = $id_of_ext_net
router_id = $your_router_id

可以考虑用下面命令实现修改

ext_net=$(quantum net-list | awk '/ext_net/ {print $2}')
sed -i -e " s/# router_id =/router_id = $router/g; s/# gateway_external_net_id =/gateway_external_net_id = $ext_net/g;" /etc/quantum/l3_agent.ini 

 

重启L3agent

service quantum-l3-agent restart

创建Floating Ip网段

quantum subnet-create --tenant-id $service --allocation-pool start=10.1.199.102,end=10.1.199.126 --gateway 10.1.199.1 ext_net 10.1.199.100/24 --enable_dhcp=False

 

设置外部网络的路由

quantum router-gateway-set $router $ext_net

 

设置IP地址

quantum port-list -- --device_id $router --device_owner network:router_gateway

 

第一个虚拟机

由于Dashboard不支持Floating IP关联，所以只能命令行下操作，下面切换到上面创建的用户环境下。参考novarc进行调整

cat > /root/chenshakerc << EOF 
export OS_USERNAME=chenshake
export OS_TENANT_NAME=bank
export OS_PASSWORD=password
export OS_AUTH_URL=http://127.0.0.1:35357/v2.0/
export PS1="[\u@\h \W(chenshake)]\$ "
EOF

 

切换到用户下

. chenshakerc

创建虚拟机，你需要根据情况调整image id

nova keypair-add oskey > oskey.priv
chmod 600 oskey.priv
nova flavor-list
nova image-list
nova boot --flavor 2 --key_name oskey --image ea3ffba1-065e-483f-bfe2-c84184ee76be test1
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

 

这个时候，你可以通过下面nova list

申请Floating IP

quantum floatingip-create ext_net

查看port，通过IP，可以知道VM的port ID

quantum port-list

 

Floating IP关联到虚拟机，把上面的两个id替换上就ok。

quantum floatingip-associate $put_id_floating_ip $put_id_vm_port

 

查看IP情况

quantum floatingip-list