OpenStack コンテナサービス(zun)追加

All-in-oneのPackstack環境にコンテナサービスのzunを追加する手順をインストールメモがてら残した。

手順はここ(https://docs.openstack.org/zun/latest/install/controller-install.html)を参照。

コントローラノード向け設定

データベースにユーザ追加

# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 3728741
Server version: 10.3.35-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> GRANT ALL PRIVILEGES ON zun.* TO 'zun' \
  IDENTIFIED BY 'ZUN_DBPASS';
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON zun.* TO 'zun'@'%' \
  IDENTIFIED BY 'ZUN_DBPASS';
Query OK, 0 rows affected (0.000 sec)

MariaDB [(none)]> quit
Bye
#

zun service作成。

$ . admin-openrc
$ openstack user create --domain default --password-prompt zun
User Password:
Repeat User Password:
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | f4f40696f6914b5b9e9054b7e9cfb55d |
| name                | zun                              |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

$ openstack role add --project services --user zun admin

$ openstack service create --name zun --description "Container Service" container
+-------------+----------------------------------+
| Field       | Value                            |
+-------------+----------------------------------+
| description | Container Service                |
| enabled     | True                             |
| id          | 74d60a5188ee4d36b1ba036c23c45f2e |
| name        | zun                              |
| type        | container                        |
+-------------+----------------------------------+
$  openstack endpoint create --region RegionOne container public http://192.168.1.10:9517/v1
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| enabled      | True                             |
| id           | ae352beaa62c45899da2ec57d376e2ab |
| interface    | public                           |
| region       | RegionOne                        |
| region_id    | RegionOne                        |
| service_id   | 74d60a5188ee4d36b1ba036c23c45f2e |
| service_name | zun                              |
| service_type | container                        |
| url          | http://192.168.1.10:9517/v1      |
+--------------+----------------------------------+
$ openstack endpoint create --region RegionOne container internal http://192.168.1.10:9517/v1
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| enabled      | True                             |
| id           | d1e241ca8f664f05b1976c5a5628a284 |
| interface    | internal                         |
| region       | RegionOne                        |
| region_id    | RegionOne                        |
| service_id   | 74d60a5188ee4d36b1ba036c23c45f2e |
| service_name | zun                              |
| service_type | container                        |
| url          | http://192.168.1.10:9517/v1      |
+--------------+----------------------------------+
$ openstack endpoint create --region RegionOne container admin http://192.168.1.10:9517/v1
+--------------+----------------------------------+
| Field        | Value                            |
+--------------+----------------------------------+
| enabled      | True                             |
| id           | 911e3f76ba09463bad2833573bfb17ef |
| interface    | admin                            |
| region       | RegionOne                        |
| region_id    | RegionOne                        |
| service_id   | 74d60a5188ee4d36b1ba036c23c45f2e |
| service_name | zun                              |
| service_type | container                        |
| url          | http://192.168.1.10:9517/v1      |
+--------------+----------------------------------+

OSユーザ追加。

# groupadd --system zun
# useradd --home-dir "/var/lib/zun" \
      --create-home \
      --system \
      --shell /bin/false \
      -g zun \
      zun
# mkdir -p /etc/zun
# chown zun:zun /etc/zun
# dnf install python3-pip git python3-devel libffi-devel gcc openssl-devel
Updating Subscription Management repositories.
Last metadata expiration check: 2:15:22 ago on Fri 06 Jan 2023 07:01:20 PM JST.
Package python3-pip-9.0.3-22.el8.noarch is already installed.
Package git-2.31.1-2.el8.x86_64 is already installed.
Package python36-devel-3.6.8-38.module+el8.5.0+12207+5c5719bc.x86_64 is already installed.
Package gcc-8.5.0-15.el8.x86_64 is already installed.
Dependencies resolved.
=============================================================================================================================================================================
 Package                                    Architecture                  Version                                 Repository                                            Size
=============================================================================================================================================================================
Installing:
 libffi-devel                               x86_64                        3.1-23.el8                              rhel-8-for-x86_64-baseos-rpms                         29 k
 openssl-devel                              x86_64                        1:1.1.1k-7.el8_6                        rhel-8-for-x86_64-baseos-rpms                        2.3 M
Installing dependencies:
 keyutils-libs-devel                        x86_64                        1.5.10-9.el8                            rhel-8-for-x86_64-baseos-rpms                         48 k
 krb5-devel                                 x86_64                        1.18.2-22.el8_7                         rhel-8-for-x86_64-baseos-rpms                        561 k
 libcom_err-devel                           x86_64                        1.45.6-5.el8                            rhel-8-for-x86_64-baseos-rpms                         39 k
 libkadm5                                   x86_64                        1.18.2-22.el8_7                         rhel-8-for-x86_64-baseos-rpms                        187 k
 libselinux-devel                           x86_64                        2.9-6.el8                               rhel-8-for-x86_64-baseos-rpms                        200 k
 libsepol-devel                             x86_64                        2.9-3.el8                               rhel-8-for-x86_64-baseos-rpms                         87 k
 libverto-devel                             x86_64                        0.3.2-2.el8                             rhel-8-for-x86_64-baseos-rpms                         18 k
 pcre2-devel                                x86_64                        10.32-3.el8_6                           rhel-8-for-x86_64-baseos-rpms                        605 k
 pcre2-utf16                                x86_64                        10.32-3.el8_6                           rhel-8-for-x86_64-baseos-rpms                        229 k
 pcre2-utf32                                x86_64                        10.32-3.el8_6                           rhel-8-for-x86_64-baseos-rpms                        220 k

Transaction Summary
=============================================================================================================================================================================
Install  12 Packages

Total download size: 4.5 M
Installed size: 8.0 M
Is this ok [y/N]: y
～中略～
Installed:
  keyutils-libs-devel-1.5.10-9.el8.x86_64       krb5-devel-1.18.2-22.el8_7.x86_64       libcom_err-devel-1.45.6-5.el8.x86_64       libffi-devel-3.1-23.el8.x86_64
  libkadm5-1.18.2-22.el8_7.x86_64               libselinux-devel-2.9-6.el8.x86_64       libsepol-devel-2.9-3.el8.x86_64            libverto-devel-0.3.2-2.el8.x86_64
  openssl-devel-1:1.1.1k-7.el8_6.x86_64         pcre2-devel-10.32-3.el8_6.x86_64        pcre2-utf16-10.32-3.el8_6.x86_64           pcre2-utf32-10.32-3.el8_6.x86_64

Complete!

# cd /var/lib/zun
# git clone https://opendev.org/openstack/zun.git
Cloning into 'zun'...
remote: Enumerating objects: 12578, done.
remote: Counting objects: 100% (12578/12578), done.
remote: Compressing objects: 100% (2355/2355), done.
remote: Total 25247 (delta 12095), reused 10223 (delta 10223), pack-reused 12669
Receiving objects: 100% (25247/25247), 3.88 MiB | 4.09 MiB/s, done.
Resolving deltas: 100% (19377/19377), done.
# chown -R zun:zun zun
# git config --global --add safe.directory /var/lib/zun/zun
# cd zun
# pwd
/var/lib/zun/zun
# pip3 install -r requirements.txt
～中略～
Successfully installed docker-5.0.3 grpcio-1.48.2 kuryr-lib-2.6.0 protobuf-3.19.6 websocket-client-1.3.1
# python3 setup.py install

# su -s /bin/sh -c "oslo-config-generator --config-file etc/zun/zun-config-generator.conf" zun
# su -s /bin/sh -c "cp etc/zun/zun.conf.sample /etc/zun/zun.conf" zun
# su -s /bin/sh -c "cp etc/zun/api-paste.ini /etc/zun" zun

zun.conf 編集

# vim /etc/zun/zun.conf
[DEFAULT]
...
transport_url = rabbit://guest:PASS@192.168.1.10:5672/

[api]
...

port = 9517
[database]
...
connection=mysql+pymysql://zun:ZUN-PASS@192.168.1.10/zun
[keystone_auth]
...
auth_url=http://192.168.1.10:5000
project_domain_name = Default
username = zun
password = PASS

[keystone_authtoken]
...

www_authenticate_uri = http://192.168.1.10:5000/

auth_version = v3

username = zun
password = PASS
auth_protocol = http

[oslo_concurrency]
lock_path = /var/lib/zun/tmp

[websocket_proxy]
base_url = ws://$wsproxy_host:$wsproxy_port/
wsproxy_host = 192.168.1.10

wsproxy_port = 6784

DBアップデート

# su -s /bin/sh  -c "/usr/local/bin/zun-db-manage upgrade"  zun

起動スクリプト作成

# /etc/systemd/system/zun-api.service
[Unit]
Description = OpenStack Container Service API

[Service]
ExecStart = /usr/local/bin/zun-api
User = zun

[Install]
WantedBy = multi-user.target

# /etc/systemd/system/zun-wsproxy.service
[Unit]
Description = OpenStack Container Service Websocket Proxy

[Service]
ExecStart = /usr/local/bin/zun-wsproxy
User = zun

[Install]
WantedBy = multi-user.target

サービス起動

# systemctl enable zun-api zun-wsproxy
# systemctl start zun-api zun-wsproxy

コンピュートノード向け設定

All-in-One構成なのでコントローラと重複するところはスキップして差分だけ実施。

DockerとKuryr，Etcdを先にインストールする。

# Docker install
dnf config-manager ―-add-repo https://download.docker.com/linux/rhel/docker-ce.repo

RHEL8からはPodmanになったとのことでリポジトリのBaseURLをCentOSに変更しないとdockerをインストールできないらしい。

# vim /etc/yum.repos.d/docker-ce.repo

[docker-ce-stable]
name=Docker CE Stable - $basearch
#baseurl=https://download.docker.com/linux/fedora/$releasever/$basearch/stable #コメントアウト
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable # 追加

Dockerインストール

# dnf install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Kuryr-libnetworkインストール

<https://docs.openstack.org/kuryr-libnetwork/latest/install/controller-install.html>

$ openstack user create --domain default --password-prompt kuryr
$ openstack role add --project services --user kuryr admin

<https://docs.openstack.org/kuryr-libnetwork/latest/install/compute-install-ubuntu.html#prerequisites>

# groupadd --system kuryr
# useradd --home-dir "/var/lib/kuryr" --create-home --system --shell /bin/false -g kuryr kuryr
#  mkdir -p /etc/kuryr
# chown kuryr:kuryr /etc/kuryr
# cd /var/lib/kuryr/
# git clone -b master https://opendev.org/openstack/kuryr-libnetwork.git
# chown -R kuryr:kuryr kuryr-libnetwork
# cd kuryr-libnetwork/
# python3 setup.py install

# su -s /bin/sh -c "./tools/generate_config_file_samples.sh" kuryr
# su -s /bin/sh -c "cp etc/kuryr.conf.sample /etc/kuryr/kuryr.conf" kuryr

設定ファイル編集。

# /etc/kuryr/kuryr.conf

[DEFAULT]
...
bindir = /usr/local/libexec/kuryr

[neutron]
...
auth_url = http://192.168.1.10:5000/
www_authenticate_uri = http://192.168.1.10:5000/
username = kuryr
user_domain_name = Default
password = PASSWORD
project_name = services
project_domain_name = Default
auth_type = password

Etcdインストール

# dnf list installed etcd

# vim /etc/etcd/etcd.conf

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.1.10:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_NAME="controller"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.10:2379"
ETCD_INITIAL_CLUSTER="default=http://192.168.1.10:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ZUN設定

# mkdir -p /etc/cni/net.d
# chown zun:zun /etc/cni/net.d/

# dnf install numactl

# su -s /bin/sh -c "cp etc/zun/rootwrap.conf /etc/zun/rootwrap.conf" zun
# su -s /bin/sh -c "mkdir -p /etc/zun/rootwrap.d" zun
# su -s /bin/sh -c "cp etc/zun/rootwrap.d/*  /etc/zun/rootwrap.d/" zun
# su -s /bin/sh -c "cp etc/cni/net.d/* /etc/cni/net.d/" zun

# echo "zun ALL=(root) NOPASSWD: /usr/local/bin/zun-rootwrap /etc/zun/rootwrap.conf *" | sudo tee /etc/sudoers.d/zun-rootwrap zun ALL=(root) NOPASSWD: /usr/local/bin/zun-rootwrap     /etc/zun/rootwrap.conf *

# /etc/zun/zun.conf

[DEFAULT]
...
state_path = /var/lib/zun

[compute]
...
host_shared_with_nova = true

Docker と Kuryr 設定

# mkdir -p /etc/systemd/system/docker.service.d

dockerサービス起動設定

# /etc/systemd/system/docker.service.d/docker.conf
[Service]

ExecStart=
ExecStart=/usr/bin/dockerd --group zun -H tcp://192.168.1.10:2375 -H unix:///var/run/docker.sock --cluster-store etcd://192.168.1.10:2379

containerd設定ファイル生成

# containerd config default > /etc/containerd/config.toml

containerd設定ファイル編集

# /etc/containerd/config.toml

[grpc]
  ...
  gid = ZUN_GROUP_ID

再起動

# systemctl restart containerd

CNIインストール&設定

# mkdir -p /opt/cni/bin
# curl -L https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz | tar -C /opt/cni/bin -xzvf - ./loopback

# install -o zun -m 0555 -D /usr/local/bin/zun-cni /opt/cni/bin/zun-cni

サービス設定

# /etc/systemd/system/zun-compute.service
[Unit]
Description = OpenStack Container Service Compute Agent

[Service]
ExecStart = /usr/local/bin/zun-compute
User = zun

[Install]
WantedBy = multi-user.target

# /etc/systemd/system/zun-cni-daemon.service
[Unit]
Description = OpenStack Container Service CNI daemon

[Service]
ExecStart = /usr/local/bin/zun-cni-daemon
User = zun

[Install]
WantedBy = multi-user.target

サービス起動

# systemctl enable zun-compute zun-cni-daemon
# systemctl start zun-compute zun-cni-daemon

ここを参考。(https://docs.openstack.org/zun-ui/latest/)

Horizen周り設定

$ git clone https://github.com/openstack/zun-ui
$ cd zun-ui

$ sudo pip3 install .

$ sudo cp ./zun_ui/enabled/* /usr/share/openstack-dashboard/openstack_dashboard/local/enabled
$ sudo python3 /usr/share/openstack-dashboard/manage.py collectstatic

$ sudo systemctrl restart httpd

これで完了。