我的博客

常规获取ck教程

your@email.com (你的名字) — Fri, 29 Dec 2023 14:44:51 +0000

常规获取ck教程

作者：你的名字（your@email.com）

这里以JD为例

打开[JD](京东登录注册 (jd.com))网站
开启调试模式（F12）
登录账号
打开网络，找到signindex
找到pt-key以及pt-pin字段

本文2023-12-29首发于我的博客，最后修改于2023-12-29

K8s 持久化存储流程

your@email.com (你的名字) — Tue, 24 Oct 2023 10:29:33 +0000

K8s 持久化存储流程

作者：你的名字（your@email.com）

**导读：**众所周知，K8s 的持久化存储（Persistent Storage）保证了应用数据独立于应用生命周期而存在，但其内部实现却少有人提及。K8s 内部的存储流程到底是怎样的？PV、PVC、StorageClass、Kubelet、CSI 插件等之间的调用关系又如何，这些谜底将在本文中一一揭晓。

本文2023-10-24首发于我的博客，最后修改于2023-10-24

Docker与Containerd的区别

your@email.com (你的名字) — Thu, 07 Sep 2023 09:38:51 +0000

Docker与Containerd的区别

作者：你的名字（your@email.com）

容器运行时

容器运行时（Container Runtime）是一种负责在操作系统层面创建和管理容器的软件工具或组件。它是容器化技术的核心组件之一，用于在容器内部运行应用程序，并提供隔离、资源管理和安全等功能。在Kubernetes中，容器运行时是负责管理和运行容器的组件。在过去，Docker是最常用的容器运行时，但随着时间的推移，containerd成为Kubernetes的另一个受欢迎的容器运行时选择。

本文2023-09-07首发于我的博客，最后修改于2023-09-07

使用 OpenTelemetry Collector 收集 Kubernetes 日志数据

your@email.com (你的名字) — Mon, 28 Aug 2023 10:52:55 +0000

使用 OpenTelemetry Collector 收集 Kubernetes 日志数据

作者：你的名字（your@email.com）

前面我们介绍了如何通过 OpenTelemetry Collector 来收集 Kubernetes 集群的指标数据，接下来我们再来了解下如何收集集群的日志记录数据。

安装 Loki

首先我们需要部署 Loki 来收集日志数据，同样我们这里使用 Helm Chart 来快速部署，不过需要注意同样不需要部署任何日志采集器，因为我们将使用 OpenTelemetry Collector 来收集日志数据，然后再将其发送到 Loki 中。

本文2023-08-28首发于我的博客，最后修改于2023-08-28

使用 OpenTelemetry Collector 采集 Kubernetes 指标数据

your@email.com (你的名字) — Mon, 28 Aug 2023 09:52:55 +0000

使用 OpenTelemetry Collector 采集 Kubernetes 指标数据

作者：你的名字（your@email.com）

Kubernetes 已成为一个被广泛采用的行业工具，对可观测性工具的需求也在不断增加。为此，OpenTelemetry 创建了许多不同的工具，来帮助 Kubernetes 用户观察他们的集群和服务。

本文2023-08-28首发于我的博客，最后修改于2023-08-28

Ubuntu apt源密钥不可用解决方案

your@email.com (你的名字) — Thu, 24 Aug 2023 08:03:51 +0000

Ubuntu apt源密钥不可用解决方案

作者：你的名字（your@email.com）

现象

报错如下

The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E

apt update报错

W: GPG error: https://repos.influxdata.com/ubuntu focal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY D8FF8E1F7DF8B07E
E: The repository 'https://repos.influxdata.com/ubuntu focal InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

解决方案

重新获取公钥

sudo gpg --keyserver keyserver.ubuntu.com --recv-keys D8FF8E1F7DF8B07E

导出公钥并加入apt信任密钥

sudo gpg --export --armor 467B942D3A79BD29 | sudo apt-key add -

apt update 不再报错

本文2023-08-24首发于我的博客，最后修改于2023-08-24

使用helm安装NFS-Client Provisioner实现动态存储

your@email.com (你的名字) — Wed, 23 Aug 2023 14:22:51 +0000

使用helm安装NFS-Client Provisioner实现动态存储

作者：你的名字（your@email.com）

本文介绍使用helm安装NFS-Client Provisioner实现动态存储，主要包括使用helm安装NFS-Client Provisioner实现动态存储使用实例、应用技巧、基本知识点总结和需要注意事项，具有一定的参考价值，需要的朋友可以参考一下。

本文2023-08-23首发于我的博客，最后修改于2023-08-23

Linux加固脚本

your@email.com (你的名字) — Wed, 16 Aug 2023 08:03:51 +0000

Linux加固脚本

作者：你的名字（your@email.com）

#!/bin/bash
# @Author: WeiyiGeek
# @Description: Ubuntu TLS Security Initiate
# @Create Time:  2019年9月1日 16:43:33
# @Last Modified time: 2021-11-15 11:06:31
# @E-mail: master@weiyigeek.top
# @Blog: https://www.weiyigeek.top
# @wechat: WeiyiGeeker
# @Github: https://github.com/WeiyiGeek/SecOpsDev/tree/master/OS-操作系统/Linux/
# @Version: 3.3
#-------------------------------------------------#
# 脚本主要功能说明:
# (1) Ubuntu 系统初始化操作包括IP地址设置、基础软件包更新以及安装加固。
# (2) Ubuntu 系统容器以及JDK相关环境安装。
# (3) Ubuntu 系统中异常错误日志解决。
# (4) Ubuntu 系统常规服务安装配置，加入数据备份目录。
# (5) Ubuntu 脚本错误优化、添加禁用cloud-init
#-------------------------------------------------#

## 系统全局变量定义
HOSTNAME=Ubuntu-Security-Template
IP=192.168.1.2
GATEWAY=192.168.1.1
DNSIP=("223.5.5.5" "223.6.6.6")
SSHPORT=20211
DefaultUser="WeiyiGeek"  # 系统创建的用户名称非root用户
ROOTPASS=WeiyiGeek       # 密码建议12位以上且包含数字、大小写字母以及特殊字符。
APPPASS=WeiyiGeek

## 名称: err 、info 、warning
## 用途：全局Log信息打印函数
## 参数: $@
log::err() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S')]: \033[31mERROR: $@ \033[0m\n"
}
log::info() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S')]: \033[32mINFO: $@ \033[0m\n"
}
log::warning() {
  printf "[$(date +'%Y-%m-%dT%H:%M:%S')]: \033[33mWARNING: $@ \033[0m\n"
}


## 名称: os::Network
## 用途: 网络配置相关操作脚本包括(IP地址修改)
## 参数: 无
os::Network () {
  log::info "[-] 操作系统网络配置相关脚本,开始执行....."

# (1) IP地址与主机名称设置
sudo cp /etc/netplan/00-installer-config.yaml{,.bak}
mkdir /opt/init/
sudo tee /opt/init/network.sh <<'EOF'
#!/bin/bash
CURRENT_IP=$(hostname -I | cut -f 1 -d " ")
GATEWAY=$(hostname -I | cut -f 1,2,3 -d ".")
if [[ $# -lt 3 ]];then
  echo "Usage: $0 IP Gateway Hostname"
  exit
fi
echo "IP:${1} # GATEWAY:${2} # HOSTNAME:${3}"
sudo sed -i "s#${CURRENT_IP}#${1}#g" /etc/netplan/00-installer-config.yaml
sudo sed -i "s#${GATEWAY}.1#${2}#g" /etc/netplan/00-installer-config.yaml
sudo hostnamectl set-hostname ${3} 
sudo netplan apply
EOF
sudo chmod +x /opt/init/network.sh

# (2) 本地主机名解析设置
sed -i "s/127.0.1.1\s.\w.*$/127.0.1.1 ${HOSTNAME}/g" /etc/hosts
grep -q "^\$(hostname -I)\s.\w.*$" /etc/hosts && sed -i "s/\$(hostname -I)\s.\w.*$/${IPADDR} ${HOSTNAME}" /etc/hosts || echo "${IPADDR} ${HOSTNAME}" >> /etc/hosts

# (3) 系统DNS域名解析服务设置
cp -a /etc/resolv.conf{,.bak}
for dns in ${DNSIP[@]};do echo "nameserver ${dns}" >> /etc/resolv.conf;done

sudo /opt/init/network.sh ${IP} ${GATEWAY} ${HOSTNAME}
log::info "[*] network configure modifiy successful! restarting Network........."
}


## 名称: os::Software
## 用途: 操作系统软件包管理及更新源配置
## 参数: 无
os::Software () {
  log::info "[-] 操作系统软件包管理及更新源配置相关脚本,开始执行....."

# (1) 卸载多余软件，例如 snap 软件及其服务
sudo systemctl stop snapd snapd.socket #停止snapd相关的进程服务
sudo apt autoremove --purge -y snapd
sudo systemctl daemon-reload
sudo rm -rf ~/snap /snap /var/snap /var/lib/snapd /var/cache/snapd /run/snapd

# (2) 软件源设置与系统更新
sudo cp /etc/apt/sources.list{,.bak}
sudo tee /etc/apt/sources.list <<'EOF'
#阿里云Mirrors - Ubuntu
deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
EOF

# (3) 内核版本升级以及常规软件安装
sudo apt autoclean && sudo apt update && sudo apt upgrade -y
sudo apt install -y nano vim git unzip wget ntpdate dos2unix net-tools tree htop ncdu nload sysstat psmisc bash-completion fail2ban  gcc g++ make jq nfs-common rpcbind libpam-cracklib

# (4) 代理方式进行更新
# sudo apt autoclean && sudo apt -o Acquire::http::proxy="http://proxy.weiyigeek.top/" update && sudo apt -o Acquire::http::proxy="http://proxy.weiyigeek.top" upgrade -y
# sudo apt install -o Acquire::http::proxy="http://proxy.weiyigeek.top/" -y nano vim git unzip wget ntpdate dos2unix net-tools tree htop ncdu nload sysstat psmisc bash-completion fail2ban
}


## 名称: os::TimedataZone
## 用途: 操作系统时间与时区同步配置
## 参数: 无
os::TimedataZone () {
  log::info "[*] 操作系统系统时间时区配置相关脚本,开始执行....."

# (1) 时间同步服务端容器(可选也可以用外部ntp服务器) : docker run -d --rm --cap-add SYS_TIME -e ALLOW_CIDR=0.0.0.0/0 -p 123:123/udp geoffh1977/chrony
echo "同步前的时间: $(date -R)"

# 方式1.Chrony 客户端配置
apt install -y chrony
grep -q "192.168.12.254" /etc/chrony/chrony.conf || sudo tee -a /etc/chrony/chrony.conf <<'EOF'
pool 192.168.4.254 iburst maxsources 1
pool 192.168.10.254 iburst maxsources 1
pool 192.168.12.254 iburst maxsources 1
pool ntp.aliyun.com iburst maxsources 4
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
# 允许跳跃式校时 如果在前 3 次校时中时间差大于 1.0s
makestep 1 3
EOF
systemctl enable chrony && systemctl restart chrony && systemctl status chrony -l

# 方式2
# sudo ntpdate 192.168.10.254 || sudo ntpdate 192.168.12.215 || sudo ntpdate ntp1.aliyun.com

# 方式3
# echo 'NTP=192.168.10.254 192.168.4.254' >> /etc/systemd/timesyncd.conf
# echo 'FallbackNTP=ntp.aliyun.com' >> /etc/systemd/timesyncd.conf
# systemctl restart systemd-timesyncd.service

# (2) 时区与地区设置: 
sudo cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
sudo timedatectl set-timezone Asia/Shanghai
# sudo dpkg-reconfigure tzdata  # 修改确认
# sudo bash -c "echo 'Asia/Shanghai' > /etc/timezone" # 与上一条命令一样
# 将当前的 UTC 时间写入硬件时钟 (硬件时间默认为UTC)
sudo timedatectl set-local-rtc 0
# 启用NTP时间同步：
sudo timedatectl set-ntp yes
# 校准时间服务器-时间同步(推荐使用chronyc进行平滑同步)
sudo chronyc tracking
# 手动校准-强制更新时间
# chronyc -a makestep
# 系统时钟同步硬件时钟
# sudo hwclock --systohc
sudo hwclock -w

# (3) 重启依赖于系统时间的服务
sudo systemctl restart rsyslog.service cron.service
log::info "[*] Tie confmigure modifiy successful! restarting chronyd rsyslog.service crond.service........."
timedatectl
}


## 名称: os::Security
## 用途: 操作系统安全加固配置脚本(符合等保要求-三级要求)
## 参数: 无
os::Security () {
  log::info "正在进行->操作系统安全加固(符合等保要求-三级要求)配置"

# (0) 系统用户核查配置
  log::info "[-] 锁定或者删除多余的系统账户以及创建低权限用户"
userdel -r lxd
groupdel lxd
defaultuser=(root daemon bin sys games man lp mail news uucp proxy www-data backup list irc gnats nobody systemd-network systemd-resolve systemd-timesync messagebus syslog _apt tss uuidd tcpdump landscape pollinate usbmux sshd systemd-coredump _chrony)
for i in $(cat /etc/passwd | cut -d ":" -f 1,7);do
  flag=0; name=${i%%:*}; terminal=${i##*:}
  if [[ "${terminal}" == "/bin/bash" || "${terminal}" == "/bin/sh" ]];then
    log::warning "${i} 用户，shell终端为 /bin/bash 或者 /bin/sh"
  fi
  for j in ${defaultuser[@]};do
    if [[ "${name}" == "${j}" ]];then
      flag=1
      break;
    fi
  done
  if [[ $flag -eq 0 ]];then
    log::warning "${i} 非默认用户"
  fi
done
cp /etc/shadow /etc/shadow-`date +%Y%m%d`.bak
passwd -l adm&>/dev/null 2&>/dev/null; passwd -l daemon&>/dev/null 2&>/dev/null; passwd -l bin&>/dev/null 2&>/dev/null; passwd -l sys&>/dev/null 2&>/dev/null; passwd -l lp&>/dev/null 2&>/dev/null; passwd -l uucp&>/dev/null 2&>/dev/null; passwd -l nuucp&>/dev/null 2&>/dev/null; passwd -l smmsplp&>/dev/null 2&>/dev/null; passwd -l mail&>/dev/null 2&>/dev/null; passwd -l operator&>/dev/null 2&>/dev/null; passwd -l games&>/dev/null 2&>/dev/null; passwd -l gopher&>/dev/null 2&>/dev/null; passwd -l ftp&>/dev/null 2&>/dev/null; passwd -l nobody&>/dev/null 2&>/dev/null; passwd -l nobody4&>/dev/null 2&>/dev/null; passwd -l noaccess&>/dev/null 2&>/dev/null; passwd -l listen&>/dev/null 2&>/dev/null; passwd -l webservd&>/dev/null 2&>/dev/null; passwd -l rpm&>/dev/null 2&>/dev/null; passwd -l dbus&>/dev/null 2&>/dev/null; passwd -l avahi&>/dev/null 2&>/dev/null; passwd -l mailnull&>/dev/null 2&>/dev/null; passwd -l nscd&>/dev/null 2&>/dev/null; passwd -l vcsa&>/dev/null 2&>/dev/null; passwd -l rpc&>/dev/null 2&>/dev/null; passwd -l rpcuser&>/dev/null 2&>/dev/null; passwd -l nfs&>/dev/null 2&>/dev/null; passwd -l sshd&>/dev/null 2&>/dev/null; passwd -l pcap&>/dev/null 2&>/dev/null; passwd -l ntp&>/dev/null 2&>/dev/null; passwd -l haldaemon&>/dev/null 2&>/dev/null; passwd -l distcache&>/dev/null 2&>/dev/null; passwd -l webalizer&>/dev/null 2&>/dev/null; passwd -l squid&>/dev/null 2&>/dev/null; passwd -l xfs&>/dev/null 2&>/dev/null; passwd -l gdm&>/dev/null 2&>/dev/null; passwd -l sabayon&>/dev/null 2&>/dev/null; passwd -l named&>/dev/null 2&>/dev/null

# (2) 用户密码设置和口令策略设置
  log::info "[-]  配置满足策略的root管理员密码 "
echo  ${ROOTPASS} | passwd --stdin root

log::info "[-] 配置满足策略的app普通用户密码(根据需求配置)"
groupadd application
useradd -m -s /bin/bash -c "application primary user" -g application app 
echo ${APPPASS} | passwd --stdin app

  log::info "[-] 强制用户在下次登录时更改密码 "
chage -d 0 -m 0 -M 90 -W 15 root && passwd --expire root 
chage -d 0 -m 0 -M 90 -W 15 ${DefaultUser} && passwd --expire ${DefaultUser} 
chage -d 0 -m 0 -M 90 -W 15 app && passwd --expire app

  log::info "[-] 用户口令复杂性策略设置 (密码过期周期0~90、到期前15天提示、密码长度至少15、复杂度设置至少有一个大小写、数字、特殊字符、密码三次不能一样、尝试次数为三次)"
egrep -q "^\s*PASS_MIN_DAYS\s+\S*(\s*#.*)?\s*$" /etc/login.defs && sed -ri "s/^(\s*)PASS_MIN_DAYS\s+\S*(\s*#.*)?\s*$/\PASS_MIN_DAYS  0/" /etc/login.defs || echo "PASS_MIN_DAYS  0" >> /etc/login.defs
egrep -q "^\s*PASS_MAX_DAYS\s+\S*(\s*#.*)?\s*$" /etc/login.defs && sed -ri "s/^(\s*)PASS_MAX_DAYS\s+\S*(\s*#.*)?\s*$/\PASS_MAX_DAYS  90/" /etc/login.defs || echo "PASS_MAX_DAYS  90" >> /etc/login.defs
egrep -q "^\s*PASS_WARN_AGE\s+\S*(\s*#.*)?\s*$" /etc/login.defs && sed -ri "s/^(\s*)PASS_WARN_AGE\s+\S*(\s*#.*)?\s*$/\PASS_WARN_AGE  15/" /etc/login.defs || echo "PASS_WARN_AGE  15" >> /etc/login.defs
egrep -q "^\s*PASS_MIN_LEN\s+\S*(\s*#.*)?\s*$" /etc/login.defs && sed -ri "s/^(\s*)PASS_MIN_LEN\s+\S*(\s*#.*)?\s*$/\PASS_MIN_LEN  15/" /etc/login.defs || echo "PASS_MIN_LEN  15" >> /etc/login.defs

egrep -q "^password\s.+pam_cracklib.so\s+\w+.*$" /etc/pam.d/common-password && sed -ri '/^password\s.+pam_cracklib.so/{s/pam_cracklib.so\s+\w+.*$/pam_cracklib.so retry=3 minlen=15 ucredit=-1 lcredit=-1 dcredit=-1 ocredit=-1 difok=1/g;}' /etc/pam.d/common-password
egrep -q "^password\s.+pam_unix.so\s+\w+.*$" /etc/pam.d/common-password && sed -ri '/^password\s.+pam_unix.so/{s/pam_unix.so\s+\w+.*$/pam_unix.so obscure use_authtok try_first_pass sha512 remember=3/g;}' /etc/pam.d/common-password

  log::info "[-] 存储用户密码的文件，其内容经过sha512加密，所以非常注意其权限"
touch /etc/security/opasswd && chown root:root /etc/security/opasswd && chmod 600 /etc/security/opasswd 


# (3) 用户sudo权限以及重要目录和文件的权限设置
  log::info "[-] 用户sudo权限以及重要目录和文件的新建默认权限设置"
# 如uBuntu安装时您创建的用户 WeiyiGeek 防止直接通过 sudo passwd 修改root密码(此时必须要求输入WeiyiGeek密码后才可修改root密码)
# Tips: Sudo允许授权用户权限以另一个用户（通常是root用户）的身份运行程序, 
# DefaultUser="weiyigeek"
sed -i "/# Members of the admin/i ${DefaultUser} ALL=(ALL) PASSWD:ALL" /etc/sudoers


  log::info "[-] 配置用户 umask 为022 "
egrep -q "^\s*umask\s+\w+.*$" /etc/profile && sed -ri "s/^\s*umask\s+\w+.*$/umask 022/" /etc/profile || echo "umask 022" >> /etc/profile
egrep -q "^\s*umask\s+\w+.*$" /etc/bash.bashrc && sed -ri "s/^\s*umask\s+\w+.*$/umask 022/" /etc/bashrc || echo "umask 022" >> /etc/bash.bashrc
# log::info "[-] 设置用户目录创建默认权限, (初始为077比较严格)，在设置 umask 为022 及 777 - 022 "
# egrep -q "^\s*(umask|UMASK)\s+\w+.*$" /etc/login.defs && sed -ri "s/^\s*(umask|UMASK)\s+\w+.*$/UMASK 022/" /etc/login.defs || echo "UMASK 022" >> /etc/login.defs

  log::info "[-] 设置或恢复重要目录和文件的权限"
chmod 755 /etc; 
chmod 777 /tmp; 
chmod 700 /etc/inetd.conf&>/dev/null 2&>/dev/null; 
chmod 755 /etc/passwd; 
chmod 755 /etc/shadow; 
chmod 644 /etc/group; 
chmod 755 /etc/security; 
chmod 644 /etc/services; 
chmod 750 /etc/rc*.d
chmod 600 ~/.ssh/authorized_keys

  log::info "[-] 删除潜在威胁文件 "
find / -maxdepth 3 -name hosts.equiv | xargs rm -rf
find / -maxdepth 3 -name .netrc | xargs rm -rf
find / -maxdepth 3 -name .rhosts | xargs rm -rf


# (4) SSHD 服务安全加固设置
log::info "[-] sshd 服务安全加固设置"
# 严格模式
sudo egrep -q "^\s*StrictModes\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*StrictModes\s+.+$/StrictModes yes/" /etc/ssh/sshd_config || echo "StrictModes yes" >> /etc/ssh/sshd_config
# 监听端口更改
if [ -e ${SSHPORT} ];then export SSHPORT=20211;fi
sudo egrep -q "^\s*Port\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*Port\s+.+$/Port ${SSHPORT}/" /etc/ssh/sshd_config || echo "Port ${SSHPORT}" >> /etc/ssh/sshd_config
# 禁用X11转发以及端口转发
sudo egrep -q "^\s*X11Forwarding\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*X11Forwarding\s+.+$/X11Forwarding no/" /etc/ssh/sshd_config || echo "X11Forwarding no" >> /etc/ssh/sshd_config
sudo egrep -q "^\s*X11UseLocalhost\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*X11UseLocalhost\s+.+$/X11UseLocalhost yes/" /etc/ssh/sshd_config || echo "X11UseLocalhost yes" >> /etc/ssh/sshd_config
sudo egrep -q "^\s*AllowTcpForwarding\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*AllowTcpForwarding\s+.+$/AllowTcpForwarding no/" /etc/ssh/sshd_config || echo "AllowTcpForwarding no" >> /etc/ssh/sshd_config
sudo egrep -q "^\s*AllowAgentForwarding\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*AllowAgentForwarding\s+.+$/AllowAgentForwarding no/" /etc/ssh/sshd_config || echo "AllowAgentForwarding no" >> /etc/ssh/sshd_config
# 关闭禁用用户的 .rhosts 文件  ~/.ssh/.rhosts 来做为认证: 缺省IgnoreRhosts yes 
egrep -q "^(#)?\s*IgnoreRhosts\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^(#)?\s*IgnoreRhosts\s+.+$/IgnoreRhosts yes/" /etc/ssh/sshd_config || echo "IgnoreRhosts yes" >> /etc/ssh/sshd_config
# 禁止root远程登录（推荐配置-根据需求配置）
egrep -q "^\s*PermitRootLogin\s+.+$" /etc/ssh/sshd_config && sed -ri "s/^\s*PermitRootLogin\s+.+$/PermitRootLogin no/" /etc/ssh/sshd_config || echo "PermitRootLogin no" >> /etc/ssh/sshd_config
# 登陆前后欢迎提示设置
egrep -q "^\s*(banner|Banner)\s+\W+.*$" /etc/ssh/sshd_config && sed -ri "s/^\s*(banner|Banner)\s+\W+.*$/Banner \/etc\/issue/" /etc/ssh/sshd_config || \
echo "Banner /etc/issue" >> /etc/ssh/sshd_config
log::info "[-] 远程SSH登录前后提示警告Banner设置"
# SSH登录前警告Banner
sudo tee /etc/issue <<'EOF'
****************** [ 安全登陆 (Security Login) ] *****************
Authorized only. All activity will be monitored and reported.By Security Center.
Owner: WeiyiGeek, Site: https://www.weiyigeek.top
EOF
# SSH登录后提示Banner
sed -i '/^fi/a\\n\necho "\\e[1;37;41;5m################## 安全运维 (Security Operation) ####################\\e[0m"\necho "\\e[32mLogin success. Please execute the commands and operation data carefully.By WeiyiGeek.\\e[0m"' /etc/update-motd.d/00-header


# (5) 用户远程登录失败次数与终端超时设置 
  log::info "[-] 用户远程连续登录失败10次锁定帐号5分钟包括root账号"
sed -ri "/^\s*auth\s+required\s+pam_tally2.so\s+.+(\s*#.*)?\s*$/d" /etc/pam.d/sshd 
sed -ri '2a auth required pam_tally2.so deny=10 unlock_time=300 even_deny_root root_unlock_time=300' /etc/pam.d/sshd 
# 宿主机控制台登陆(可选)
# sed -ri "/^\s*auth\s+required\s+pam_tally2.so\s+.+(\s*#.*)?\s*$/d" /etc/pam.d/login
# sed -ri '2a auth required pam_tally2.so deny=5 unlock_time=300 even_deny_root root_unlock_time=300' /etc/pam.d/login

  log::info "[-] 设置登录超时时间为10分钟 "
egrep -q "^\s*(export|)\s*TMOUT\S\w+.*$" /etc/profile && sed -ri "s/^\s*(export|)\s*TMOUT.\S\w+.*$/export TMOUT=600\nreadonly TMOUT/" /etc/profile || echo -e "export TMOUT=600\nreadonly TMOUT" >> /etc/profile
egrep -q "^\s*.*ClientAliveInterval\s\w+.*$" /etc/ssh/sshd_config && sed -ri "s/^\s*.*ClientAliveInterval\s\w+.*$/ClientAliveInterval 600/" /etc/ssh/sshd_config || echo "ClientAliveInterval 600" >> /etc/ssh/sshd_config


# (5) 切换用户日志记录或者切换命令更改(可选)
  log::info "[-] 切换用户日志记录和切换命令更改名称为SU "
egrep -q "^(\s*)SULOG_FILE\s+\S*(\s*#.*)?\s*$" /etc/login.defs && sed -ri "s/^(\s*)SULOG_FILE\s+\S*(\s*#.*)?\s*$/\SULOG_FILE  \/var\/log\/.history\/sulog/" /etc/login.defs || echo "SULOG_FILE  /var/log/.history/sulog" >> /etc/login.defs
egrep -q "^\s*SU_NAME\s+\S*(\s*#.*)?\s*$" /etc/login.defs && sed -ri "s/^(\s*)SU_NAME\s+\S*(\s*#.*)?\s*$/\SU_NAME  SU/" /etc/login.defs || echo "SU_NAME  SU" >> /etc/login.defs
mkdir -vp /usr/local/bin /var/log/.backups /var/log/.history /var/log/.history/sulog
cp /usr/bin/su /var/log/.backups/su.bak
mv /usr/bin/su /usr/bin/SU
# 只能写入不能删除其目标目录中的文件
chmod -R 1777 /var/log/.history
chattr -R +a /var/log/.history 
chattr +a /var/log/.backups

# (6) 用户终端执行的历史命令记录
log::info "[-] 用户终端执行的历史命令记录 "
egrep -q "^HISTSIZE\W\w+.*$" /etc/profile && sed -ri "s/^HISTSIZE\W\w+.*$/HISTSIZE=101/" /etc/profile || echo "HISTSIZE=101" >> /etc/profile
# 方式1
sudo tee /etc/profile.d/history-record.sh <<'EOF'
# 历史命令执行记录文件路径
LOGTIME=$(date +%Y%m%d-%H-%M-%S)
export HISTFILE="/var/log/.history/${USER}.${LOGTIME}.history"
if [ ! -f ${HISTFILE} ];then
  touch ${HISTFILE}
fi
chmod 600 /var/log/.history/${USER}.${LOGTIME}.history
# 历史命令执行文件大小记录设置
HISTFILESIZE=128
HISTTIMEFORMAT="%F_%T $(whoami)#$(who -u am i 2>/dev/null| awk '{print $NF}'|sed -e 's/[()]//g'):"
EOF
# 方式2.未能成功(如后续有小伙伴成功了欢迎留言分享)
# sudo tee /usr/local/bin/history.sh <<'EOF'
# #!/bin/bash
# logfiletime=$(date +%Y%m%d-%H-%M-%S)
# # unalias "history"
# if [ $# -eq 0 ];then history;fi
# for i in $*;do
#   if [ "$i" = "-c" ];then 
#     mv ~/.bash_history > /var/log/.history/${logfiletime}.history
#     history -c
#     continue;
#   fi
# done
# alias history="source /usr/local/bin/history.sh"
# EOF


# (7) GRUB 安全设置 （需要手动设置请按照需求设置）
  log::info "[-] 系统 GRUB 安全设置(防止物理接触从grub菜单中修改密码) "
# Grub 关键文件备份
cp -a /etc/grub.d/00_header /var/log/.backups 
cp -a /etc/grub.d/10_linux /var/log/.backups 
# 设置Grub菜单界面显示时间
sed -i -e 's|GRUB_TIMEOUT_STYLE=hidden|#GRUB_TIMEOUT_STYLE=hidden|g' -e 's|GRUB_TIMEOUT=0|GRUB_TIMEOUT=3|g' /etc/default/grub
sed -i -e 's|set timeout_style=${style}|#set timeout_style=${style}|g' -e 's|set timeout=${timeout}|set timeout=3|g' /etc/grub.d/00_header
# 创建认证密码 (此处密码: WeiyiGeek)
sudo grub-mkpasswd-pbkdf2
# Enter password:
# Reenter password:
# PBKDF2 hash of your password is grub.pbkdf2.sha512.10000.21AC9CEF61B96972BF6F918D2037EFBEB8280001045ED32DFDDCC260591CC6BC8957CF25A6755904A7053E97940A9E4CD5C1EF833C1651C1BCF09D899BED4C7C.9691521F5BB34CD8AEFCED85F4B830A86EC93B61A31885BCBE3FEE927D54EFDEE69FA8B51DBC00FCBDB618D4082BC22B2B6BA4161C7E6B990C4E5CFC9E9748D7
# 设置认证用户以及password_pbkdf2认证
tee -a /etc/grub.d/00_header <<'END'
cat <<'EOF'
# GRUB Authentication
set superusers="grub"
password_pbkdf2 grub grub.pbkdf2.sha512.10000.21AC9CEF61B96972BF6F918D2037EFBEB8280001045ED32DFDDCC260591CC6BC8957CF25A6755904A7053E97940A9E4CD5C1EF833C1651C1BCF09D899BED4C7C.9691521F5BB34CD8AEFCED85F4B830A86EC93B61A31885BCBE3FEE927D54EFDEE69FA8B51DBC00FCBDB618D4082BC22B2B6BA4161C7E6B990C4E5CFC9E9748D7
EOF
END
# 设置进入正式系统不需要认证如进入单用户模式进行重置账号密码时需要进行认证。 （高敏感数据库系统不建议下述操作）
# 在191和193 分别加入--user=grub 和 --unrestricted
# 191       echo "menuentry --user=grub '$(echo "$title" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-$version-$type-$boot_device_id' {" | sed "s/^/$submenu_indentation/"  # 如果按e进行menu菜单则需要用grub进行认证
# 192   else
# 193       echo "menuentry --unrestricted '$(echo "$os" | grub_quote)' ${CLASS} \$menuentry_id_option 'gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/"          # 正常进入系统则不认证

sed -i '/echo "$title" | grub_quote/ { s/menuentry /menuentry --user=grub /;}' /etc/grub.d/10_linux
sed -i '/echo "$os" | grub_quote/ { s/menuentry /menuentry --unrestricted /;}' /etc/grub.d/10_linux

# Ubuntu 方式更新GRUB从而生成boot启动文件。
update-grub


# (8) 操作系统防火墙启用以及策略设置
  log::info "[-] 系统防火墙启用以及规则设置 "
systemctl enable ufw.service && systemctl start ufw.service && ufw enable
sudo ufw allow proto tcp to any port 20211
# 重启修改配置相关服务
systemctl restart sshd
}



## 名称: os::Operation 
## 用途: 操作系统安全运维设置
## 参数: 无
os::Operation () {
  log::info "[-] 操作系统安全运维设置相关脚本"

# (0) 禁用ctrl+alt+del组合键对系统重启 (必须要配置,我曾入过坑)
  log::info "[-] 禁用控制台ctrl+alt+del组合键重启"
mv /usr/lib/systemd/system/ctrl-alt-del.target /var/log/.backups/ctrl-alt-del.target-$(date +%Y%m%d).bak

# (1) 设置文件删除回收站别名
  log::info "[-] 设置文件删除回收站别名(防止误删文件) "
sudo tee /etc/profile.d/alias.sh <<'EOF'
# User specific aliases and functions
# 删除回收站
# find ~/.trash -delete
# 删除空目录
# find ~/.trash -type d -delete
alias rm="sh /usr/local/bin/remove.sh"
EOF
sudo tee /usr/local/bin/remove.sh <<'EOF'
#!/bin/sh
# 定义回收站文件夹目录.trash
trash="/.trash"
deltime=$(date +%Y%m%d-%H-%M-%S)
TRASH_DIR="${HOME}${trash}/${deltime}"
# 建立回收站目录当不存在的时候
if [ ! -e ${TRASH_DIR} ];then
   mkdir -p ${TRASH_DIR}
fi
for i in $*;do
  if [ "$i" = "-rf" ];then continue;fi
  # 防止误操作
  if [ "$i" = "/" ];then echo '# Danger delete command, Not delete / directory!';exit -1;fi
  #定义秒时间戳
  STAMP=$(date +%s)
  #得到文件名称(非文件夹)，参考man basename
  fileName=$(basename $i)
  #将输入的参数，对应文件mv至.trash目录，文件后缀，为当前的时间戳
  mv $i ${TRASH_DIR}/${fileName}.${STAMP}
done
EOF
sudo chmod +775 /usr/local/bin/remove.sh /etc/profile.d/alias.sh /etc/profile.d/history-record.sh
sudo chmod a+x /usr/local/bin/remove.sh /etc/profile.d/alias.sh /etc/profile.d/history-record.sh
source /etc/profile.d/alias.sh  /etc/profile.d/history-record.sh


# (2) 解决普通定时任务无法后台定时执行
log::info "[-] 解决普通定时任务无法后台定时执行 "
linenumber=`expr $(egrep -n "pam_unix.so\s$" /etc/pam.d/common-session-noninteractive | cut -f 1 -d ":") - 2`
sudo sed -ri "${linenumber}a session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid" /etc/pam.d/common-session-noninteractive


# (3) 解决 ubuntu20.04 multipath add missing path 错误
# 添加以下内容,sda视本地环境做调整
tee -a /etc/multipath.conf <<'EOF'
blacklist {
  devnode "^sda"
}
EOF
# 重启multipath-tools服务
sudo service multipath-tools restart

# (4) 禁用 Ubuntu 中的 cloud-init
# 在 /etc/cloud 目录下创建 cloud-init.disabled 文件,注意重启后生效
sudo touch /etc/cloud/cloud-init.disabled
}



## 名称: os::optimizationn
## 用途: 操作系统优化设置(内核参数)
## 参数: 无
os::optimizationn () {
log::info "[-] 正在进行操作系统内核参数优化设置......."

# (1) 系统内核参数的配置(/etc/sysctl.conf)
log::info "[-] 系统内核参数的配置/etc/sysctl.conf"

# /etc/sysctl.d/99-kubernetes-cri.conf
egrep -q "^(#)?net.ipv4.ip_forward.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.ipv4.ip_forward.*|net.ipv4.ip_forward = 1|g"  /etc/sysctl.conf || echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
# egrep -q "^(#)?net.bridge.bridge-nf-call-ip6tables.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.bridge.bridge-nf-call-ip6tables.*|net.bridge.bridge-nf-call-ip6tables = 1|g" /etc/sysctl.conf || echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf 
# egrep -q "^(#)?net.bridge.bridge-nf-call-iptables.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.bridge.bridge-nf-call-iptables.*|net.bridge.bridge-nf-call-iptables = 1|g" /etc/sysctl.conf || echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
egrep -q "^(#)?net.ipv6.conf.all.disable_ipv6.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.ipv6.conf.all.disable_ipv6.*|net.ipv6.conf.all.disable_ipv6 = 1|g" /etc/sysctl.conf || echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
egrep -q "^(#)?net.ipv6.conf.default.disable_ipv6.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.ipv6.conf.default.disable_ipv6.*|net.ipv6.conf.default.disable_ipv6 = 1|g" /etc/sysctl.conf || echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
egrep -q "^(#)?net.ipv6.conf.lo.disable_ipv6.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.ipv6.conf.lo.disable_ipv6.*|net.ipv6.conf.lo.disable_ipv6 = 1|g" /etc/sysctl.conf || echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
egrep -q "^(#)?net.ipv6.conf.all.forwarding.*" /etc/sysctl.conf && sed -ri "s|^(#)?net.ipv6.conf.all.forwarding.*|net.ipv6.conf.all.forwarding = 1|g" /etc/sysctl.conf || echo "net.ipv6.conf.all.forwarding = 1"  >> /etc/sysctl.conf
egrep -q "^(#)?vm.max_map_count.*" /etc/sysctl.conf && sed -ri "s|^(#)?vm.max_map_count.*|vm.max_map_count = 262144|g" /etc/sysctl.conf || echo "vm.max_map_count = 262144"  >> /etc/sysctl.conf

tee -a /etc/sysctl.conf <<'EOF'
# 调整提升服务器负载能力之外,还能够防御小流量的Dos、CC和SYN攻击
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
# net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_fastopen = 3
# 优化TCP的可使用端口范围及提升服务器并发能力(注意一般流量小的服务器上没必要设置如下参数)
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 16384
net.ipv4.ip_local_port_range = 1024 65535
# 优化核套接字TCP的缓存区
net.core.netdev_max_backlog = 8192
net.core.somaxconn = 8192
net.core.rmem_max = 12582912
net.core.rmem_default = 6291456
net.core.wmem_max = 12582912
net.core.wmem_default = 6291456
EOF


# (2) Linux 系统的最大进程数和最大文件打开数限制
log::info "[-] Linux 系统的最大进程数和最大文件打开数限制"
egrep -q "^\s*ulimit -HSn\s+\w+.*$" /etc/profile && sed -ri "s/^\s*ulimit -HSn\s+\w+.*$/ulimit -HSn 65535/" /etc/profile || echo "ulimit -HSn 65535" >> /etc/profile
egrep -q "^\s*ulimit -HSu\s+\w+.*$" /etc/profile && sed -ri "s/^\s*ulimit -HSu\s+\w+.*$/ulimit -HSu 65535/" /etc/profile || echo "ulimit -HSu 65535" >> /etc/profile

tee -a /etc/security/limits.conf <<'EOF'
# ulimit -HSn 65535
# ulimit -HSu 65535
*  soft  nofile  65535
*  hard  nofile  65535
*  soft  nproc   65535
*  hard  nproc   65535
# End of file
EOF
# sed -i "/# End/i *  soft  nproc   65535" /etc/security/limits.conf
# sed -i "/# End/i *  hard  nproc   65535" /etc/security/limits.conf
sysctl -p

# 需重启生效
reboot
}

## 名称: system::swap
## 用途: Liunx 系统创建SWAP交换分区(默认2G)
## 参数: $1(几G)
system::swap () {
  if [ -e $1 ];then
    sudo dd if=/dev/zero of=/swapfile bs=1024 count=2097152   # 2G Swap 分区 1024 * 1024 , centos 以 1000 为标准
  else
    number=$(echo "${1}*1024*1024"|bc)
    sudo dd if=/dev/zero of=/swapfile bs=1024 count=${number}   # 2G Swap 分区 1024 * 1024 , centos 以 1000 为标准
  fi

  sudo mkswap /swapfile && sudo swapon /swapfile
  if [ $(grep -c "/swapfile" /etc/fstab) -eq 0 ];then
sudo tee -a /etc/fstab <<'EOF'
/swapfile swap swap default 0 0
EOF
fi
sudo swapon --show && sudo free -h
}




## 名称: software::Java
## 用途: java 环境安装配置
## 参数: 无
software::Java () {
  # 基础变量
  JAVA_FILE="/root/Downloads/jdk-8u211-linux-x64.tar.gz"
  JAVA_SRC="/usr/local/"
  JAVA_DIR="/usr/local/jdk"
  # 环境配置
  sudo tar -zxvf ${JAVA_FILE} -C ${JAVA_SRC}
  sudo rm -rf /usr/local/jdk 
  JAVA_SRC=$(ls /usr/local/ | grep "jdk")
  sudo ln -s ${JAVA_SRC} ${JAVA_DIR}
  export PATH=${JAVA_DIR}/bin:${PATH}
  sudo cp /etc/profile /etc/profile.$(date +%Y%m%d-%H%M%S).bak
  sudo tee -a /etc/profile <<'EOF'
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=/usr/local/jdk/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH
EOF
  java -version
}


## 名称: software::docker
## 用途: 软件安装之Docker安装
## 参数: 无
# 帮助: https://docs.docker.com/engine/install/ubuntu/
# Ubuntu Focal 20.04 (LTS)
# Ubuntu Bionic 18.04 (LTS)
# Ubuntu Xenial 16.04 (LTS)
function InstallDocker(){
  # 1.卸载旧版本 
  sudo apt-get remove docker docker-engine docker.io containerd runc
  
  # 2.更新apt包索引并安装包以允许apt在HTTPS上使用存储库
  sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

  # 3.添加Docker官方GPG密钥 # -fsSL
  sudo curl  https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

  # 4.通过搜索指纹的最后8个字符进行密钥验证
  sudo apt-key fingerprint 0EBFCD88

  # 5.设置稳定存储库
  sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

  # 6.Install Docker Engine 默认最新版本
  sudo apt-get update && sudo apt-get install -y docker-ce=5:20.10.7~3-0~ubuntu-focal docker-ce-cli=5:20.10.7~3-0~ubuntu-focal containerd.io docker-compose
  # - 强制IPv4
  # sudo apt-get -o Acquire::ForceIPv4=true  install -y docker-ce=5:19.03.15~3-0~ubuntu-focal docker-ce-cli=5:19.03.15~3-0~ubuntu-focal containerd.io docker-compose

  # 7.安装特定版本的Docker引擎，请在repo中列出可用的版本
  apt-cache madison docker-ce
  # docker-ce | 5:20.10.6~3-0~ubuntu-focal| https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
  # docker-ce | 5:19.03.15~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu  xenial/stable amd64 Packages
  # 使用第二列中的版本字符串安装特定的版本，例如:5:18.09.1~3-0~ubuntu-xenial。
  # $sudo apt-get install docker-ce= docker-ce-cli= containerd.io

  #8.将当前用户加入docker用户组然后重新登陆当前用户使得低权限用户
  sudo gpasswd -a ${USER} docker
  # sudo gpasswd -a weiyigeek docker

  #9.加速器建立
  mkdir -vp /etc/docker/
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://xlx9erfu.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"],
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10"
  },
  "live-restore": true,
  "dns": ["192.168.12.254","223.6.6.6"],
  "insecure-registries": [ "harbor.weiyigeek.top","harbor.cloud"]
}
EOF
  # "data-root":"/monitor/container",
  # "insecure-registries": ["harbor.weiyigeek.top"]
  # 9.自启与启动
  sudo systemctl daemon-reload
  sudo systemctl enable docker 
  sudo systemctl restart docker

  # 10.退出登陆生效
  exit
}


## 名称: disk::Lvsmanager
## 用途: uBuntu操作系统磁盘 LVS 逻辑卷添加与配置(扩容流程)
## 参数: 无
disk::Lvsmanager () {
  echo "\n分区信息:"
  sudo df -Th
  sudo lsblk
  echo -e "\n 磁盘信息："
  sudo fdisk -l
  echo -e "\n PV物理卷查看："
  sudo pvscan
  echo -e "\n vgs虚拟卷查看："
  sudo vgs
  echo -e "\n lvscan逻辑卷扫描:"
  sudo lvscan
  echo -e "\n 分区扩展"
  echo "Ubuntu \n lvextend -L +74G /dev/ubuntu-vg/ubuntu-lv"
  echo "lsblk"
  echo -e "ubuntu general \n # resize2fs -p -F /dev/mapper/ubuntu--vg-ubuntu--lv"
}

# 安全加固过程临时文件清理为基线镜像做准备
unalias rm
find ~/.trash/* -delete
find /home/ -type d -name .trash -exec find {} -delete \;
find /var/log -name "*.gz" -delete
find /var/log -name "*log.*" -delete
find /var/log -name "vmware-*.*.log" -delete
find /var/log -name "*.log-*" -delete
find /var/log -name "*.log" -exec truncate -s 0 {} \;
find /tmp/* -delete

本文2023-08-16首发于我的博客，最后修改于2023-08-16

在kubernetes中部署rookCeph

your@email.com (你的名字) — Tue, 15 Aug 2023 08:54:38 +0000

在kubernetes中部署rookCeph

作者：你的名字（your@email.com）

安装CRD资源

##############################################################################
# 在创建 Rook 集群之前创建所需的 CRD。
# 这些资源*必须*在 cluster.yaml 或其变体之前创建。
##############################################################################
---
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephblockpoolradosnamespaces.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephBlockPoolRadosNamespace
    listKind: CephBlockPoolRadosNamespaceList
    plural: cephblockpoolradosnamespaces
    singular: cephblockpoolradosnamespace
  scope: Namespaced
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                blockPoolName:
                  type: string
              required:
                - blockPoolName
              type: object
            status:
              properties:
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephblockpools.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephBlockPool
    listKind: CephBlockPoolList
    plural: cephblockpools
    singular: cephblockpool
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                compressionMode:
                  enum:
                    - none
                    - passive
                    - aggressive
                    - force
                    - ""
                  nullable: true
                  type: string
                crushRoot:
                  nullable: true
                  type: string
                deviceClass:
                  nullable: true
                  type: string
                enableRBDStats:
                  type: boolean
                erasureCoded:
                  properties:
                    algorithm:
                      type: string
                    codingChunks:
                      minimum: 0
                      type: integer
                    dataChunks:
                      minimum: 0
                      type: integer
                  required:
                    - codingChunks
                    - dataChunks
                  type: object
                failureDomain:
                  type: string
                mirroring:
                  properties:
                    enabled:
                      type: boolean
                    mode:
                      type: string
                    peers:
                      nullable: true
                      properties:
                        secretNames:
                          items:
                            type: string
                          type: array
                      type: object
                    snapshotSchedules:
                      items:
                        properties:
                          interval:
                            type: string
                          path:
                            type: string
                          startTime:
                            type: string
                        type: object
                      type: array
                  type: object
                name:
                  enum:
                    - device_health_metrics
                    - .nfs
                    - .mgr
                  type: string
                parameters:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                quotas:
                  nullable: true
                  properties:
                    maxBytes:
                      format: int64
                      type: integer
                    maxObjects:
                      format: int64
                      type: integer
                    maxSize:
                      pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                      type: string
                  type: object
                replicated:
                  properties:
                    hybridStorage:
                      nullable: true
                      properties:
                        primaryDeviceClass:
                          minLength: 1
                          type: string
                        secondaryDeviceClass:
                          minLength: 1
                          type: string
                      required:
                        - primaryDeviceClass
                        - secondaryDeviceClass
                      type: object
                    replicasPerFailureDomain:
                      minimum: 1
                      type: integer
                    requireSafeReplicaSize:
                      type: boolean
                    size:
                      minimum: 0
                      type: integer
                    subFailureDomain:
                      type: string
                    targetSizeRatio:
                      type: number
                  required:
                    - size
                  type: object
                statusCheck:
                  properties:
                    mirror:
                      nullable: true
                      properties:
                        disabled:
                          type: boolean
                        interval:
                          type: string
                        timeout:
                          type: string
                      type: object
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                mirroringInfo:
                  properties:
                    details:
                      type: string
                    lastChanged:
                      type: string
                    lastChecked:
                      type: string
                    mode:
                      type: string
                    peers:
                      items:
                        properties:
                          client_name:
                            type: string
                          direction:
                            type: string
                          mirror_uuid:
                            type: string
                          site_name:
                            type: string
                          uuid:
                            type: string
                        type: object
                      type: array
                    site_name:
                      type: string
                  type: object
                mirroringStatus:
                  properties:
                    details:
                      type: string
                    lastChanged:
                      type: string
                    lastChecked:
                      type: string
                    summary:
                      properties:
                        daemon_health:
                          type: string
                        health:
                          type: string
                        image_health:
                          type: string
                        states:
                          nullable: true
                          properties:
                            error:
                              type: integer
                            replaying:
                              type: integer
                            starting_replay:
                              type: integer
                            stopped:
                              type: integer
                            stopping_replay:
                              type: integer
                            syncing:
                              type: integer
                            unknown:
                              type: integer
                          type: object
                      type: object
                  type: object
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
                snapshotScheduleStatus:
                  properties:
                    details:
                      type: string
                    lastChanged:
                      type: string
                    lastChecked:
                      type: string
                    snapshotSchedules:
                      items:
                        properties:
                          image:
                            type: string
                          items:
                            items:
                              properties:
                                interval:
                                  type: string
                                start_time:
                                  type: string
                              type: object
                            type: array
                          namespace:
                            type: string
                          pool:
                            type: string
                        type: object
                      nullable: true
                      type: array
                  type: object
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephbucketnotifications.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephBucketNotification
    listKind: CephBucketNotificationList
    plural: cephbucketnotifications
    singular: cephbucketnotification
  scope: Namespaced
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                events:
                  items:
                    enum:
                      - s3:ObjectCreated:*
                      - s3:ObjectCreated:Put
                      - s3:ObjectCreated:Post
                      - s3:ObjectCreated:Copy
                      - s3:ObjectCreated:CompleteMultipartUpload
                      - s3:ObjectRemoved:*
                      - s3:ObjectRemoved:Delete
                      - s3:ObjectRemoved:DeleteMarkerCreated
                    type: string
                  type: array
                filter:
                  properties:
                    keyFilters:
                      items:
                        properties:
                          name:
                            enum:
                              - prefix
                              - suffix
                              - regex
                            type: string
                          value:
                            type: string
                        required:
                          - name
                          - value
                        type: object
                      type: array
                    metadataFilters:
                      items:
                        properties:
                          name:
                            minLength: 1
                            type: string
                          value:
                            type: string
                        required:
                          - name
                          - value
                        type: object
                      type: array
                    tagFilters:
                      items:
                        properties:
                          name:
                            minLength: 1
                            type: string
                          value:
                            type: string
                        required:
                          - name
                          - value
                        type: object
                      type: array
                  type: object
                topic:
                  minLength: 1
                  type: string
              required:
                - topic
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephbuckettopics.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephBucketTopic
    listKind: CephBucketTopicList
    plural: cephbuckettopics
    singular: cephbuckettopic
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                endpoint:
                  properties:
                    amqp:
                      properties:
                        ackLevel:
                          default: broker
                          enum:
                            - none
                            - broker
                            - routeable
                          type: string
                        disableVerifySSL:
                          type: boolean
                        exchange:
                          minLength: 1
                          type: string
                        uri:
                          minLength: 1
                          type: string
                      required:
                        - exchange
                        - uri
                      type: object
                    http:
                      properties:
                        disableVerifySSL:
                          type: boolean
                        sendCloudEvents:
                          type: boolean
                        uri:
                          minLength: 1
                          type: string
                      required:
                        - uri
                      type: object
                    kafka:
                      properties:
                        ackLevel:
                          default: broker
                          enum:
                            - none
                            - broker
                          type: string
                        disableVerifySSL:
                          type: boolean
                        uri:
                          minLength: 1
                          type: string
                        useSSL:
                          type: boolean
                      required:
                        - uri
                      type: object
                  type: object
                objectStoreName:
                  minLength: 1
                  type: string
                objectStoreNamespace:
                  minLength: 1
                  type: string
                opaqueData:
                  type: string
                persistent:
                  type: boolean
              required:
                - endpoint
                - objectStoreName
                - objectStoreNamespace
              type: object
            status:
              properties:
                ARN:
                  nullable: true
                  type: string
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephclients.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephClient
    listKind: CephClientList
    plural: cephclients
    singular: cephclient
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                caps:
                  additionalProperties:
                    type: string
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                name:
                  type: string
              required:
                - caps
              type: object
            status:
              properties:
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephclusters.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephCluster
    listKind: CephClusterList
    plural: cephclusters
    singular: cephcluster
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
          jsonPath: .spec.dataDirHostPath
          name: DataDirHostPath
          type: string
          jsonPath: .spec.mon.count
          name: MonCount
          type: string
        - jsonPath: .metadata.creationTimestamp
          name: Age
          type: date
        - jsonPath: .status.phase
          name: Phase
          type: string
          jsonPath: .status.message
          name: Message
          type: string
          jsonPath: .status.ceph.health
          name: Health
          type: string
        - jsonPath: .spec.external.enable
          name: External
          type: boolean
          jsonPath: .status.ceph.fsid
          name: FSID
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                annotations:
                  additionalProperties:
                    additionalProperties:
                      type: string
                    type: object
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                cephVersion:
                  nullable: true
                  properties:
                    allowUnsupported:
                      type: boolean
                    image:
                      type: string
                    imagePullPolicy:
                      enum:
                        - IfNotPresent
                        - Always
                        - Never
                        - ""
                      type: string
                  type: object
                cleanupPolicy:
                  nullable: true
                  properties:
                    allowUninstallWithVolumes:
                      type: boolean
                    confirmation:
                      nullable: true
                      pattern: ^$|^yes-really-destroy-data$
                      type: string
                    sanitizeDisks:
                      nullable: true
                      properties:
                        dataSource:
                          enum:
                            - zero
                            - random
                          type: string
                        iteration:
                          format: int32
                          type: integer
                        method:
                          enum:
                            - complete
                            - quick
                          type: string
                      type: object
                  type: object
                continueUpgradeAfterChecksEvenIfNotHealthy:
                  type: boolean
                crashCollector:
                  nullable: true
                  properties:
                    daysToRetain:
                      type: integer
                    disable:
                      type: boolean
                  type: object
                dashboard:
                  nullable: true
                  properties:
                    enabled:
                      type: boolean
                    port:
                      maximum: 65535
                      minimum: 0
                      type: integer
                    ssl:
                      type: boolean
                    urlPrefix:
                      type: string
                  type: object
                dataDirHostPath:
                  pattern: ^/(\S+)
                  type: string
                disruptionManagement:
                  nullable: true
                  properties:
                    machineDisruptionBudgetNamespace:
                      type: string
                    manageMachineDisruptionBudgets:
                      type: boolean
                    managePodBudgets:
                      type: boolean
                    osdMaintenanceTimeout:
                      format: int64
                      type: integer
                    pgHealthCheckTimeout:
                      format: int64
                      type: integer
                  type: object
                external:
                  nullable: true
                  properties:
                    enable:
                      type: boolean
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                healthCheck:
                  nullable: true
                  properties:
                    daemonHealth:
                      nullable: true
                      properties:
                        mon:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                        osd:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                        status:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                      type: object
                    livenessProbe:
                      additionalProperties:
                        properties:
                          disabled:
                            type: boolean
                          probe:
                            properties:
                              exec:
                                properties:
                                  command:
                                    items:
                                      type: string
                                    type: array
                                type: object
                              failureThreshold:
                                format: int32
                                type: integer
                              grpc:
                                properties:
                                  port:
                                    format: int32
                                    type: integer
                                  service:
                                    type: string
                                required:
                                  - port
                                type: object
                              httpGet:
                                properties:
                                  host:
                                    type: string
                                  httpHeaders:
                                    items:
                                      properties:
                                        name:
                                          type: string
                                        value:
                                          type: string
                                      required:
                                        - name
                                        - value
                                      type: object
                                    type: array
                                  path:
                                    type: string
                                  port:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    x-kubernetes-int-or-string: true
                                  scheme:
                                    type: string
                                required:
                                  - port
                                type: object
                              initialDelaySeconds:
                                format: int32
                                type: integer
                              periodSeconds:
                                format: int32
                                type: integer
                              successThreshold:
                                format: int32
                                type: integer
                              tcpSocket:
                                properties:
                                  host:
                                    type: string
                                  port:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    x-kubernetes-int-or-string: true
                                required:
                                  - port
                                type: object
                              terminationGracePeriodSeconds:
                                format: int64
                                type: integer
                              timeoutSeconds:
                                format: int32
                                type: integer
                            type: object
                        type: object
                      type: object
                    startupProbe:
                      additionalProperties:
                        properties:
                          disabled:
                            type: boolean
                          probe:
                            properties:
                              exec:
                                properties:
                                  command:
                                    items:
                                      type: string
                                    type: array
                                type: object
                              failureThreshold:
                                format: int32
                                type: integer
                              grpc:
                                properties:
                                  port:
                                    format: int32
                                    type: integer
                                  service:
                                    type: string
                                required:
                                  - port
                                type: object
                              httpGet:
                                properties:
                                  host:
                                    type: string
                                  httpHeaders:
                                    items:
                                      properties:
                                        name:
                                          type: string
                                        value:
                                          type: string
                                      required:
                                        - name
                                        - value
                                      type: object
                                    type: array
                                  path:
                                    type: string
                                  port:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    x-kubernetes-int-or-string: true
                                  scheme:
                                    type: string
                                required:
                                  - port
                                type: object
                              initialDelaySeconds:
                                format: int32
                                type: integer
                              periodSeconds:
                                format: int32
                                type: integer
                              successThreshold:
                                format: int32
                                type: integer
                              tcpSocket:
                                properties:
                                  host:
                                    type: string
                                  port:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    x-kubernetes-int-or-string: true
                                required:
                                  - port
                                type: object
                              terminationGracePeriodSeconds:
                                format: int64
                                type: integer
                              timeoutSeconds:
                                format: int32
                                type: integer
                            type: object
                        type: object
                      type: object
                  type: object
                labels:
                  additionalProperties:
                    additionalProperties:
                      type: string
                    type: object
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                logCollector:
                  nullable: true
                  properties:
                    enabled:
                      type: boolean
                    maxLogSize:
                      anyOf:
                        - type: integer
                        - type: string
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                    periodicity:
                      pattern: ^$|^(hourly|daily|weekly|monthly|1h|24h|1d)$
                      type: string
                  type: object
                mgr:
                  nullable: true
                  properties:
                    allowMultiplePerNode:
                      type: boolean
                    count:
                      maximum: 2
                      minimum: 0
                      type: integer
                    modules:
                      items:
                        properties:
                          enabled:
                            type: boolean
                          name:
                            type: string
                        type: object
                      nullable: true
                      type: array
                  type: object
                mon:
                  nullable: true
                  properties:
                    allowMultiplePerNode:
                      type: boolean
                    count:
                      maximum: 9
                      minimum: 0
                      type: integer
                    stretchCluster:
                      properties:
                        failureDomainLabel:
                          type: string
                        subFailureDomain:
                          type: string
                        zones:
                          items:
                            properties:
                              arbiter:
                                type: boolean
                              name:
                                type: string
                              volumeClaimTemplate:
                                properties:
                                  apiVersion:
                                    type: string
                                  kind:
                                    type: string
                                  metadata:
                                    properties:
                                      annotations:
                                        additionalProperties:
                                          type: string
                                        type: object
                                      finalizers:
                                        items:
                                          type: string
                                        type: array
                                      labels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                      name:
                                        type: string
                                      namespace:
                                        type: string
                                    type: object
                                  spec:
                                    properties:
                                      accessModes:
                                        items:
                                          type: string
                                        type: array
                                      dataSource:
                                        properties:
                                          apiGroup:
                                            type: string
                                          kind:
                                            type: string
                                          name:
                                            type: string
                                        required:
                                          - kind
                                          - name
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      dataSourceRef:
                                        properties:
                                          apiGroup:
                                            type: string
                                          kind:
                                            type: string
                                          name:
                                            type: string
                                          namespace:
                                            type: string
                                        required:
                                          - kind
                                          - name
                                        type: object
                                      resources:
                                        properties:
                                          claims:
                                            items:
                                              properties:
                                                name:
                                                  type: string
                                              required:
                                                - name
                                              type: object
                                            type: array
                                            x-kubernetes-list-map-keys:
                                              - name
                                            x-kubernetes-list-type: map
                                          limits:
                                            additionalProperties:
                                              anyOf:
                                                - type: integer
                                                - type: string
                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                              x-kubernetes-int-or-string: true
                                            type: object
                                          requests:
                                            additionalProperties:
                                              anyOf:
                                                - type: integer
                                                - type: string
                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                              x-kubernetes-int-or-string: true
                                            type: object
                                        type: object
                                      selector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      storageClassName:
                                        type: string
                                      volumeMode:
                                        type: string
                                      volumeName:
                                        type: string
                                    type: object
                                  status:
                                    properties:
                                      accessModes:
                                        items:
                                          type: string
                                        type: array
                                      allocatedResources:
                                        additionalProperties:
                                          anyOf:
                                            - type: integer
                                            - type: string
                                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                          x-kubernetes-int-or-string: true
                                        type: object
                                      capacity:
                                        additionalProperties:
                                          anyOf:
                                            - type: integer
                                            - type: string
                                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                          x-kubernetes-int-or-string: true
                                        type: object
                                      conditions:
                                        items:
                                          properties:
                                            lastProbeTime:
                                              format: date-time
                                              type: string
                                            lastTransitionTime:
                                              format: date-time
                                              type: string
                                            message:
                                              type: string
                                            reason:
                                              type: string
                                            status:
                                              type: string
                                            type:
                                              type: string
                                          required:
                                            - status
                                            - type
                                          type: object
                                        type: array
                                      phase:
                                        type: string
                                      resizeStatus:
                                        type: string
                                    type: object
                                type: object
                                x-kubernetes-preserve-unknown-fields: true
                            type: object
                          nullable: true
                          type: array
                      type: object
                    volumeClaimTemplate:
                      properties:
                        apiVersion:
                          type: string
                        kind:
                          type: string
                        metadata:
                          properties:
                            annotations:
                              additionalProperties:
                                type: string
                              type: object
                            finalizers:
                              items:
                                type: string
                              type: array
                            labels:
                              additionalProperties:
                                type: string
                              type: object
                            name:
                              type: string
                            namespace:
                              type: string
                          type: object
                        spec:
                          properties:
                            accessModes:
                              items:
                                type: string
                              type: array
                            dataSource:
                              properties:
                                apiGroup:
                                  type: string
                                kind:
                                  type: string
                                name:
                                  type: string
                              required:
                                - kind
                                - name
                              type: object
                              x-kubernetes-map-type: atomic
                            dataSourceRef:
                              properties:
                                apiGroup:
                                  type: string
                                kind:
                                  type: string
                                name:
                                  type: string
                                namespace:
                                  type: string
                              required:
                                - kind
                                - name
                              type: object
                            resources:
                              properties:
                                claims:
                                  items:
                                    properties:
                                      name:
                                        type: string
                                    required:
                                      - name
                                    type: object
                                  type: array
                                  x-kubernetes-list-map-keys:
                                    - name
                                  x-kubernetes-list-type: map
                                limits:
                                  additionalProperties:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                    x-kubernetes-int-or-string: true
                                  type: object
                                requests:
                                  additionalProperties:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                    x-kubernetes-int-or-string: true
                                  type: object
                              type: object
                            selector:
                              properties:
                                matchExpressions:
                                  items:
                                    properties:
                                      key:
                                        type: string
                                      operator:
                                        type: string
                                      values:
                                        items:
                                          type: string
                                        type: array
                                    required:
                                      - key
                                      - operator
                                    type: object
                                  type: array
                                matchLabels:
                                  additionalProperties:
                                    type: string
                                  type: object
                              type: object
                              x-kubernetes-map-type: atomic
                            storageClassName:
                              type: string
                            volumeMode:
                              type: string
                            volumeName:
                              type: string
                          type: object
                        status:
                          properties:
                            accessModes:
                              items:
                                type: string
                              type: array
                            allocatedResources:
                              additionalProperties:
                                anyOf:
                                  - type: integer
                                  - type: string
                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                x-kubernetes-int-or-string: true
                              type: object
                            capacity:
                              additionalProperties:
                                anyOf:
                                  - type: integer
                                  - type: string
                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                x-kubernetes-int-or-string: true
                              type: object
                            conditions:
                              items:
                                properties:
                                  lastProbeTime:
                                    format: date-time
                                    type: string
                                  lastTransitionTime:
                                    format: date-time
                                    type: string
                                  message:
                                    type: string
                                  reason:
                                    type: string
                                  status:
                                    type: string
                                  type:
                                    type: string
                                required:
                                  - status
                                  - type
                                type: object
                              type: array
                            phase:
                              type: string
                            resizeStatus:
                              type: string
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  type: object
                monitoring:
                  nullable: true
                  properties:
                    enabled:
                      type: boolean
                    externalMgrEndpoints:
                      items:
                        properties:
                          hostname:
                            type: string
                          ip:
                            type: string
                          nodeName:
                            type: string
                          targetRef:
                            properties:
                              apiVersion:
                                type: string
                              fieldPath:
                                type: string
                              kind:
                                type: string
                              name:
                                type: string
                              namespace:
                                type: string
                              resourceVersion:
                                type: string
                              uid:
                                type: string
                            type: object
                            x-kubernetes-map-type: atomic
                        required:
                          - ip
                        type: object
                        x-kubernetes-map-type: atomic
                      nullable: true
                      type: array
                    externalMgrPrometheusPort:
                      maximum: 65535
                      minimum: 0
                      type: integer
                    interval:
                      type: string
                    metricsDisabled:
                      type: boolean
                    port:
                      maximum: 65535
                      minimum: 0
                      type: integer
                  type: object
                network:
                  nullable: true
                  properties:
                    connections:
                      nullable: true
                      properties:
                        compression:
                          nullable: true
                          properties:
                            enabled:
                              type: boolean
                          type: object
                        encryption:
                          nullable: true
                          properties:
                            enabled:
                              type: boolean
                          type: object
                        requireMsgr2:
                          type: boolean
                      type: object
                    dualStack:
                      type: boolean
                    hostNetwork:
                      type: boolean
                    ipFamily:
                      enum:
                        - IPv4
                        - IPv6
                      nullable: true
                      type: string
                    multiClusterService:
                      properties:
                        clusterID:
                          type: string
                        enabled:
                          type: boolean
                      type: object
                    provider:
                      nullable: true
                      type: string
                    selectors:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                placement:
                  additionalProperties:
                    properties:
                      nodeAffinity:
                        properties:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            items:
                              properties:
                                preference:
                                  properties:
                                    matchExpressions:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                    matchFields:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                  type: object
                                  x-kubernetes-map-type: atomic
                                weight:
                                  format: int32
                                  type: integer
                              required:
                                - preference
                                - weight
                              type: object
                            type: array
                          requiredDuringSchedulingIgnoredDuringExecution:
                            properties:
                              nodeSelectorTerms:
                                items:
                                  properties:
                                    matchExpressions:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                    matchFields:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                  type: object
                                  x-kubernetes-map-type: atomic
                                type: array
                            required:
                              - nodeSelectorTerms
                            type: object
                            x-kubernetes-map-type: atomic
                        type: object
                      podAffinity:
                        properties:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            items:
                              properties:
                                podAffinityTerm:
                                  properties:
                                    labelSelector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    namespaceSelector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    namespaces:
                                      items:
                                        type: string
                                      type: array
                                    topologyKey:
                                      type: string
                                  required:
                                    - topologyKey
                                  type: object
                                weight:
                                  format: int32
                                  type: integer
                              required:
                                - podAffinityTerm
                                - weight
                              type: object
                            type: array
                          requiredDuringSchedulingIgnoredDuringExecution:
                            items:
                              properties:
                                labelSelector:
                                  properties:
                                    matchExpressions:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      type: object
                                  type: object
                                  x-kubernetes-map-type: atomic
                                namespaceSelector:
                                  properties:
                                    matchExpressions:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      type: object
                                  type: object
                                  x-kubernetes-map-type: atomic
                                namespaces:
                                  items:
                                    type: string
                                  type: array
                                topologyKey:
                                  type: string
                              required:
                                - topologyKey
                              type: object
                            type: array
                        type: object
                      podAntiAffinity:
                        properties:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            items:
                              properties:
                                podAffinityTerm:
                                  properties:
                                    labelSelector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    namespaceSelector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    namespaces:
                                      items:
                                        type: string
                                      type: array
                                    topologyKey:
                                      type: string
                                  required:
                                    - topologyKey
                                  type: object
                                weight:
                                  format: int32
                                  type: integer
                              required:
                                - podAffinityTerm
                                - weight
                              type: object
                            type: array
                          requiredDuringSchedulingIgnoredDuringExecution:
                            items:
                              properties:
                                labelSelector:
                                  properties:
                                    matchExpressions:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      type: object
                                  type: object
                                  x-kubernetes-map-type: atomic
                                namespaceSelector:
                                  properties:
                                    matchExpressions:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          operator:
                                            type: string
                                          values:
                                            items:
                                              type: string
                                            type: array
                                        required:
                                          - key
                                          - operator
                                        type: object
                                      type: array
                                    matchLabels:
                                      additionalProperties:
                                        type: string
                                      type: object
                                  type: object
                                  x-kubernetes-map-type: atomic
                                namespaces:
                                  items:
                                    type: string
                                  type: array
                                topologyKey:
                                  type: string
                              required:
                                - topologyKey
                              type: object
                            type: array
                        type: object
                      tolerations:
                        items:
                          properties:
                            effect:
                              type: string
                            key:
                              type: string
                            operator:
                              type: string
                            tolerationSeconds:
                              format: int64
                              type: integer
                            value:
                              type: string
                          type: object
                        type: array
                      topologySpreadConstraints:
                        items:
                          properties:
                            labelSelector:
                              properties:
                                matchExpressions:
                                  items:
                                    properties:
                                      key:
                                        type: string
                                      operator:
                                        type: string
                                      values:
                                        items:
                                          type: string
                                        type: array
                                    required:
                                      - key
                                      - operator
                                    type: object
                                  type: array
                                matchLabels:
                                  additionalProperties:
                                    type: string
                                  type: object
                              type: object
                              x-kubernetes-map-type: atomic
                            matchLabelKeys:
                              items:
                                type: string
                              type: array
                              x-kubernetes-list-type: atomic
                            maxSkew:
                              format: int32
                              type: integer
                            minDomains:
                              format: int32
                              type: integer
                            nodeAffinityPolicy:
                              type: string
                            nodeTaintsPolicy:
                              type: string
                            topologyKey:
                              type: string
                            whenUnsatisfiable:
                              type: string
                          required:
                            - maxSkew
                            - topologyKey
                            - whenUnsatisfiable
                          type: object
                        type: array
                    type: object
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                priorityClassNames:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                removeOSDsIfOutAndSafeToRemove:
                  type: boolean
                resources:
                  additionalProperties:
                    properties:
                      claims:
                        items:
                          properties:
                            name:
                              type: string
                          required:
                            - name
                          type: object
                        type: array
                        x-kubernetes-list-map-keys:
                          - name
                        x-kubernetes-list-type: map
                      limits:
                        additionalProperties:
                          anyOf:
                            - type: integer
                            - type: string
                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                          x-kubernetes-int-or-string: true
                        type: object
                      requests:
                        additionalProperties:
                          anyOf:
                            - type: integer
                            - type: string
                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                          x-kubernetes-int-or-string: true
                        type: object
                    type: object
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                security:
                  nullable: true
                  properties:
                    keyRotation:
                      nullable: true
                      properties:
                        enabled:
                          default: false
                          type: boolean
                        schedule:
                          type: string
                      type: object
                    kms:
                      nullable: true
                      properties:
                        connectionDetails:
                          additionalProperties:
                            type: string
                          nullable: true
                          type: object
                          x-kubernetes-preserve-unknown-fields: true
                        tokenSecretName:
                          type: string
                      type: object
                  type: object
                skipUpgradeChecks:
                  type: boolean
                storage:
                  nullable: true
                  properties:
                    config:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    deviceFilter:
                      type: string
                    devicePathFilter:
                      type: string
                    devices:
                      items:
                        properties:
                          config:
                            additionalProperties:
                              type: string
                            nullable: true
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          fullpath:
                            type: string
                          name:
                            type: string
                        type: object
                      nullable: true
                      type: array
                      x-kubernetes-preserve-unknown-fields: true
                    nodes:
                      items:
                        properties:
                          config:
                            additionalProperties:
                              type: string
                            nullable: true
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          deviceFilter:
                            type: string
                          devicePathFilter:
                            type: string
                          devices:
                            items:
                              properties:
                                config:
                                  additionalProperties:
                                    type: string
                                  nullable: true
                                  type: object
                                  x-kubernetes-preserve-unknown-fields: true
                                fullpath:
                                  type: string
                                name:
                                  type: string
                              type: object
                            nullable: true
                            type: array
                            x-kubernetes-preserve-unknown-fields: true
                          name:
                            type: string
                          resources:
                            nullable: true
                            properties:
                              claims:
                                items:
                                  properties:
                                    name:
                                      type: string
                                  required:
                                    - name
                                  type: object
                                type: array
                                x-kubernetes-list-map-keys:
                                  - name
                                x-kubernetes-list-type: map
                              limits:
                                additionalProperties:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                  x-kubernetes-int-or-string: true
                                type: object
                              requests:
                                additionalProperties:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                  x-kubernetes-int-or-string: true
                                type: object
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          useAllDevices:
                            type: boolean
                          volumeClaimTemplates:
                            items:
                              properties:
                                apiVersion:
                                  type: string
                                kind:
                                  type: string
                                metadata:
                                  properties:
                                    annotations:
                                      additionalProperties:
                                        type: string
                                      type: object
                                    finalizers:
                                      items:
                                        type: string
                                      type: array
                                    labels:
                                      additionalProperties:
                                        type: string
                                      type: object
                                    name:
                                      type: string
                                    namespace:
                                      type: string
                                  type: object
                                spec:
                                  properties:
                                    accessModes:
                                      items:
                                        type: string
                                      type: array
                                    dataSource:
                                      properties:
                                        apiGroup:
                                          type: string
                                        kind:
                                          type: string
                                        name:
                                          type: string
                                      required:
                                        - kind
                                        - name
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    dataSourceRef:
                                      properties:
                                        apiGroup:
                                          type: string
                                        kind:
                                          type: string
                                        name:
                                          type: string
                                        namespace:
                                          type: string
                                      required:
                                        - kind
                                        - name
                                      type: object
                                    resources:
                                      properties:
                                        claims:
                                          items:
                                            properties:
                                              name:
                                                type: string
                                            required:
                                              - name
                                            type: object
                                          type: array
                                          x-kubernetes-list-map-keys:
                                            - name
                                          x-kubernetes-list-type: map
                                        limits:
                                          additionalProperties:
                                            anyOf:
                                              - type: integer
                                              - type: string
                                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                            x-kubernetes-int-or-string: true
                                          type: object
                                        requests:
                                          additionalProperties:
                                            anyOf:
                                              - type: integer
                                              - type: string
                                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                            x-kubernetes-int-or-string: true
                                          type: object
                                      type: object
                                    selector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    storageClassName:
                                      type: string
                                    volumeMode:
                                      type: string
                                    volumeName:
                                      type: string
                                  type: object
                                status:
                                  properties:
                                    accessModes:
                                      items:
                                        type: string
                                      type: array
                                    allocatedResources:
                                      additionalProperties:
                                        anyOf:
                                          - type: integer
                                          - type: string
                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                        x-kubernetes-int-or-string: true
                                      type: object
                                    capacity:
                                      additionalProperties:
                                        anyOf:
                                          - type: integer
                                          - type: string
                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                        x-kubernetes-int-or-string: true
                                      type: object
                                    conditions:
                                      items:
                                        properties:
                                          lastProbeTime:
                                            format: date-time
                                            type: string
                                          lastTransitionTime:
                                            format: date-time
                                            type: string
                                          message:
                                            type: string
                                          reason:
                                            type: string
                                          status:
                                            type: string
                                          type:
                                            type: string
                                        required:
                                          - status
                                          - type
                                        type: object
                                      type: array
                                    phase:
                                      type: string
                                    resizeStatus:
                                      type: string
                                  type: object
                              type: object
                            type: array
                        type: object
                      nullable: true
                      type: array
                    onlyApplyOSDPlacement:
                      type: boolean
                    storageClassDeviceSets:
                      items:
                        properties:
                          config:
                            additionalProperties:
                              type: string
                            nullable: true
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          count:
                            minimum: 1
                            type: integer
                          encrypted:
                            type: boolean
                          name:
                            type: string
                          placement:
                            nullable: true
                            properties:
                              nodeAffinity:
                                properties:
                                  preferredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        preference:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchFields:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        weight:
                                          format: int32
                                          type: integer
                                      required:
                                        - preference
                                        - weight
                                      type: object
                                    type: array
                                  requiredDuringSchedulingIgnoredDuringExecution:
                                    properties:
                                      nodeSelectorTerms:
                                        items:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchFields:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        type: array
                                    required:
                                      - nodeSelectorTerms
                                    type: object
                                    x-kubernetes-map-type: atomic
                                type: object
                              podAffinity:
                                properties:
                                  preferredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        podAffinityTerm:
                                          properties:
                                            labelSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaceSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaces:
                                              items:
                                                type: string
                                              type: array
                                            topologyKey:
                                              type: string
                                          required:
                                            - topologyKey
                                          type: object
                                        weight:
                                          format: int32
                                          type: integer
                                      required:
                                        - podAffinityTerm
                                        - weight
                                      type: object
                                    type: array
                                  requiredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        labelSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaceSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaces:
                                          items:
                                            type: string
                                          type: array
                                        topologyKey:
                                          type: string
                                      required:
                                        - topologyKey
                                      type: object
                                    type: array
                                type: object
                              podAntiAffinity:
                                properties:
                                  preferredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        podAffinityTerm:
                                          properties:
                                            labelSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaceSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaces:
                                              items:
                                                type: string
                                              type: array
                                            topologyKey:
                                              type: string
                                          required:
                                            - topologyKey
                                          type: object
                                        weight:
                                          format: int32
                                          type: integer
                                      required:
                                        - podAffinityTerm
                                        - weight
                                      type: object
                                    type: array
                                  requiredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        labelSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaceSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaces:
                                          items:
                                            type: string
                                          type: array
                                        topologyKey:
                                          type: string
                                      required:
                                        - topologyKey
                                      type: object
                                    type: array
                                type: object
                              tolerations:
                                items:
                                  properties:
                                    effect:
                                      type: string
                                    key:
                                      type: string
                                    operator:
                                      type: string
                                    tolerationSeconds:
                                      format: int64
                                      type: integer
                                    value:
                                      type: string
                                  type: object
                                type: array
                              topologySpreadConstraints:
                                items:
                                  properties:
                                    labelSelector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    matchLabelKeys:
                                      items:
                                        type: string
                                      type: array
                                      x-kubernetes-list-type: atomic
                                    maxSkew:
                                      format: int32
                                      type: integer
                                    minDomains:
                                      format: int32
                                      type: integer
                                    nodeAffinityPolicy:
                                      type: string
                                    nodeTaintsPolicy:
                                      type: string
                                    topologyKey:
                                      type: string
                                    whenUnsatisfiable:
                                      type: string
                                  required:
                                    - maxSkew
                                    - topologyKey
                                    - whenUnsatisfiable
                                  type: object
                                type: array
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          portable:
                            type: boolean
                          preparePlacement:
                            nullable: true
                            properties:
                              nodeAffinity:
                                properties:
                                  preferredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        preference:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchFields:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        weight:
                                          format: int32
                                          type: integer
                                      required:
                                        - preference
                                        - weight
                                      type: object
                                    type: array
                                  requiredDuringSchedulingIgnoredDuringExecution:
                                    properties:
                                      nodeSelectorTerms:
                                        items:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchFields:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        type: array
                                    required:
                                      - nodeSelectorTerms
                                    type: object
                                    x-kubernetes-map-type: atomic
                                type: object
                              podAffinity:
                                properties:
                                  preferredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        podAffinityTerm:
                                          properties:
                                            labelSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaceSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaces:
                                              items:
                                                type: string
                                              type: array
                                            topologyKey:
                                              type: string
                                          required:
                                            - topologyKey
                                          type: object
                                        weight:
                                          format: int32
                                          type: integer
                                      required:
                                        - podAffinityTerm
                                        - weight
                                      type: object
                                    type: array
                                  requiredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        labelSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaceSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaces:
                                          items:
                                            type: string
                                          type: array
                                        topologyKey:
                                          type: string
                                      required:
                                        - topologyKey
                                      type: object
                                    type: array
                                type: object
                              podAntiAffinity:
                                properties:
                                  preferredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        podAffinityTerm:
                                          properties:
                                            labelSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaceSelector:
                                              properties:
                                                matchExpressions:
                                                  items:
                                                    properties:
                                                      key:
                                                        type: string
                                                      operator:
                                                        type: string
                                                      values:
                                                        items:
                                                          type: string
                                                        type: array
                                                    required:
                                                      - key
                                                      - operator
                                                    type: object
                                                  type: array
                                                matchLabels:
                                                  additionalProperties:
                                                    type: string
                                                  type: object
                                              type: object
                                              x-kubernetes-map-type: atomic
                                            namespaces:
                                              items:
                                                type: string
                                              type: array
                                            topologyKey:
                                              type: string
                                          required:
                                            - topologyKey
                                          type: object
                                        weight:
                                          format: int32
                                          type: integer
                                      required:
                                        - podAffinityTerm
                                        - weight
                                      type: object
                                    type: array
                                  requiredDuringSchedulingIgnoredDuringExecution:
                                    items:
                                      properties:
                                        labelSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaceSelector:
                                          properties:
                                            matchExpressions:
                                              items:
                                                properties:
                                                  key:
                                                    type: string
                                                  operator:
                                                    type: string
                                                  values:
                                                    items:
                                                      type: string
                                                    type: array
                                                required:
                                                  - key
                                                  - operator
                                                type: object
                                              type: array
                                            matchLabels:
                                              additionalProperties:
                                                type: string
                                              type: object
                                          type: object
                                          x-kubernetes-map-type: atomic
                                        namespaces:
                                          items:
                                            type: string
                                          type: array
                                        topologyKey:
                                          type: string
                                      required:
                                        - topologyKey
                                      type: object
                                    type: array
                                type: object
                              tolerations:
                                items:
                                  properties:
                                    effect:
                                      type: string
                                    key:
                                      type: string
                                    operator:
                                      type: string
                                    tolerationSeconds:
                                      format: int64
                                      type: integer
                                    value:
                                      type: string
                                  type: object
                                type: array
                              topologySpreadConstraints:
                                items:
                                  properties:
                                    labelSelector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    matchLabelKeys:
                                      items:
                                        type: string
                                      type: array
                                      x-kubernetes-list-type: atomic
                                    maxSkew:
                                      format: int32
                                      type: integer
                                    minDomains:
                                      format: int32
                                      type: integer
                                    nodeAffinityPolicy:
                                      type: string
                                    nodeTaintsPolicy:
                                      type: string
                                    topologyKey:
                                      type: string
                                    whenUnsatisfiable:
                                      type: string
                                  required:
                                    - maxSkew
                                    - topologyKey
                                    - whenUnsatisfiable
                                  type: object
                                type: array
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          resources:
                            nullable: true
                            properties:
                              claims:
                                items:
                                  properties:
                                    name:
                                      type: string
                                  required:
                                    - name
                                  type: object
                                type: array
                                x-kubernetes-list-map-keys:
                                  - name
                                x-kubernetes-list-type: map
                              limits:
                                additionalProperties:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                  x-kubernetes-int-or-string: true
                                type: object
                              requests:
                                additionalProperties:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                  x-kubernetes-int-or-string: true
                                type: object
                            type: object
                            x-kubernetes-preserve-unknown-fields: true
                          schedulerName:
                            type: string
                          tuneDeviceClass:
                            type: boolean
                          tuneFastDeviceClass:
                            type: boolean
                          volumeClaimTemplates:
                            items:
                              properties:
                                apiVersion:
                                  type: string
                                kind:
                                  type: string
                                metadata:
                                  properties:
                                    annotations:
                                      additionalProperties:
                                        type: string
                                      type: object
                                      x-kubernetes-preserve-unknown-fields: true
                                    finalizers:
                                      items:
                                        type: string
                                      type: array
                                    labels:
                                      additionalProperties:
                                        type: string
                                      type: object
                                    name:
                                      type: string
                                    namespace:
                                      type: string
                                  type: object
                                spec:
                                  properties:
                                    accessModes:
                                      items:
                                        type: string
                                      type: array
                                    dataSource:
                                      properties:
                                        apiGroup:
                                          type: string
                                        kind:
                                          type: string
                                        name:
                                          type: string
                                      required:
                                        - kind
                                        - name
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    dataSourceRef:
                                      properties:
                                        apiGroup:
                                          type: string
                                        kind:
                                          type: string
                                        name:
                                          type: string
                                        namespace:
                                          type: string
                                      required:
                                        - kind
                                        - name
                                      type: object
                                    resources:
                                      properties:
                                        claims:
                                          items:
                                            properties:
                                              name:
                                                type: string
                                            required:
                                              - name
                                            type: object
                                          type: array
                                          x-kubernetes-list-map-keys:
                                            - name
                                          x-kubernetes-list-type: map
                                        limits:
                                          additionalProperties:
                                            anyOf:
                                              - type: integer
                                              - type: string
                                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                            x-kubernetes-int-or-string: true
                                          type: object
                                        requests:
                                          additionalProperties:
                                            anyOf:
                                              - type: integer
                                              - type: string
                                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                            x-kubernetes-int-or-string: true
                                          type: object
                                      type: object
                                    selector:
                                      properties:
                                        matchExpressions:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              operator:
                                                type: string
                                              values:
                                                items:
                                                  type: string
                                                type: array
                                            required:
                                              - key
                                              - operator
                                            type: object
                                          type: array
                                        matchLabels:
                                          additionalProperties:
                                            type: string
                                          type: object
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    storageClassName:
                                      type: string
                                    volumeMode:
                                      type: string
                                    volumeName:
                                      type: string
                                  type: object
                                status:
                                  properties:
                                    accessModes:
                                      items:
                                        type: string
                                      type: array
                                    allocatedResources:
                                      additionalProperties:
                                        anyOf:
                                          - type: integer
                                          - type: string
                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                        x-kubernetes-int-or-string: true
                                      type: object
                                    capacity:
                                      additionalProperties:
                                        anyOf:
                                          - type: integer
                                          - type: string
                                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                        x-kubernetes-int-or-string: true
                                      type: object
                                    conditions:
                                      items:
                                        properties:
                                          lastProbeTime:
                                            format: date-time
                                            type: string
                                          lastTransitionTime:
                                            format: date-time
                                            type: string
                                          message:
                                            type: string
                                          reason:
                                            type: string
                                          status:
                                            type: string
                                          type:
                                            type: string
                                        required:
                                          - status
                                          - type
                                        type: object
                                      type: array
                                    phase:
                                      type: string
                                    resizeStatus:
                                      type: string
                                  type: object
                              type: object
                            type: array
                        required:
                          - count
                          - name
                          - volumeClaimTemplates
                        type: object
                      nullable: true
                      type: array
                    useAllDevices:
                      type: boolean
                    useAllNodes:
                      type: boolean
                    volumeClaimTemplates:
                      items:
                        properties:
                          apiVersion:
                            type: string
                          kind:
                            type: string
                          metadata:
                            properties:
                              annotations:
                                additionalProperties:
                                  type: string
                                type: object
                              finalizers:
                                items:
                                  type: string
                                type: array
                              labels:
                                additionalProperties:
                                  type: string
                                type: object
                              name:
                                type: string
                              namespace:
                                type: string
                            type: object
                          spec:
                            properties:
                              accessModes:
                                items:
                                  type: string
                                type: array
                              dataSource:
                                properties:
                                  apiGroup:
                                    type: string
                                  kind:
                                    type: string
                                  name:
                                    type: string
                                required:
                                  - kind
                                  - name
                                type: object
                                x-kubernetes-map-type: atomic
                              dataSourceRef:
                                properties:
                                  apiGroup:
                                    type: string
                                  kind:
                                    type: string
                                  name:
                                    type: string
                                  namespace:
                                    type: string
                                required:
                                  - kind
                                  - name
                                type: object
                              resources:
                                properties:
                                  claims:
                                    items:
                                      properties:
                                        name:
                                          type: string
                                      required:
                                        - name
                                      type: object
                                    type: array
                                    x-kubernetes-list-map-keys:
                                      - name
                                    x-kubernetes-list-type: map
                                  limits:
                                    additionalProperties:
                                      anyOf:
                                        - type: integer
                                        - type: string
                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                      x-kubernetes-int-or-string: true
                                    type: object
                                  requests:
                                    additionalProperties:
                                      anyOf:
                                        - type: integer
                                        - type: string
                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                      x-kubernetes-int-or-string: true
                                    type: object
                                type: object
                              selector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              storageClassName:
                                type: string
                              volumeMode:
                                type: string
                              volumeName:
                                type: string
                            type: object
                          status:
                            properties:
                              accessModes:
                                items:
                                  type: string
                                type: array
                              allocatedResources:
                                additionalProperties:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                  x-kubernetes-int-or-string: true
                                type: object
                              capacity:
                                additionalProperties:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                  x-kubernetes-int-or-string: true
                                type: object
                              conditions:
                                items:
                                  properties:
                                    lastProbeTime:
                                      format: date-time
                                      type: string
                                    lastTransitionTime:
                                      format: date-time
                                      type: string
                                    message:
                                      type: string
                                    reason:
                                      type: string
                                    status:
                                      type: string
                                    type:
                                      type: string
                                  required:
                                    - status
                                    - type
                                  type: object
                                type: array
                              phase:
                                type: string
                              resizeStatus:
                                type: string
                            type: object
                        type: object
                      type: array
                  type: object
                waitTimeoutForHealthyOSDInMinutes:
                  format: int64
                  type: integer
              type: object
            status:
              nullable: true
              properties:
                ceph:
                  properties:
                    capacity:
                      properties:
                        bytesAvailable:
                          format: int64
                          type: integer
                        bytesTotal:
                          format: int64
                          type: integer
                        bytesUsed:
                          format: int64
                          type: integer
                        lastUpdated:
                          type: string
                      type: object
                    details:
                      additionalProperties:
                        properties:
                          message:
                            type: string
                          severity:
                            type: string
                        required:
                          - message
                          - severity
                        type: object
                      type: object
                    fsid:
                      type: string
                    health:
                      type: string
                    lastChanged:
                      type: string
                    lastChecked:
                      type: string
                    previousHealth:
                      type: string
                    versions:
                      properties:
                        cephfs-mirror:
                          additionalProperties:
                            type: integer
                          type: object
                        mds:
                          additionalProperties:
                            type: integer
                          type: object
                        mgr:
                          additionalProperties:
                            type: integer
                          type: object
                        mon:
                          additionalProperties:
                            type: integer
                          type: object
                        osd:
                          additionalProperties:
                            type: integer
                          type: object
                        overall:
                          additionalProperties:
                            type: integer
                          type: object
                        rbd-mirror:
                          additionalProperties:
                            type: integer
                          type: object
                        rgw:
                          additionalProperties:
                            type: integer
                          type: object
                      type: object
                  type: object
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                message:
                  type: string
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
                state:
                  type: string
                storage:
                  properties:
                    deviceClasses:
                      items:
                        properties:
                          name:
                            type: string
                        type: object
                      type: array
                  type: object
                version:
                  properties:
                    image:
                      type: string
                    version:
                      type: string
                  type: object
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephfilesystemmirrors.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephFilesystemMirror
    listKind: CephFilesystemMirrorList
    plural: cephfilesystemmirrors
    singular: cephfilesystemmirror
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                annotations:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                labels:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                placement:
                  nullable: true
                  properties:
                    nodeAffinity:
                      properties:
                        preferredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              preference:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchFields:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                type: object
                                x-kubernetes-map-type: atomic
                              weight:
                                format: int32
                                type: integer
                            required:
                              - preference
                              - weight
                            type: object
                          type: array
                        requiredDuringSchedulingIgnoredDuringExecution:
                          properties:
                            nodeSelectorTerms:
                              items:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchFields:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                type: object
                                x-kubernetes-map-type: atomic
                              type: array
                          required:
                            - nodeSelectorTerms
                          type: object
                          x-kubernetes-map-type: atomic
                      type: object
                    podAffinity:
                      properties:
                        preferredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              podAffinityTerm:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              weight:
                                format: int32
                                type: integer
                            required:
                              - podAffinityTerm
                              - weight
                            type: object
                          type: array
                        requiredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaceSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaces:
                                items:
                                  type: string
                                type: array
                              topologyKey:
                                type: string
                            required:
                              - topologyKey
                            type: object
                          type: array
                      type: object
                    podAntiAffinity:
                      properties:
                        preferredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              podAffinityTerm:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              weight:
                                format: int32
                                type: integer
                            required:
                              - podAffinityTerm
                              - weight
                            type: object
                          type: array
                        requiredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaceSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaces:
                                items:
                                  type: string
                                type: array
                              topologyKey:
                                type: string
                            required:
                              - topologyKey
                            type: object
                          type: array
                      type: object
                    tolerations:
                      items:
                        properties:
                          effect:
                            type: string
                          key:
                            type: string
                          operator:
                            type: string
                          tolerationSeconds:
                            format: int64
                            type: integer
                          value:
                            type: string
                        type: object
                      type: array
                    topologySpreadConstraints:
                      items:
                        properties:
                          labelSelector:
                            properties:
                              matchExpressions:
                                items:
                                  properties:
                                    key:
                                      type: string
                                    operator:
                                      type: string
                                    values:
                                      items:
                                        type: string
                                      type: array
                                  required:
                                    - key
                                    - operator
                                  type: object
                                type: array
                              matchLabels:
                                additionalProperties:
                                  type: string
                                type: object
                            type: object
                            x-kubernetes-map-type: atomic
                          matchLabelKeys:
                            items:
                              type: string
                            type: array
                            x-kubernetes-list-type: atomic
                          maxSkew:
                            format: int32
                            type: integer
                          minDomains:
                            format: int32
                            type: integer
                          nodeAffinityPolicy:
                            type: string
                          nodeTaintsPolicy:
                            type: string
                          topologyKey:
                            type: string
                          whenUnsatisfiable:
                            type: string
                        required:
                          - maxSkew
                          - topologyKey
                          - whenUnsatisfiable
                        type: object
                      type: array
                  type: object
                priorityClassName:
                  type: string
                resources:
                  nullable: true
                  properties:
                    claims:
                      items:
                        properties:
                          name:
                            type: string
                        required:
                          - name
                        type: object
                      type: array
                      x-kubernetes-list-map-keys:
                        - name
                      x-kubernetes-list-type: map
                    limits:
                      additionalProperties:
                        anyOf:
                          - type: integer
                          - type: string
                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                        x-kubernetes-int-or-string: true
                      type: object
                    requests:
                      additionalProperties:
                        anyOf:
                          - type: integer
                          - type: string
                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                        x-kubernetes-int-or-string: true
                      type: object
                  type: object
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephfilesystems.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephFilesystem
    listKind: CephFilesystemList
    plural: cephfilesystems
    singular: cephfilesystem
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
          jsonPath: .spec.metadataServer.activeCount
          name: ActiveMDS
          type: string
        - jsonPath: .metadata.creationTimestamp
          name: Age
          type: date
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                dataPools:
                  items:
                    properties:
                      compressionMode:
                        enum:
                          - none
                          - passive
                          - aggressive
                          - force
                          - ""
                        nullable: true
                        type: string
                      crushRoot:
                        nullable: true
                        type: string
                      deviceClass:
                        nullable: true
                        type: string
                      enableRBDStats:
                        type: boolean
                      erasureCoded:
                        properties:
                          algorithm:
                            type: string
                          codingChunks:
                            minimum: 0
                            type: integer
                          dataChunks:
                            minimum: 0
                            type: integer
                        required:
                          - codingChunks
                          - dataChunks
                        type: object
                      failureDomain:
                        type: string
                      mirroring:
                        properties:
                          enabled:
                            type: boolean
                          mode:
                            type: string
                          peers:
                            nullable: true
                            properties:
                              secretNames:
                                items:
                                  type: string
                                type: array
                            type: object
                          snapshotSchedules:
                            items:
                              properties:
                                interval:
                                  type: string
                                path:
                                  type: string
                                startTime:
                                  type: string
                              type: object
                            type: array
                        type: object
                      name:
                        type: string
                      parameters:
                        additionalProperties:
                          type: string
                        nullable: true
                        type: object
                        x-kubernetes-preserve-unknown-fields: true
                      quotas:
                        nullable: true
                        properties:
                          maxBytes:
                            format: int64
                            type: integer
                          maxObjects:
                            format: int64
                            type: integer
                          maxSize:
                            pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                            type: string
                        type: object
                      replicated:
                        properties:
                          hybridStorage:
                            nullable: true
                            properties:
                              primaryDeviceClass:
                                minLength: 1
                                type: string
                              secondaryDeviceClass:
                                minLength: 1
                                type: string
                            required:
                              - primaryDeviceClass
                              - secondaryDeviceClass
                            type: object
                          replicasPerFailureDomain:
                            minimum: 1
                            type: integer
                          requireSafeReplicaSize:
                            type: boolean
                          size:
                            minimum: 0
                            type: integer
                          subFailureDomain:
                            type: string
                          targetSizeRatio:
                            type: number
                        required:
                          - size
                        type: object
                      statusCheck:
                        properties:
                          mirror:
                            nullable: true
                            properties:
                              disabled:
                                type: boolean
                              interval:
                                type: string
                              timeout:
                                type: string
                            type: object
                        type: object
                        x-kubernetes-preserve-unknown-fields: true
                    type: object
                  nullable: true
                  type: array
                metadataPool:
                  nullable: true
                  properties:
                    compressionMode:
                      enum:
                        - none
                        - passive
                        - aggressive
                        - force
                        - ""
                      nullable: true
                      type: string
                    crushRoot:
                      nullable: true
                      type: string
                    deviceClass:
                      nullable: true
                      type: string
                    enableRBDStats:
                      type: boolean
                    erasureCoded:
                      properties:
                        algorithm:
                          type: string
                        codingChunks:
                          minimum: 0
                          type: integer
                        dataChunks:
                          minimum: 0
                          type: integer
                      required:
                        - codingChunks
                        - dataChunks
                      type: object
                    failureDomain:
                      type: string
                    mirroring:
                      properties:
                        enabled:
                          type: boolean
                        mode:
                          type: string
                        peers:
                          nullable: true
                          properties:
                            secretNames:
                              items:
                                type: string
                              type: array
                          type: object
                        snapshotSchedules:
                          items:
                            properties:
                              interval:
                                type: string
                              path:
                                type: string
                              startTime:
                                type: string
                            type: object
                          type: array
                      type: object
                    parameters:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    quotas:
                      nullable: true
                      properties:
                        maxBytes:
                          format: int64
                          type: integer
                        maxObjects:
                          format: int64
                          type: integer
                        maxSize:
                          pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                          type: string
                      type: object
                    replicated:
                      properties:
                        hybridStorage:
                          nullable: true
                          properties:
                            primaryDeviceClass:
                              minLength: 1
                              type: string
                            secondaryDeviceClass:
                              minLength: 1
                              type: string
                          required:
                            - primaryDeviceClass
                            - secondaryDeviceClass
                          type: object
                        replicasPerFailureDomain:
                          minimum: 1
                          type: integer
                        requireSafeReplicaSize:
                          type: boolean
                        size:
                          minimum: 0
                          type: integer
                        subFailureDomain:
                          type: string
                        targetSizeRatio:
                          type: number
                      required:
                        - size
                      type: object
                    statusCheck:
                      properties:
                        mirror:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  type: object
                metadataServer:
                  properties:
                    activeCount:
                      format: int32
                      maximum: 10
                      minimum: 1
                      type: integer
                    activeStandby:
                      type: boolean
                    annotations:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    labels:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    livenessProbe:
                      properties:
                        disabled:
                          type: boolean
                        probe:
                          properties:
                            exec:
                              properties:
                                command:
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              format: int32
                              type: integer
                            grpc:
                              properties:
                                port:
                                  format: int32
                                  type: integer
                                service:
                                  type: string
                              required:
                                - port
                              type: object
                            httpGet:
                              properties:
                                host:
                                  type: string
                                httpHeaders:
                                  items:
                                    properties:
                                      name:
                                        type: string
                                      value:
                                        type: string
                                    required:
                                      - name
                                      - value
                                    type: object
                                  type: array
                                path:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  type: string
                              required:
                                - port
                              type: object
                            initialDelaySeconds:
                              format: int32
                              type: integer
                            periodSeconds:
                              format: int32
                              type: integer
                            successThreshold:
                              format: int32
                              type: integer
                            tcpSocket:
                              properties:
                                host:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                              required:
                                - port
                              type: object
                            terminationGracePeriodSeconds:
                              format: int64
                              type: integer
                            timeoutSeconds:
                              format: int32
                              type: integer
                          type: object
                      type: object
                    placement:
                      nullable: true
                      properties:
                        nodeAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  preference:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchFields:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - preference
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              properties:
                                nodeSelectorTerms:
                                  items:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchFields:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  type: array
                              required:
                                - nodeSelectorTerms
                              type: object
                              x-kubernetes-map-type: atomic
                          type: object
                        podAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  podAffinityTerm:
                                    properties:
                                      labelSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaceSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaces:
                                        items:
                                          type: string
                                        type: array
                                      topologyKey:
                                        type: string
                                    required:
                                      - topologyKey
                                    type: object
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - podAffinityTerm
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              type: array
                          type: object
                        podAntiAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  podAffinityTerm:
                                    properties:
                                      labelSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaceSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaces:
                                        items:
                                          type: string
                                        type: array
                                      topologyKey:
                                        type: string
                                    required:
                                      - topologyKey
                                    type: object
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - podAffinityTerm
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              type: array
                          type: object
                        tolerations:
                          items:
                            properties:
                              effect:
                                type: string
                              key:
                                type: string
                              operator:
                                type: string
                              tolerationSeconds:
                                format: int64
                                type: integer
                              value:
                                type: string
                            type: object
                          type: array
                        topologySpreadConstraints:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              matchLabelKeys:
                                items:
                                  type: string
                                type: array
                                x-kubernetes-list-type: atomic
                              maxSkew:
                                format: int32
                                type: integer
                              minDomains:
                                format: int32
                                type: integer
                              nodeAffinityPolicy:
                                type: string
                              nodeTaintsPolicy:
                                type: string
                              topologyKey:
                                type: string
                              whenUnsatisfiable:
                                type: string
                            required:
                              - maxSkew
                              - topologyKey
                              - whenUnsatisfiable
                            type: object
                          type: array
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    priorityClassName:
                      type: string
                    resources:
                      nullable: true
                      properties:
                        claims:
                          items:
                            properties:
                              name:
                                type: string
                            required:
                              - name
                            type: object
                          type: array
                          x-kubernetes-list-map-keys:
                            - name
                          x-kubernetes-list-type: map
                        limits:
                          additionalProperties:
                            anyOf:
                              - type: integer
                              - type: string
                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                            x-kubernetes-int-or-string: true
                          type: object
                        requests:
                          additionalProperties:
                            anyOf:
                              - type: integer
                              - type: string
                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                            x-kubernetes-int-or-string: true
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    startupProbe:
                      properties:
                        disabled:
                          type: boolean
                        probe:
                          properties:
                            exec:
                              properties:
                                command:
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              format: int32
                              type: integer
                            grpc:
                              properties:
                                port:
                                  format: int32
                                  type: integer
                                service:
                                  type: string
                              required:
                                - port
                              type: object
                            httpGet:
                              properties:
                                host:
                                  type: string
                                httpHeaders:
                                  items:
                                    properties:
                                      name:
                                        type: string
                                      value:
                                        type: string
                                    required:
                                      - name
                                      - value
                                    type: object
                                  type: array
                                path:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  type: string
                              required:
                                - port
                              type: object
                            initialDelaySeconds:
                              format: int32
                              type: integer
                            periodSeconds:
                              format: int32
                              type: integer
                            successThreshold:
                              format: int32
                              type: integer
                            tcpSocket:
                              properties:
                                host:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                              required:
                                - port
                              type: object
                            terminationGracePeriodSeconds:
                              format: int64
                              type: integer
                            timeoutSeconds:
                              format: int32
                              type: integer
                          type: object
                      type: object
                  required:
                    - activeCount
                  type: object
                mirroring:
                  nullable: true
                  properties:
                    enabled:
                      type: boolean
                    peers:
                      nullable: true
                      properties:
                        secretNames:
                          items:
                            type: string
                          type: array
                      type: object
                    snapshotRetention:
                      items:
                        properties:
                          duration:
                            type: string
                          path:
                            type: string
                        type: object
                      type: array
                    snapshotSchedules:
                      items:
                        properties:
                          interval:
                            type: string
                          path:
                            type: string
                          startTime:
                            type: string
                        type: object
                      type: array
                  type: object
                preserveFilesystemOnDelete:
                  type: boolean
                preservePoolsOnDelete:
                  type: boolean
                statusCheck:
                  properties:
                    mirror:
                      nullable: true
                      properties:
                        disabled:
                          type: boolean
                        interval:
                          type: string
                        timeout:
                          type: string
                      type: object
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
              required:
                - dataPools
                - metadataPool
                - metadataServer
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                mirroringStatus:
                  properties:
                    daemonsStatus:
                      items:
                        properties:
                          daemon_id:
                            type: integer
                          filesystems:
                            items:
                              properties:
                                directory_count:
                                  type: integer
                                filesystem_id:
                                  type: integer
                                name:
                                  type: string
                                peers:
                                  items:
                                    properties:
                                      remote:
                                        properties:
                                          client_name:
                                            type: string
                                          cluster_name:
                                            type: string
                                          fs_name:
                                            type: string
                                        type: object
                                      stats:
                                        properties:
                                          failure_count:
                                            type: integer
                                          recovery_count:
                                            type: integer
                                        type: object
                                      uuid:
                                        type: string
                                    type: object
                                  type: array
                              type: object
                            type: array
                        type: object
                      nullable: true
                      type: array
                    details:
                      type: string
                    lastChanged:
                      type: string
                    lastChecked:
                      type: string
                  type: object
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
                snapshotScheduleStatus:
                  properties:
                    details:
                      type: string
                    lastChanged:
                      type: string
                    lastChecked:
                      type: string
                    snapshotSchedules:
                      items:
                        properties:
                          fs:
                            type: string
                          path:
                            type: string
                          rel_path:
                            type: string
                          retention:
                            properties:
                              active:
                                type: boolean
                              created:
                                type: string
                              created_count:
                                type: integer
                              first:
                                type: string
                              last:
                                type: string
                              last_pruned:
                                type: string
                              pruned_count:
                                type: integer
                              start:
                                type: string
                            type: object
                          schedule:
                            type: string
                          subvol:
                            type: string
                        type: object
                      nullable: true
                      type: array
                  type: object
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephfilesystemsubvolumegroups.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephFilesystemSubVolumeGroup
    listKind: CephFilesystemSubVolumeGroupList
    plural: cephfilesystemsubvolumegroups
    singular: cephfilesystemsubvolumegroup
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                filesystemName:
                  type: string
              required:
                - filesystemName
              type: object
            status:
              properties:
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephnfses.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephNFS
    listKind: CephNFSList
    plural: cephnfses
    shortNames:
      - nfs
    singular: cephnfs
  scope: Namespaced
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                rados:
                  nullable: true
                  properties:
                    namespace:
                      type: string
                    pool:
                      type: string
                  type: object
                security:
                  nullable: true
                  properties:
                    kerberos:
                      nullable: true
                      properties:
                        configFiles:
                          properties:
                            volumeSource:
                              properties:
                                configMap:
                                  properties:
                                    defaultMode:
                                      format: int32
                                      type: integer
                                    items:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          mode:
                                            format: int32
                                            type: integer
                                          path:
                                            type: string
                                        required:
                                          - key
                                          - path
                                        type: object
                                      type: array
                                    name:
                                      type: string
                                    optional:
                                      type: boolean
                                  type: object
                                  x-kubernetes-map-type: atomic
                                emptyDir:
                                  properties:
                                    medium:
                                      type: string
                                    sizeLimit:
                                      anyOf:
                                        - type: integer
                                        - type: string
                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                      x-kubernetes-int-or-string: true
                                  type: object
                                hostPath:
                                  properties:
                                    path:
                                      type: string
                                    type:
                                      type: string
                                  required:
                                    - path
                                  type: object
                                persistentVolumeClaim:
                                  properties:
                                    claimName:
                                      type: string
                                    readOnly:
                                      type: boolean
                                  required:
                                    - claimName
                                  type: object
                                projected:
                                  properties:
                                    defaultMode:
                                      format: int32
                                      type: integer
                                    sources:
                                      items:
                                        properties:
                                          configMap:
                                            properties:
                                              items:
                                                items:
                                                  properties:
                                                    key:
                                                      type: string
                                                    mode:
                                                      format: int32
                                                      type: integer
                                                    path:
                                                      type: string
                                                  required:
                                                    - key
                                                    - path
                                                  type: object
                                                type: array
                                              name:
                                                type: string
                                              optional:
                                                type: boolean
                                            type: object
                                            x-kubernetes-map-type: atomic
                                          downwardAPI:
                                            properties:
                                              items:
                                                items:
                                                  properties:
                                                    fieldRef:
                                                      properties:
                                                        apiVersion:
                                                          type: string
                                                        fieldPath:
                                                          type: string
                                                      required:
                                                        - fieldPath
                                                      type: object
                                                      x-kubernetes-map-type: atomic
                                                    mode:
                                                      format: int32
                                                      type: integer
                                                    path:
                                                      type: string
                                                    resourceFieldRef:
                                                      properties:
                                                        containerName:
                                                          type: string
                                                        divisor:
                                                          anyOf:
                                                            - type: integer
                                                            - type: string
                                                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                                          x-kubernetes-int-or-string: true
                                                        resource:
                                                          type: string
                                                      required:
                                                        - resource
                                                      type: object
                                                      x-kubernetes-map-type: atomic
                                                  required:
                                                    - path
                                                  type: object
                                                type: array
                                            type: object
                                          secret:
                                            properties:
                                              items:
                                                items:
                                                  properties:
                                                    key:
                                                      type: string
                                                    mode:
                                                      format: int32
                                                      type: integer
                                                    path:
                                                      type: string
                                                  required:
                                                    - key
                                                    - path
                                                  type: object
                                                type: array
                                              name:
                                                type: string
                                              optional:
                                                type: boolean
                                            type: object
                                            x-kubernetes-map-type: atomic
                                          serviceAccountToken:
                                            properties:
                                              audience:
                                                type: string
                                              expirationSeconds:
                                                format: int64
                                                type: integer
                                              path:
                                                type: string
                                            required:
                                              - path
                                            type: object
                                        type: object
                                      type: array
                                  type: object
                                secret:
                                  properties:
                                    defaultMode:
                                      format: int32
                                      type: integer
                                    items:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          mode:
                                            format: int32
                                            type: integer
                                          path:
                                            type: string
                                        required:
                                          - key
                                          - path
                                        type: object
                                      type: array
                                    optional:
                                      type: boolean
                                    secretName:
                                      type: string
                                  type: object
                              type: object
                          type: object
                        domainName:
                          type: string
                        keytabFile:
                          properties:
                            volumeSource:
                              properties:
                                configMap:
                                  properties:
                                    defaultMode:
                                      format: int32
                                      type: integer
                                    items:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          mode:
                                            format: int32
                                            type: integer
                                          path:
                                            type: string
                                        required:
                                          - key
                                          - path
                                        type: object
                                      type: array
                                    name:
                                      type: string
                                    optional:
                                      type: boolean
                                  type: object
                                  x-kubernetes-map-type: atomic
                                emptyDir:
                                  properties:
                                    medium:
                                      type: string
                                    sizeLimit:
                                      anyOf:
                                        - type: integer
                                        - type: string
                                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                      x-kubernetes-int-or-string: true
                                  type: object
                                hostPath:
                                  properties:
                                    path:
                                      type: string
                                    type:
                                      type: string
                                  required:
                                    - path
                                  type: object
                                persistentVolumeClaim:
                                  properties:
                                    claimName:
                                      type: string
                                    readOnly:
                                      type: boolean
                                  required:
                                    - claimName
                                  type: object
                                projected:
                                  properties:
                                    defaultMode:
                                      format: int32
                                      type: integer
                                    sources:
                                      items:
                                        properties:
                                          configMap:
                                            properties:
                                              items:
                                                items:
                                                  properties:
                                                    key:
                                                      type: string
                                                    mode:
                                                      format: int32
                                                      type: integer
                                                    path:
                                                      type: string
                                                  required:
                                                    - key
                                                    - path
                                                  type: object
                                                type: array
                                              name:
                                                type: string
                                              optional:
                                                type: boolean
                                            type: object
                                            x-kubernetes-map-type: atomic
                                          downwardAPI:
                                            properties:
                                              items:
                                                items:
                                                  properties:
                                                    fieldRef:
                                                      properties:
                                                        apiVersion:
                                                          type: string
                                                        fieldPath:
                                                          type: string
                                                      required:
                                                        - fieldPath
                                                      type: object
                                                      x-kubernetes-map-type: atomic
                                                    mode:
                                                      format: int32
                                                      type: integer
                                                    path:
                                                      type: string
                                                    resourceFieldRef:
                                                      properties:
                                                        containerName:
                                                          type: string
                                                        divisor:
                                                          anyOf:
                                                            - type: integer
                                                            - type: string
                                                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                                          x-kubernetes-int-or-string: true
                                                        resource:
                                                          type: string
                                                      required:
                                                        - resource
                                                      type: object
                                                      x-kubernetes-map-type: atomic
                                                  required:
                                                    - path
                                                  type: object
                                                type: array
                                            type: object
                                          secret:
                                            properties:
                                              items:
                                                items:
                                                  properties:
                                                    key:
                                                      type: string
                                                    mode:
                                                      format: int32
                                                      type: integer
                                                    path:
                                                      type: string
                                                  required:
                                                    - key
                                                    - path
                                                  type: object
                                                type: array
                                              name:
                                                type: string
                                              optional:
                                                type: boolean
                                            type: object
                                            x-kubernetes-map-type: atomic
                                          serviceAccountToken:
                                            properties:
                                              audience:
                                                type: string
                                              expirationSeconds:
                                                format: int64
                                                type: integer
                                              path:
                                                type: string
                                            required:
                                              - path
                                            type: object
                                        type: object
                                      type: array
                                  type: object
                                secret:
                                  properties:
                                    defaultMode:
                                      format: int32
                                      type: integer
                                    items:
                                      items:
                                        properties:
                                          key:
                                            type: string
                                          mode:
                                            format: int32
                                            type: integer
                                          path:
                                            type: string
                                        required:
                                          - key
                                          - path
                                        type: object
                                      type: array
                                    optional:
                                      type: boolean
                                    secretName:
                                      type: string
                                  type: object
                              type: object
                          type: object
                        principalName:
                          default: nfs
                          type: string
                      type: object
                    sssd:
                      nullable: true
                      properties:
                        sidecar:
                          properties:
                            additionalFiles:
                              items:
                                properties:
                                  subPath:
                                    minLength: 1
                                    pattern: ^[^:]+$
                                    type: string
                                  volumeSource:
                                    properties:
                                      configMap:
                                        properties:
                                          defaultMode:
                                            format: int32
                                            type: integer
                                          items:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                mode:
                                                  format: int32
                                                  type: integer
                                                path:
                                                  type: string
                                              required:
                                                - key
                                                - path
                                              type: object
                                            type: array
                                          name:
                                            type: string
                                          optional:
                                            type: boolean
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      emptyDir:
                                        properties:
                                          medium:
                                            type: string
                                          sizeLimit:
                                            anyOf:
                                              - type: integer
                                              - type: string
                                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                            x-kubernetes-int-or-string: true
                                        type: object
                                      hostPath:
                                        properties:
                                          path:
                                            type: string
                                          type:
                                            type: string
                                        required:
                                          - path
                                        type: object
                                      persistentVolumeClaim:
                                        properties:
                                          claimName:
                                            type: string
                                          readOnly:
                                            type: boolean
                                        required:
                                          - claimName
                                        type: object
                                      projected:
                                        properties:
                                          defaultMode:
                                            format: int32
                                            type: integer
                                          sources:
                                            items:
                                              properties:
                                                configMap:
                                                  properties:
                                                    items:
                                                      items:
                                                        properties:
                                                          key:
                                                            type: string
                                                          mode:
                                                            format: int32
                                                            type: integer
                                                          path:
                                                            type: string
                                                        required:
                                                          - key
                                                          - path
                                                        type: object
                                                      type: array
                                                    name:
                                                      type: string
                                                    optional:
                                                      type: boolean
                                                  type: object
                                                  x-kubernetes-map-type: atomic
                                                downwardAPI:
                                                  properties:
                                                    items:
                                                      items:
                                                        properties:
                                                          fieldRef:
                                                            properties:
                                                              apiVersion:
                                                                type: string
                                                              fieldPath:
                                                                type: string
                                                            required:
                                                              - fieldPath
                                                            type: object
                                                            x-kubernetes-map-type: atomic
                                                          mode:
                                                            format: int32
                                                            type: integer
                                                          path:
                                                            type: string
                                                          resourceFieldRef:
                                                            properties:
                                                              containerName:
                                                                type: string
                                                              divisor:
                                                                anyOf:
                                                                  - type: integer
                                                                  - type: string
                                                                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                                                x-kubernetes-int-or-string: true
                                                              resource:
                                                                type: string
                                                            required:
                                                              - resource
                                                            type: object
                                                            x-kubernetes-map-type: atomic
                                                        required:
                                                          - path
                                                        type: object
                                                      type: array
                                                  type: object
                                                secret:
                                                  properties:
                                                    items:
                                                      items:
                                                        properties:
                                                          key:
                                                            type: string
                                                          mode:
                                                            format: int32
                                                            type: integer
                                                          path:
                                                            type: string
                                                        required:
                                                          - key
                                                          - path
                                                        type: object
                                                      type: array
                                                    name:
                                                      type: string
                                                    optional:
                                                      type: boolean
                                                  type: object
                                                  x-kubernetes-map-type: atomic
                                                serviceAccountToken:
                                                  properties:
                                                    audience:
                                                      type: string
                                                    expirationSeconds:
                                                      format: int64
                                                      type: integer
                                                    path:
                                                      type: string
                                                  required:
                                                    - path
                                                  type: object
                                              type: object
                                            type: array
                                        type: object
                                      secret:
                                        properties:
                                          defaultMode:
                                            format: int32
                                            type: integer
                                          items:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                mode:
                                                  format: int32
                                                  type: integer
                                                path:
                                                  type: string
                                              required:
                                                - key
                                                - path
                                              type: object
                                            type: array
                                          optional:
                                            type: boolean
                                          secretName:
                                            type: string
                                        type: object
                                    type: object
                                required:
                                  - subPath
                                  - volumeSource
                                type: object
                              type: array
                            debugLevel:
                              maximum: 10
                              minimum: 0
                              type: integer
                            image:
                              minLength: 1
                              type: string
                            resources:
                              properties:
                                claims:
                                  items:
                                    properties:
                                      name:
                                        type: string
                                    required:
                                      - name
                                    type: object
                                  type: array
                                  x-kubernetes-list-map-keys:
                                    - name
                                  x-kubernetes-list-type: map
                                limits:
                                  additionalProperties:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                    x-kubernetes-int-or-string: true
                                  type: object
                                requests:
                                  additionalProperties:
                                    anyOf:
                                      - type: integer
                                      - type: string
                                    pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                    x-kubernetes-int-or-string: true
                                  type: object
                              type: object
                            sssdConfigFile:
                              properties:
                                volumeSource:
                                  properties:
                                    configMap:
                                      properties:
                                        defaultMode:
                                          format: int32
                                          type: integer
                                        items:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              mode:
                                                format: int32
                                                type: integer
                                              path:
                                                type: string
                                            required:
                                              - key
                                              - path
                                            type: object
                                          type: array
                                        name:
                                          type: string
                                        optional:
                                          type: boolean
                                      type: object
                                      x-kubernetes-map-type: atomic
                                    emptyDir:
                                      properties:
                                        medium:
                                          type: string
                                        sizeLimit:
                                          anyOf:
                                            - type: integer
                                            - type: string
                                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                          x-kubernetes-int-or-string: true
                                      type: object
                                    hostPath:
                                      properties:
                                        path:
                                          type: string
                                        type:
                                          type: string
                                      required:
                                        - path
                                      type: object
                                    persistentVolumeClaim:
                                      properties:
                                        claimName:
                                          type: string
                                        readOnly:
                                          type: boolean
                                      required:
                                        - claimName
                                      type: object
                                    projected:
                                      properties:
                                        defaultMode:
                                          format: int32
                                          type: integer
                                        sources:
                                          items:
                                            properties:
                                              configMap:
                                                properties:
                                                  items:
                                                    items:
                                                      properties:
                                                        key:
                                                          type: string
                                                        mode:
                                                          format: int32
                                                          type: integer
                                                        path:
                                                          type: string
                                                      required:
                                                        - key
                                                        - path
                                                      type: object
                                                    type: array
                                                  name:
                                                    type: string
                                                  optional:
                                                    type: boolean
                                                type: object
                                                x-kubernetes-map-type: atomic
                                              downwardAPI:
                                                properties:
                                                  items:
                                                    items:
                                                      properties:
                                                        fieldRef:
                                                          properties:
                                                            apiVersion:
                                                              type: string
                                                            fieldPath:
                                                              type: string
                                                          required:
                                                            - fieldPath
                                                          type: object
                                                          x-kubernetes-map-type: atomic
                                                        mode:
                                                          format: int32
                                                          type: integer
                                                        path:
                                                          type: string
                                                        resourceFieldRef:
                                                          properties:
                                                            containerName:
                                                              type: string
                                                            divisor:
                                                              anyOf:
                                                                - type: integer
                                                                - type: string
                                                              pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                                                              x-kubernetes-int-or-string: true
                                                            resource:
                                                              type: string
                                                          required:
                                                            - resource
                                                          type: object
                                                          x-kubernetes-map-type: atomic
                                                      required:
                                                        - path
                                                      type: object
                                                    type: array
                                                type: object
                                              secret:
                                                properties:
                                                  items:
                                                    items:
                                                      properties:
                                                        key:
                                                          type: string
                                                        mode:
                                                          format: int32
                                                          type: integer
                                                        path:
                                                          type: string
                                                      required:
                                                        - key
                                                        - path
                                                      type: object
                                                    type: array
                                                  name:
                                                    type: string
                                                  optional:
                                                    type: boolean
                                                type: object
                                                x-kubernetes-map-type: atomic
                                              serviceAccountToken:
                                                properties:
                                                  audience:
                                                    type: string
                                                  expirationSeconds:
                                                    format: int64
                                                    type: integer
                                                  path:
                                                    type: string
                                                required:
                                                  - path
                                                type: object
                                            type: object
                                          type: array
                                      type: object
                                    secret:
                                      properties:
                                        defaultMode:
                                          format: int32
                                          type: integer
                                        items:
                                          items:
                                            properties:
                                              key:
                                                type: string
                                              mode:
                                                format: int32
                                                type: integer
                                              path:
                                                type: string
                                            required:
                                              - key
                                              - path
                                            type: object
                                          type: array
                                        optional:
                                          type: boolean
                                        secretName:
                                          type: string
                                      type: object
                                  type: object
                              type: object
                          required:
                            - image
                          type: object
                      type: object
                  type: object
                server:
                  properties:
                    active:
                      type: integer
                    annotations:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    hostNetwork:
                      nullable: true
                      type: boolean
                    labels:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    logLevel:
                      type: string
                    placement:
                      nullable: true
                      properties:
                        nodeAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  preference:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchFields:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - preference
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              properties:
                                nodeSelectorTerms:
                                  items:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchFields:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  type: array
                              required:
                                - nodeSelectorTerms
                              type: object
                              x-kubernetes-map-type: atomic
                          type: object
                        podAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  podAffinityTerm:
                                    properties:
                                      labelSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaceSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaces:
                                        items:
                                          type: string
                                        type: array
                                      topologyKey:
                                        type: string
                                    required:
                                      - topologyKey
                                    type: object
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - podAffinityTerm
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              type: array
                          type: object
                        podAntiAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  podAffinityTerm:
                                    properties:
                                      labelSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaceSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaces:
                                        items:
                                          type: string
                                        type: array
                                      topologyKey:
                                        type: string
                                    required:
                                      - topologyKey
                                    type: object
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - podAffinityTerm
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              type: array
                          type: object
                        tolerations:
                          items:
                            properties:
                              effect:
                                type: string
                              key:
                                type: string
                              operator:
                                type: string
                              tolerationSeconds:
                                format: int64
                                type: integer
                              value:
                                type: string
                            type: object
                          type: array
                        topologySpreadConstraints:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              matchLabelKeys:
                                items:
                                  type: string
                                type: array
                                x-kubernetes-list-type: atomic
                              maxSkew:
                                format: int32
                                type: integer
                              minDomains:
                                format: int32
                                type: integer
                              nodeAffinityPolicy:
                                type: string
                              nodeTaintsPolicy:
                                type: string
                              topologyKey:
                                type: string
                              whenUnsatisfiable:
                                type: string
                            required:
                              - maxSkew
                              - topologyKey
                              - whenUnsatisfiable
                            type: object
                          type: array
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    priorityClassName:
                      type: string
                    resources:
                      nullable: true
                      properties:
                        claims:
                          items:
                            properties:
                              name:
                                type: string
                            required:
                              - name
                            type: object
                          type: array
                          x-kubernetes-list-map-keys:
                            - name
                          x-kubernetes-list-type: map
                        limits:
                          additionalProperties:
                            anyOf:
                              - type: integer
                              - type: string
                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                            x-kubernetes-int-or-string: true
                          type: object
                        requests:
                          additionalProperties:
                            anyOf:
                              - type: integer
                              - type: string
                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                            x-kubernetes-int-or-string: true
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  required:
                    - active
                  type: object
              required:
                - server
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephobjectrealms.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephObjectRealm
    listKind: CephObjectRealmList
    plural: cephobjectrealms
    singular: cephobjectrealm
  scope: Namespaced
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              nullable: true
              properties:
                pull:
                  properties:
                    endpoint:
                      pattern: ^https*://
                      type: string
                  type: object
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephobjectstores.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephObjectStore
    listKind: CephObjectStoreList
    plural: cephobjectstores
    singular: cephobjectstore
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                dataPool:
                  nullable: true
                  properties:
                    compressionMode:
                      enum:
                        - none
                        - passive
                        - aggressive
                        - force
                        - ""
                      nullable: true
                      type: string
                    crushRoot:
                      nullable: true
                      type: string
                    deviceClass:
                      nullable: true
                      type: string
                    enableRBDStats:
                      type: boolean
                    erasureCoded:
                      properties:
                        algorithm:
                          type: string
                        codingChunks:
                          minimum: 0
                          type: integer
                        dataChunks:
                          minimum: 0
                          type: integer
                      required:
                        - codingChunks
                        - dataChunks
                      type: object
                    failureDomain:
                      type: string
                    mirroring:
                      properties:
                        enabled:
                          type: boolean
                        mode:
                          type: string
                        peers:
                          nullable: true
                          properties:
                            secretNames:
                              items:
                                type: string
                              type: array
                          type: object
                        snapshotSchedules:
                          items:
                            properties:
                              interval:
                                type: string
                              path:
                                type: string
                              startTime:
                                type: string
                            type: object
                          type: array
                      type: object
                    parameters:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    quotas:
                      nullable: true
                      properties:
                        maxBytes:
                          format: int64
                          type: integer
                        maxObjects:
                          format: int64
                          type: integer
                        maxSize:
                          pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                          type: string
                      type: object
                    replicated:
                      properties:
                        hybridStorage:
                          nullable: true
                          properties:
                            primaryDeviceClass:
                              minLength: 1
                              type: string
                            secondaryDeviceClass:
                              minLength: 1
                              type: string
                          required:
                            - primaryDeviceClass
                            - secondaryDeviceClass
                          type: object
                        replicasPerFailureDomain:
                          minimum: 1
                          type: integer
                        requireSafeReplicaSize:
                          type: boolean
                        size:
                          minimum: 0
                          type: integer
                        subFailureDomain:
                          type: string
                        targetSizeRatio:
                          type: number
                      required:
                        - size
                      type: object
                    statusCheck:
                      properties:
                        mirror:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  type: object
                gateway:
                  nullable: true
                  properties:
                    annotations:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    caBundleRef:
                      nullable: true
                      type: string
                    dashboardEnabled:
                      nullable: true
                      type: boolean
                      x-kubernetes-preserve-unknown-fields: true
                    externalRgwEndpoints:
                      items:
                        properties:
                          hostname:
                            type: string
                          ip:
                            type: string
                        type: object
                        x-kubernetes-map-type: atomic
                      nullable: true
                      type: array
                    hostNetwork:
                      nullable: true
                      type: boolean
                      x-kubernetes-preserve-unknown-fields: true
                    instances:
                      format: int32
                      nullable: true
                      type: integer
                    labels:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    placement:
                      nullable: true
                      properties:
                        nodeAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  preference:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchFields:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - preference
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              properties:
                                nodeSelectorTerms:
                                  items:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchFields:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  type: array
                              required:
                                - nodeSelectorTerms
                              type: object
                              x-kubernetes-map-type: atomic
                          type: object
                        podAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  podAffinityTerm:
                                    properties:
                                      labelSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaceSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaces:
                                        items:
                                          type: string
                                        type: array
                                      topologyKey:
                                        type: string
                                    required:
                                      - topologyKey
                                    type: object
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - podAffinityTerm
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              type: array
                          type: object
                        podAntiAffinity:
                          properties:
                            preferredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  podAffinityTerm:
                                    properties:
                                      labelSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaceSelector:
                                        properties:
                                          matchExpressions:
                                            items:
                                              properties:
                                                key:
                                                  type: string
                                                operator:
                                                  type: string
                                                values:
                                                  items:
                                                    type: string
                                                  type: array
                                              required:
                                                - key
                                                - operator
                                              type: object
                                            type: array
                                          matchLabels:
                                            additionalProperties:
                                              type: string
                                            type: object
                                        type: object
                                        x-kubernetes-map-type: atomic
                                      namespaces:
                                        items:
                                          type: string
                                        type: array
                                      topologyKey:
                                        type: string
                                    required:
                                      - topologyKey
                                    type: object
                                  weight:
                                    format: int32
                                    type: integer
                                required:
                                  - podAffinityTerm
                                  - weight
                                type: object
                              type: array
                            requiredDuringSchedulingIgnoredDuringExecution:
                              items:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              type: array
                          type: object
                        tolerations:
                          items:
                            properties:
                              effect:
                                type: string
                              key:
                                type: string
                              operator:
                                type: string
                              tolerationSeconds:
                                format: int64
                                type: integer
                              value:
                                type: string
                            type: object
                          type: array
                        topologySpreadConstraints:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              matchLabelKeys:
                                items:
                                  type: string
                                type: array
                                x-kubernetes-list-type: atomic
                              maxSkew:
                                format: int32
                                type: integer
                              minDomains:
                                format: int32
                                type: integer
                              nodeAffinityPolicy:
                                type: string
                              nodeTaintsPolicy:
                                type: string
                              topologyKey:
                                type: string
                              whenUnsatisfiable:
                                type: string
                            required:
                              - maxSkew
                              - topologyKey
                              - whenUnsatisfiable
                            type: object
                          type: array
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    port:
                      format: int32
                      type: integer
                    priorityClassName:
                      type: string
                    resources:
                      nullable: true
                      properties:
                        claims:
                          items:
                            properties:
                              name:
                                type: string
                            required:
                              - name
                            type: object
                          type: array
                          x-kubernetes-list-map-keys:
                            - name
                          x-kubernetes-list-type: map
                        limits:
                          additionalProperties:
                            anyOf:
                              - type: integer
                              - type: string
                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                            x-kubernetes-int-or-string: true
                          type: object
                        requests:
                          additionalProperties:
                            anyOf:
                              - type: integer
                              - type: string
                            pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                            x-kubernetes-int-or-string: true
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    securePort:
                      format: int32
                      maximum: 65535
                      minimum: 0
                      nullable: true
                      type: integer
                    service:
                      nullable: true
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    sslCertificateRef:
                      nullable: true
                      type: string
                  type: object
                healthCheck:
                  nullable: true
                  properties:
                    readinessProbe:
                      properties:
                        disabled:
                          type: boolean
                        probe:
                          properties:
                            exec:
                              properties:
                                command:
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              format: int32
                              type: integer
                            grpc:
                              properties:
                                port:
                                  format: int32
                                  type: integer
                                service:
                                  type: string
                              required:
                                - port
                              type: object
                            httpGet:
                              properties:
                                host:
                                  type: string
                                httpHeaders:
                                  items:
                                    properties:
                                      name:
                                        type: string
                                      value:
                                        type: string
                                    required:
                                      - name
                                      - value
                                    type: object
                                  type: array
                                path:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  type: string
                              required:
                                - port
                              type: object
                            initialDelaySeconds:
                              format: int32
                              type: integer
                            periodSeconds:
                              format: int32
                              type: integer
                            successThreshold:
                              format: int32
                              type: integer
                            tcpSocket:
                              properties:
                                host:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                              required:
                                - port
                              type: object
                            terminationGracePeriodSeconds:
                              format: int64
                              type: integer
                            timeoutSeconds:
                              format: int32
                              type: integer
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    startupProbe:
                      properties:
                        disabled:
                          type: boolean
                        probe:
                          properties:
                            exec:
                              properties:
                                command:
                                  items:
                                    type: string
                                  type: array
                              type: object
                            failureThreshold:
                              format: int32
                              type: integer
                            grpc:
                              properties:
                                port:
                                  format: int32
                                  type: integer
                                service:
                                  type: string
                              required:
                                - port
                              type: object
                            httpGet:
                              properties:
                                host:
                                  type: string
                                httpHeaders:
                                  items:
                                    properties:
                                      name:
                                        type: string
                                      value:
                                        type: string
                                    required:
                                      - name
                                      - value
                                    type: object
                                  type: array
                                path:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                                scheme:
                                  type: string
                              required:
                                - port
                              type: object
                            initialDelaySeconds:
                              format: int32
                              type: integer
                            periodSeconds:
                              format: int32
                              type: integer
                            successThreshold:
                              format: int32
                              type: integer
                            tcpSocket:
                              properties:
                                host:
                                  type: string
                                port:
                                  anyOf:
                                    - type: integer
                                    - type: string
                                  x-kubernetes-int-or-string: true
                              required:
                                - port
                              type: object
                            terminationGracePeriodSeconds:
                              format: int64
                              type: integer
                            timeoutSeconds:
                              format: int32
                              type: integer
                          type: object
                      type: object
                  type: object
                metadataPool:
                  nullable: true
                  properties:
                    compressionMode:
                      enum:
                        - none
                        - passive
                        - aggressive
                        - force
                        - ""
                      nullable: true
                      type: string
                    crushRoot:
                      nullable: true
                      type: string
                    deviceClass:
                      nullable: true
                      type: string
                    enableRBDStats:
                      type: boolean
                    erasureCoded:
                      properties:
                        algorithm:
                          type: string
                        codingChunks:
                          minimum: 0
                          type: integer
                        dataChunks:
                          minimum: 0
                          type: integer
                      required:
                        - codingChunks
                        - dataChunks
                      type: object
                    failureDomain:
                      type: string
                    mirroring:
                      properties:
                        enabled:
                          type: boolean
                        mode:
                          type: string
                        peers:
                          nullable: true
                          properties:
                            secretNames:
                              items:
                                type: string
                              type: array
                          type: object
                        snapshotSchedules:
                          items:
                            properties:
                              interval:
                                type: string
                              path:
                                type: string
                              startTime:
                                type: string
                            type: object
                          type: array
                      type: object
                    parameters:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    quotas:
                      nullable: true
                      properties:
                        maxBytes:
                          format: int64
                          type: integer
                        maxObjects:
                          format: int64
                          type: integer
                        maxSize:
                          pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                          type: string
                      type: object
                    replicated:
                      properties:
                        hybridStorage:
                          nullable: true
                          properties:
                            primaryDeviceClass:
                              minLength: 1
                              type: string
                            secondaryDeviceClass:
                              minLength: 1
                              type: string
                          required:
                            - primaryDeviceClass
                            - secondaryDeviceClass
                          type: object
                        replicasPerFailureDomain:
                          minimum: 1
                          type: integer
                        requireSafeReplicaSize:
                          type: boolean
                        size:
                          minimum: 0
                          type: integer
                        subFailureDomain:
                          type: string
                        targetSizeRatio:
                          type: number
                      required:
                        - size
                      type: object
                    statusCheck:
                      properties:
                        mirror:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  type: object
                preservePoolsOnDelete:
                  type: boolean
                security:
                  nullable: true
                  properties:
                    keyRotation:
                      nullable: true
                      properties:
                        enabled:
                          default: false
                          type: boolean
                        schedule:
                          type: string
                      type: object
                    kms:
                      nullable: true
                      properties:
                        connectionDetails:
                          additionalProperties:
                            type: string
                          nullable: true
                          type: object
                          x-kubernetes-preserve-unknown-fields: true
                        tokenSecretName:
                          type: string
                      type: object
                    s3:
                      nullable: true
                      properties:
                        connectionDetails:
                          additionalProperties:
                            type: string
                          nullable: true
                          type: object
                          x-kubernetes-preserve-unknown-fields: true
                        tokenSecretName:
                          type: string
                      type: object
                  type: object
                zone:
                  nullable: true
                  properties:
                    name:
                      type: string
                  required:
                    - name
                  type: object
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                endpoints:
                  properties:
                    insecure:
                      items:
                        type: string
                      nullable: true
                      type: array
                    secure:
                      items:
                        type: string
                      nullable: true
                      type: array
                  type: object
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                message:
                  type: string
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephobjectstoreusers.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephObjectStoreUser
    listKind: CephObjectStoreUserList
    plural: cephobjectstoreusers
    shortNames:
      - rcou
      - objectuser
    singular: cephobjectstoreuser
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                capabilities:
                  nullable: true
                  properties:
                    amz-cache:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    bilog:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    bucket:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    datalog:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    info:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    mdlog:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    metadata:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    oidc-provider:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    ratelimit:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    roles:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    usage:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    user:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    user-policy:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                    zone:
                      enum:
                        - '*'
                        - read
                        - write
                        - read, write
                      type: string
                  type: object
                displayName:
                  type: string
                quotas:
                  nullable: true
                  properties:
                    maxBuckets:
                      nullable: true
                      type: integer
                    maxObjects:
                      format: int64
                      nullable: true
                      type: integer
                    maxSize:
                      anyOf:
                        - type: integer
                        - type: string
                      nullable: true
                      pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                      x-kubernetes-int-or-string: true
                  type: object
                store:
                  type: string
              type: object
            status:
              properties:
                info:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephobjectzonegroups.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephObjectZoneGroup
    listKind: CephObjectZoneGroupList
    plural: cephobjectzonegroups
    singular: cephobjectzonegroup
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                realm:
                  type: string
              required:
                - realm
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephobjectzones.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephObjectZone
    listKind: CephObjectZoneList
    plural: cephobjectzones
    singular: cephobjectzone
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                customEndpoints:
                  items:
                    type: string
                  nullable: true
                  type: array
                dataPool:
                  nullable: true
                  properties:
                    compressionMode:
                      enum:
                        - none
                        - passive
                        - aggressive
                        - force
                        - ""
                      nullable: true
                      type: string
                    crushRoot:
                      nullable: true
                      type: string
                    deviceClass:
                      nullable: true
                      type: string
                    enableRBDStats:
                      type: boolean
                    erasureCoded:
                      properties:
                        algorithm:
                          type: string
                        codingChunks:
                          minimum: 0
                          type: integer
                        dataChunks:
                          minimum: 0
                          type: integer
                      required:
                        - codingChunks
                        - dataChunks
                      type: object
                    failureDomain:
                      type: string
                    mirroring:
                      properties:
                        enabled:
                          type: boolean
                        mode:
                          type: string
                        peers:
                          nullable: true
                          properties:
                            secretNames:
                              items:
                                type: string
                              type: array
                          type: object
                        snapshotSchedules:
                          items:
                            properties:
                              interval:
                                type: string
                              path:
                                type: string
                              startTime:
                                type: string
                            type: object
                          type: array
                      type: object
                    parameters:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    quotas:
                      nullable: true
                      properties:
                        maxBytes:
                          format: int64
                          type: integer
                        maxObjects:
                          format: int64
                          type: integer
                        maxSize:
                          pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                          type: string
                      type: object
                    replicated:
                      properties:
                        hybridStorage:
                          nullable: true
                          properties:
                            primaryDeviceClass:
                              minLength: 1
                              type: string
                            secondaryDeviceClass:
                              minLength: 1
                              type: string
                          required:
                            - primaryDeviceClass
                            - secondaryDeviceClass
                          type: object
                        replicasPerFailureDomain:
                          minimum: 1
                          type: integer
                        requireSafeReplicaSize:
                          type: boolean
                        size:
                          minimum: 0
                          type: integer
                        subFailureDomain:
                          type: string
                        targetSizeRatio:
                          type: number
                      required:
                        - size
                      type: object
                    statusCheck:
                      properties:
                        mirror:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  type: object
                metadataPool:
                  nullable: true
                  properties:
                    compressionMode:
                      enum:
                        - none
                        - passive
                        - aggressive
                        - force
                        - ""
                      nullable: true
                      type: string
                    crushRoot:
                      nullable: true
                      type: string
                    deviceClass:
                      nullable: true
                      type: string
                    enableRBDStats:
                      type: boolean
                    erasureCoded:
                      properties:
                        algorithm:
                          type: string
                        codingChunks:
                          minimum: 0
                          type: integer
                        dataChunks:
                          minimum: 0
                          type: integer
                      required:
                        - codingChunks
                        - dataChunks
                      type: object
                    failureDomain:
                      type: string
                    mirroring:
                      properties:
                        enabled:
                          type: boolean
                        mode:
                          type: string
                        peers:
                          nullable: true
                          properties:
                            secretNames:
                              items:
                                type: string
                              type: array
                          type: object
                        snapshotSchedules:
                          items:
                            properties:
                              interval:
                                type: string
                              path:
                                type: string
                              startTime:
                                type: string
                            type: object
                          type: array
                      type: object
                    parameters:
                      additionalProperties:
                        type: string
                      nullable: true
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    quotas:
                      nullable: true
                      properties:
                        maxBytes:
                          format: int64
                          type: integer
                        maxObjects:
                          format: int64
                          type: integer
                        maxSize:
                          pattern: ^[0-9]+[\.]?[0-9]*([KMGTPE]i|[kMGTPE])?$
                          type: string
                      type: object
                    replicated:
                      properties:
                        hybridStorage:
                          nullable: true
                          properties:
                            primaryDeviceClass:
                              minLength: 1
                              type: string
                            secondaryDeviceClass:
                              minLength: 1
                              type: string
                          required:
                            - primaryDeviceClass
                            - secondaryDeviceClass
                          type: object
                        replicasPerFailureDomain:
                          minimum: 1
                          type: integer
                        requireSafeReplicaSize:
                          type: boolean
                        size:
                          minimum: 0
                          type: integer
                        subFailureDomain:
                          type: string
                        targetSizeRatio:
                          type: number
                      required:
                        - size
                      type: object
                    statusCheck:
                      properties:
                        mirror:
                          nullable: true
                          properties:
                            disabled:
                              type: boolean
                            interval:
                              type: string
                            timeout:
                              type: string
                          type: object
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                  type: object
                preservePoolsOnDelete:
                  default: true
                  type: boolean
                zoneGroup:
                  type: string
              required:
                - dataPool
                - metadataPool
                - zoneGroup
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.11.3
  creationTimestamp: null
  name: cephrbdmirrors.ceph.rook.io
spec:
  group: ceph.rook.io
  names:
    kind: CephRBDMirror
    listKind: CephRBDMirrorList
    plural: cephrbdmirrors
    singular: cephrbdmirror
  scope: Namespaced
  versions:
    - additionalPrinterColumns:
        - jsonPath: .status.phase
          name: Phase
          type: string
      name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            spec:
              properties:
                annotations:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                count:
                  minimum: 1
                  type: integer
                labels:
                  additionalProperties:
                    type: string
                  nullable: true
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                peers:
                  nullable: true
                  properties:
                    secretNames:
                      items:
                        type: string
                      type: array
                  type: object
                placement:
                  nullable: true
                  properties:
                    nodeAffinity:
                      properties:
                        preferredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              preference:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchFields:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                type: object
                                x-kubernetes-map-type: atomic
                              weight:
                                format: int32
                                type: integer
                            required:
                              - preference
                              - weight
                            type: object
                          type: array
                        requiredDuringSchedulingIgnoredDuringExecution:
                          properties:
                            nodeSelectorTerms:
                              items:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchFields:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                type: object
                                x-kubernetes-map-type: atomic
                              type: array
                          required:
                            - nodeSelectorTerms
                          type: object
                          x-kubernetes-map-type: atomic
                      type: object
                    podAffinity:
                      properties:
                        preferredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              podAffinityTerm:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              weight:
                                format: int32
                                type: integer
                            required:
                              - podAffinityTerm
                              - weight
                            type: object
                          type: array
                        requiredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaceSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaces:
                                items:
                                  type: string
                                type: array
                              topologyKey:
                                type: string
                            required:
                              - topologyKey
                            type: object
                          type: array
                      type: object
                    podAntiAffinity:
                      properties:
                        preferredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              podAffinityTerm:
                                properties:
                                  labelSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaceSelector:
                                    properties:
                                      matchExpressions:
                                        items:
                                          properties:
                                            key:
                                              type: string
                                            operator:
                                              type: string
                                            values:
                                              items:
                                                type: string
                                              type: array
                                          required:
                                            - key
                                            - operator
                                          type: object
                                        type: array
                                      matchLabels:
                                        additionalProperties:
                                          type: string
                                        type: object
                                    type: object
                                    x-kubernetes-map-type: atomic
                                  namespaces:
                                    items:
                                      type: string
                                    type: array
                                  topologyKey:
                                    type: string
                                required:
                                  - topologyKey
                                type: object
                              weight:
                                format: int32
                                type: integer
                            required:
                              - podAffinityTerm
                              - weight
                            type: object
                          type: array
                        requiredDuringSchedulingIgnoredDuringExecution:
                          items:
                            properties:
                              labelSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaceSelector:
                                properties:
                                  matchExpressions:
                                    items:
                                      properties:
                                        key:
                                          type: string
                                        operator:
                                          type: string
                                        values:
                                          items:
                                            type: string
                                          type: array
                                      required:
                                        - key
                                        - operator
                                      type: object
                                    type: array
                                  matchLabels:
                                    additionalProperties:
                                      type: string
                                    type: object
                                type: object
                                x-kubernetes-map-type: atomic
                              namespaces:
                                items:
                                  type: string
                                type: array
                              topologyKey:
                                type: string
                            required:
                              - topologyKey
                            type: object
                          type: array
                      type: object
                    tolerations:
                      items:
                        properties:
                          effect:
                            type: string
                          key:
                            type: string
                          operator:
                            type: string
                          tolerationSeconds:
                            format: int64
                            type: integer
                          value:
                            type: string
                        type: object
                      type: array
                    topologySpreadConstraints:
                      items:
                        properties:
                          labelSelector:
                            properties:
                              matchExpressions:
                                items:
                                  properties:
                                    key:
                                      type: string
                                    operator:
                                      type: string
                                    values:
                                      items:
                                        type: string
                                      type: array
                                  required:
                                    - key
                                    - operator
                                  type: object
                                type: array
                              matchLabels:
                                additionalProperties:
                                  type: string
                                type: object
                            type: object
                            x-kubernetes-map-type: atomic
                          matchLabelKeys:
                            items:
                              type: string
                            type: array
                            x-kubernetes-list-type: atomic
                          maxSkew:
                            format: int32
                            type: integer
                          minDomains:
                            format: int32
                            type: integer
                          nodeAffinityPolicy:
                            type: string
                          nodeTaintsPolicy:
                            type: string
                          topologyKey:
                            type: string
                          whenUnsatisfiable:
                            type: string
                        required:
                          - maxSkew
                          - topologyKey
                          - whenUnsatisfiable
                        type: object
                      type: array
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
                priorityClassName:
                  type: string
                resources:
                  nullable: true
                  properties:
                    claims:
                      items:
                        properties:
                          name:
                            type: string
                        required:
                          - name
                        type: object
                      type: array
                      x-kubernetes-list-map-keys:
                        - name
                      x-kubernetes-list-type: map
                    limits:
                      additionalProperties:
                        anyOf:
                          - type: integer
                          - type: string
                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                        x-kubernetes-int-or-string: true
                      type: object
                    requests:
                      additionalProperties:
                        anyOf:
                          - type: integer
                          - type: string
                        pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                        x-kubernetes-int-or-string: true
                      type: object
                  type: object
                  x-kubernetes-preserve-unknown-fields: true
              required:
                - count
              type: object
            status:
              properties:
                conditions:
                  items:
                    properties:
                      lastHeartbeatTime:
                        format: date-time
                        type: string
                      lastTransitionTime:
                        format: date-time
                        type: string
                      message:
                        type: string
                      reason:
                        type: string
                      status:
                        type: string
                      type:
                        type: string
                    type: object
                  type: array
                observedGeneration:
                  format: int64
                  type: integer
                phase:
                  type: string
              type: object
              x-kubernetes-preserve-unknown-fields: true
          required:
            - metadata
            - spec
          type: object
      served: true
      storage: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: objectbucketclaims.objectbucket.io
spec:
  group: objectbucket.io
  names:
    kind: ObjectBucketClaim
    listKind: ObjectBucketClaimList
    plural: objectbucketclaims
    singular: objectbucketclaim
    shortNames:
      - obc
      - obcs
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                storageClassName:
                  type: string
                bucketName:
                  type: string
                generateBucketName:
                  type: string
                additionalConfig:
                  type: object
                  nullable: true
                  x-kubernetes-preserve-unknown-fields: true
                objectBucketName:
                  type: string
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      subresources:
        status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: objectbuckets.objectbucket.io
spec:
  group: objectbucket.io
  names:
    kind: ObjectBucket
    listKind: ObjectBucketList
    plural: objectbuckets
    singular: objectbucket
    shortNames:
      - ob
      - obs
  scope: Cluster
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                storageClassName:
                  type: string
                endpoint:
                  type: object
                  nullable: true
                  properties:
                    bucketHost:
                      type: string
                    bucketPort:
                      type: integer
                      format: int32
                    bucketName:
                      type: string
                    region:
                      type: string
                    subRegion:
                      type: string
                    additionalConfig:
                      type: object
                      nullable: true
                      x-kubernetes-preserve-unknown-fields: true
                authentication:
                  type: object
                  nullable: true
                  items:
                    type: object
                    x-kubernetes-preserve-unknown-fields: true
                additionalState:
                  type: object
                  nullable: true
                  x-kubernetes-preserve-unknown-fields: true
                reclaimPolicy:
                  type: string
                claimRef:
                  type: object
                  nullable: true
                  x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      subresources:
        status: {}

创建启动Operator和ceph集群所需的公共资源

# Namespace where the operator and other rook resources are created
apiVersion: v1
kind: Namespace
metadata:
  name: rook-ceph # namespace:cluster
---
# TODO: remove this, once https://github.com/rook/rook/issues/10141
# is resolved.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-csi-nodeplugin
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "patch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list", "watch", "patch", "update"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents/status"]
    verbs: ["update", "patch"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list", "watch", "patch", "update"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents/status"]
    verbs: ["update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
---
# The cluster role for managing all the cluster-specific resources in a namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rook-ceph-cluster-mgmt
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  - apiGroups:
      - ""
      - apps
      - extensions
    resources:
      - secrets
      - pods
      - pods/log
      - services
      - configmaps
      - deployments
      - daemonsets
    verbs:
      - get
      - list
      - watch
      - patch
      - create
      - update
      - delete
---
# The cluster role for managing the Rook CRDs
apiVersion: rbac.authorization.k8s.io/v1
# Rook watches for its CRDs in all namespaces, so this should be a cluster-scoped role unless the
# operator config `ROOK_CURRENT_NAMESPACE_ONLY=true`.
kind: ClusterRole
metadata:
  name: rook-ceph-global
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  - apiGroups:
      - ""
    resources:
      # Pod access is needed for fencing
      - pods
      # Node access is needed for determining nodes where mons should run
      - nodes
      - nodes/proxy
      - services
      # Rook watches secrets which it uses to configure access to external resources.
      # e.g., external Ceph cluster; TLS certificates for the admission controller or object store
      - secrets
      # Rook watches for changes to the rook-operator-config configmap
      - configmaps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      # Rook creates events for its custom resources
      - events
      # Rook creates PVs and PVCs for OSDs managed by the Rook provisioner
      - persistentvolumes
      - persistentvolumeclaims
      # Rook creates endpoints for mgr and object store access
      - endpoints
    verbs:
      - get
      - list
      - watch
      - patch
      - create
      - update
      - delete
  - apiGroups:
      - storage.k8s.io
    resources:
      - storageclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - batch
    resources:
      - jobs
      - cronjobs
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
      - deletecollection
  # The Rook operator must be able to watch all ceph.rook.io resources to reconcile them.
  - apiGroups: ["ceph.rook.io"]
    resources:
      - cephclients
      - cephclusters
      - cephblockpools
      - cephfilesystems
      - cephnfses
      - cephobjectstores
      - cephobjectstoreusers
      - cephobjectrealms
      - cephobjectzonegroups
      - cephobjectzones
      - cephbuckettopics
      - cephbucketnotifications
      - cephrbdmirrors
      - cephfilesystemmirrors
      - cephfilesystemsubvolumegroups
      - cephblockpoolradosnamespaces
    verbs:
      - get
      - list
      - watch
      # Ideally the update permission is not required, but Rook needs it to add finalizers to resources.
      - update
  # Rook must have update access to status subresources for its custom resources.
  - apiGroups: ["ceph.rook.io"]
    resources:
      - cephclients/status
      - cephclusters/status
      - cephblockpools/status
      - cephfilesystems/status
      - cephnfses/status
      - cephobjectstores/status
      - cephobjectstoreusers/status
      - cephobjectrealms/status
      - cephobjectzonegroups/status
      - cephobjectzones/status
      - cephbuckettopics/status
      - cephbucketnotifications/status
      - cephrbdmirrors/status
      - cephfilesystemmirrors/status
      - cephfilesystemsubvolumegroups/status
      - cephblockpoolradosnamespaces/status
    verbs: ["update"]
  # The "*/finalizers" permission may need to be strictly given for K8s clusters where
  # OwnerReferencesPermissionEnforcement is enabled so that Rook can set blockOwnerDeletion on
  # resources owned by Rook CRs (e.g., a Secret owned by an OSD Deployment). See more:
  # https://kubernetes.io/docs/reference/access-authn-authz/_print/#ownerreferencespermissionenforcement
  - apiGroups: ["ceph.rook.io"]
    resources:
      - cephclients/finalizers
      - cephclusters/finalizers
      - cephblockpools/finalizers
      - cephfilesystems/finalizers
      - cephnfses/finalizers
      - cephobjectstores/finalizers
      - cephobjectstoreusers/finalizers
      - cephobjectrealms/finalizers
      - cephobjectzonegroups/finalizers
      - cephobjectzones/finalizers
      - cephbuckettopics/finalizers
      - cephbucketnotifications/finalizers
      - cephrbdmirrors/finalizers
      - cephfilesystemmirrors/finalizers
      - cephfilesystemsubvolumegroups/finalizers
      - cephblockpoolradosnamespaces/finalizers
    verbs: ["update"]
  - apiGroups:
      - policy
      - apps
      - extensions
    resources:
      # This is for the clusterdisruption controller
      - poddisruptionbudgets
      # This is for both clusterdisruption and nodedrain controllers
      - deployments
      - replicasets
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
      - deletecollection
  - apiGroups:
      - apps
    resources:
      # This is to add osd deployment owner ref on key rotation
      # cron jobs.
      - deployments/finalizers
    verbs:
      - update
  - apiGroups:
      - healthchecking.openshift.io
    resources:
      - machinedisruptionbudgets
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
  - apiGroups:
      - machine.openshift.io
    resources:
      - machines
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
  - apiGroups:
      - storage.k8s.io
    resources:
      - csidrivers
    verbs:
      - create
      - delete
      - get
      - update
  - apiGroups:
      - k8s.cni.cncf.io
    resources:
      - network-attachment-definitions
    verbs:
      - get
---
# Aspects of ceph-mgr that require cluster-wide access
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-mgr-cluster
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - nodes
      - nodes/proxy
      - persistentvolumes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
      - list
      - get
      - watch
  - apiGroups:
      - storage.k8s.io
    resources:
      - storageclasses
    verbs:
      - get
      - list
      - watch
---
# Aspects of ceph-mgr that require access to the system namespace
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-mgr-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - list
      - watch
---
# Used for provisioning ObjectBuckets (OBs) in response to ObjectBucketClaims (OBCs).
# Note: Rook runs a copy of the lib-bucket-provisioner's OBC controller.
# OBCs can be created in any Kubernetes namespace, so this must be a cluster-scoped role.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-object-bucket
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    verbs:
      # OBC controller creates secrets and configmaps containing information for users about how to
      # connect to object buckets. It deletes them when an OBC is deleted.
      - get
      - create
      - update
      - delete
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs:
      # OBC controller gets parameters from the OBC's storageclass
      # Rook gets additional parameters from the OBC's storageclass
      - get
  - apiGroups: ["objectbucket.io"]
    resources: ["objectbucketclaims"]
    verbs:
      # OBC controller needs to list/watch OBCs and get latest version of a reconciled OBC
      - list
      - watch
      - get
      # Ideally, update should not be needed, but the OBC controller updates the OBC with bucket
      # information outside of the status subresource
      - update
      # OBC controller does not delete OBCs; users do this
  - apiGroups: ["objectbucket.io"]
    resources: ["objectbuckets"]
    verbs:
      # OBC controller needs to list/watch OBs and get latest version of a reconciled OB
      - list
      - watch
      - get
      # OBC controller creates an OB when an OBC's bucket has been provisioned by Ceph, updates them
      # when an OBC is updated, and deletes them when the OBC is de-provisioned.
      - create
      - update
      - delete
  - apiGroups: ["objectbucket.io"]
    resources: ["objectbucketclaims/status", "objectbuckets/status"]
    verbs:
      # OBC controller updates OBC and OB statuses
      - update
  - apiGroups: ["objectbucket.io"]
    # This does not strictly allow the OBC/OB controllers to update finalizers. That is handled by
    # the direct "update" permissions above. Instead, this allows Rook's controller to create
    # resources which are owned by OBs/OBCs and where blockOwnerDeletion is set.
    resources: ["objectbucketclaims/finalizers", "objectbuckets/finalizers"]
    verbs:
      - update
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-osd
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-system
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  # Most resources are represented by a string representation of their name, such as "pods", just as it appears in the URL for the relevant API endpoint.
  # However, some Kubernetes APIs involve a "subresource", such as the logs for a pod. [...]
  # To represent this in an RBAC role, use a slash to delimit the resource and subresource.
  # https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]
  - apiGroups: ["admissionregistration.k8s.io"]
    resources: ["validatingwebhookconfigurations"]
    verbs: ["create", "get", "delete", "update"]
---
# This is required by operator-sdk to map the cluster/clusterrolebindings with SA
# otherwise operator-sdk will create a individual file for these.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-csi-nodeplugin-role
subjects:
  - kind: ServiceAccount
    name: rook-csi-cephfs-plugin-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: ClusterRole
  name: cephfs-csi-nodeplugin
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-csi-provisioner-role
subjects:
  - kind: ServiceAccount
    name: rook-csi-cephfs-provisioner-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: ClusterRole
  name: cephfs-external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin
subjects:
  - kind: ServiceAccount
    name: rook-csi-rbd-plugin-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: ClusterRole
  name: rbd-csi-nodeplugin
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-provisioner-role
subjects:
  - kind: ServiceAccount
    name: rook-csi-rbd-provisioner-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: ClusterRole
  name: rbd-external-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
# Grant the rook system daemons cluster-wide access to manage the Rook CRDs, PVCs, and storage classes
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-global
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-global
subjects:
  - kind: ServiceAccount
    name: rook-ceph-system
    namespace: rook-ceph # namespace:operator
---
# Allow the ceph mgr to access cluster-wide resources necessary for the mgr modules
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-mgr-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-mgr-cluster
subjects:
  - kind: ServiceAccount
    name: rook-ceph-mgr
    namespace: rook-ceph # namespace:cluster
---
kind: ClusterRoleBinding
# Give Rook-Ceph Operator permissions to provision ObjectBuckets in response to ObjectBucketClaims.
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-object-bucket
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-object-bucket
subjects:
  - kind: ServiceAccount
    name: rook-ceph-system
    namespace: rook-ceph # namespace:operator
---
# Allow the ceph osd to access cluster-wide resources necessary for determining their topology location
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-osd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-osd
subjects:
  - kind: ServiceAccount
    name: rook-ceph-osd
    namespace: rook-ceph # namespace:cluster
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-system
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-system
subjects:
  - kind: ServiceAccount
    name: rook-ceph-system
    namespace: rook-ceph # namespace:operator
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-external-provisioner-cfg
  namespace: rook-ceph # namespace:operator
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin
  namespace: rook-ceph # namespace:operator
rules:
  - apiGroups: ["csiaddons.openshift.io"]
    resources: ["csiaddonsnodes"]
    verbs: ["create"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-external-provisioner-cfg
  namespace: rook-ceph # namespace:operator
rules:
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
  - apiGroups: ["csiaddons.openshift.io"]
    resources: ["csiaddonsnodes"]
    verbs: ["create"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-cmd-reporter
  namespace: rook-ceph # namespace:cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
---
# Aspects of ceph-mgr that operate within the cluster's namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph # namespace:cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - services
      - pods/log
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
  - apiGroups:
      - ceph.rook.io
    resources:
      - cephclients
      - cephclusters
      - cephblockpools
      - cephfilesystems
      - cephnfses
      - cephobjectstores
      - cephobjectstoreusers
      - cephobjectrealms
      - cephobjectzonegroups
      - cephobjectzones
      - cephbuckettopics
      - cephbucketnotifications
      - cephrbdmirrors
      - cephfilesystemmirrors
      - cephfilesystemsubvolumegroups
      - cephblockpoolradosnamespaces
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
      - patch
  - apiGroups:
      - apps
    resources:
      - deployments/scale
      - deployments
    verbs:
      - patch
      - delete
  - apiGroups:
      - ''
    resources:
      - persistentvolumeclaims
    verbs:
      - delete
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-osd
  namespace: rook-ceph # namespace:cluster
rules:
  # this is needed for rook's "key-management" CLI to fetch the vault token from the secret when
  # validating the connection details and for key rotation operations.
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "update"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "delete"]
  - apiGroups: ["ceph.rook.io"]
    resources: ["cephclusters", "cephclusters/finalizers"]
    verbs: ["get", "list", "create", "update", "delete"]
---
# Aspects of ceph osd purge job that require access to the cluster namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-purge-osd
  namespace: rook-ceph # namespace:cluster
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "delete"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "update", "delete", "list"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-rgw
  namespace: rook-ceph # namespace:cluster
rules:
  # Placeholder role so the rgw service account will
  # be generated in the csv. Remove this role and role binding
  # when fixing https://github.com/rook/rook/issues/10141.
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
---
# Allow the operator to manage resources in its own namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rook-ceph-system
  namespace: rook-ceph # namespace:operator
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
      - services
    verbs:
      - get
      - list
      - watch
      - patch
      - create
      - update
      - delete
  - apiGroups:
      - apps
      - extensions
    resources:
      - daemonsets
      - statefulsets
      - deployments
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - delete
      - deletecollection
  - apiGroups:
      - batch
    resources:
      - cronjobs
    verbs:
      - delete
  - apiGroups:
      - cert-manager.io
    resources:
      - certificates
      - issuers
    verbs:
      - get
      - create
      - delete
  - apiGroups:
      - multicluster.x-k8s.io
    resources:
      - serviceexports
    verbs:
      - get
      - create
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-csi-provisioner-role-cfg
  namespace: rook-ceph # namespace:operator
subjects:
  - kind: ServiceAccount
    name: rook-csi-cephfs-provisioner-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: Role
  name: cephfs-external-provisioner-cfg
  apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-nodeplugin-role-cfg
  namespace: rook-ceph # namespace:operator
subjects:
  - kind: ServiceAccount
    name: rook-csi-rbd-plugin-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: Role
  name: rbd-csi-nodeplugin
  apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-csi-provisioner-role-cfg
  namespace: rook-ceph # namespace:operator
subjects:
  - kind: ServiceAccount
    name: rook-csi-rbd-provisioner-sa
    namespace: rook-ceph # namespace:operator
roleRef:
  kind: Role
  name: rbd-external-provisioner-cfg
  apiGroup: rbac.authorization.k8s.io
---
# Allow the operator to create resources in this cluster's namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-cluster-mgmt
  namespace: rook-ceph # namespace:cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-cluster-mgmt
subjects:
  - kind: ServiceAccount
    name: rook-ceph-system
    namespace: rook-ceph # namespace:operator
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-cmd-reporter
  namespace: rook-ceph # namespace:cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-cmd-reporter
subjects:
  - kind: ServiceAccount
    name: rook-ceph-cmd-reporter
    namespace: rook-ceph # namespace:cluster
---
# Allow the ceph mgr to access resources scoped to the CephCluster namespace necessary for mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph # namespace:cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-mgr
subjects:
  - kind: ServiceAccount
    name: rook-ceph-mgr
    namespace: rook-ceph # namespace:cluster
---
# Allow the ceph mgr to access resources in the Rook operator namespace necessary for mgr modules
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-mgr-system
  namespace: rook-ceph # namespace:operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rook-ceph-mgr-system
subjects:
  - kind: ServiceAccount
    name: rook-ceph-mgr
    namespace: rook-ceph # namespace:cluster
---
# Allow the osd pods in this namespace to work with configmaps
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-osd
  namespace: rook-ceph # namespace:cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-osd
subjects:
  - kind: ServiceAccount
    name: rook-ceph-osd
    namespace: rook-ceph # namespace:cluster
---
# Allow the osd purge job to run in this namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-purge-osd
  namespace: rook-ceph # namespace:cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-purge-osd
subjects:
  - kind: ServiceAccount
    name: rook-ceph-purge-osd
    namespace: rook-ceph # namespace:cluster
---
# Allow the rgw pods in this namespace to work with configmaps
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-rgw
  namespace: rook-ceph # namespace:cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-rgw
subjects:
  - kind: ServiceAccount
    name: rook-ceph-rgw
    namespace: rook-ceph # namespace:cluster
---
# Grant the operator, agent, and discovery agents access to resources in the rook-ceph-system namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rook-ceph-system
  namespace: rook-ceph # namespace:operator
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rook-ceph-system
subjects:
  - kind: ServiceAccount
    name: rook-ceph-system
    namespace: rook-ceph # namespace:operator
---
# Service account for the job that reports the Ceph version in an image
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-cmd-reporter
  namespace: rook-ceph # namespace:cluster
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for Ceph mgrs
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-mgr
  namespace: rook-ceph # namespace:cluster
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for Ceph OSDs
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-osd
  namespace: rook-ceph # namespace:cluster
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for job that purges OSDs from a Rook-Ceph cluster
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-purge-osd
  namespace: rook-ceph # namespace:cluster
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for RGW server
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-rgw
  namespace: rook-ceph # namespace:cluster
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for the Rook-Ceph operator
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-ceph-system
  namespace: rook-ceph # namespace:operator
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/part-of: rook-ceph-operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for the CephFS CSI driver
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-csi-cephfs-plugin-sa
  namespace: rook-ceph # namespace:operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for the CephFS CSI provisioner
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-csi-cephfs-provisioner-sa
  namespace: rook-ceph # namespace:operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for the RBD CSI driver
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-csi-rbd-plugin-sa
  namespace: rook-ceph # namespace:operator
# imagePullSecrets:
#   - name: my-registry-secret
---
# Service account for the RBD CSI provisioner
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rook-csi-rbd-provisioner-sa
  namespace: rook-ceph # namespace:operator
# imagePullSecrets:
#   - name: my-registry-secret

创建operator

# Rook Ceph Operator Config ConfigMap
# Use this ConfigMap to override Rook-Ceph Operator configurations.
# NOTE! Precedence will be given to this config if the same Env Var config also exists in the
#       Operator Deployment.
# To move a configuration(s) from the Operator Deployment to this ConfigMap, add the config
# here. It is recommended to then remove it from the Deployment to eliminate any future confusion.
kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-ceph-operator-config
  # should be in the namespace of the operator
  namespace: rook-ceph # namespace:operator
data:
  # The logging level for the operator: ERROR | WARNING | INFO | DEBUG
  ROOK_LOG_LEVEL: "INFO"

  # Allow using loop devices for osds in test clusters.
  ROOK_CEPH_ALLOW_LOOP_DEVICES: "false"

  # Enable the CSI driver.
  # To run the non-default version of the CSI driver, see the override-able image properties in operator.yaml
  ROOK_CSI_ENABLE_CEPHFS: "true"
  # Enable the default version of the CSI RBD driver. To start another version of the CSI driver, see image properties below.
  ROOK_CSI_ENABLE_RBD: "true"
  # Enable the CSI NFS driver. To start another version of the CSI driver, see image properties below.
  ROOK_CSI_ENABLE_NFS: "false"
  ROOK_CSI_ENABLE_GRPC_METRICS: "false"

  # Set to true to enable Ceph CSI pvc encryption support.
  CSI_ENABLE_ENCRYPTION: "false"

  # Set to true to enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary
  # in some network configurations where the SDN does not provide access to an external cluster or
  # there is significant drop in read/write performance.
  # CSI_ENABLE_HOST_NETWORK: "true"

  # Set to true to enable adding volume metadata on the CephFS subvolume and RBD images.
  # Not all users might be interested in getting volume/snapshot details as metadata on CephFS subvolume and RBD images.
  # Hence enable metadata is false by default.
  # CSI_ENABLE_METADATA: "true"

  # cluster name identifier to set as metadata on the CephFS subvolume and RBD images. This will be useful in cases
  # like for example, when two container orchestrator clusters (Kubernetes/OCP) are using a single ceph cluster.
  # CSI_CLUSTER_NAME: "my-prod-cluster"

  # Set logging level for cephCSI containers maintained by the cephCSI.
  # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
  # CSI_LOG_LEVEL: "0"

  # Set logging level for Kubernetes-csi sidecar containers.
  # Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity.
  # CSI_SIDECAR_LOG_LEVEL: "0"

  # Set replicas for csi provisioner deployment.
  CSI_PROVISIONER_REPLICAS: "2"

  # OMAP generator will generate the omap mapping between the PV name and the RBD image.
  # CSI_ENABLE_OMAP_GENERATOR need to be enabled when we are using rbd mirroring feature.
  # By default OMAP generator sidecar is deployed with CSI provisioner pod, to disable
  # it set it to false.
  # CSI_ENABLE_OMAP_GENERATOR: "false"

  # set to false to disable deployment of snapshotter container in CephFS provisioner pod.
  CSI_ENABLE_CEPHFS_SNAPSHOTTER: "true"

  # set to false to disable deployment of snapshotter container in NFS provisioner pod.
  CSI_ENABLE_NFS_SNAPSHOTTER: "true"

  # set to false to disable deployment of snapshotter container in RBD provisioner pod.
  CSI_ENABLE_RBD_SNAPSHOTTER: "true"

  # Enable cephfs kernel driver instead of ceph-fuse.
  # If you disable the kernel client, your application may be disrupted during upgrade.
  # See the upgrade guide: https://rook.io/docs/rook/latest/ceph-upgrade.html
  # NOTE! cephfs quota is not supported in kernel version < 4.17
  CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"

  # (Optional) policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  CSI_RBD_FSGROUPPOLICY: "File"

  # (Optional) policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  CSI_CEPHFS_FSGROUPPOLICY: "File"

  # (Optional) policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted.
  # supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html
  CSI_NFS_FSGROUPPOLICY: "File"

  # (Optional) Allow starting unsupported ceph-csi image
  ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false"

  # (Optional) control the host mount of /etc/selinux for csi plugin pods.
  CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT: "false"

  # The default version of CSI supported by Rook will be started. To change the version
  # of the CSI driver to something other than what is officially supported, change
  # these images to the desired release of the CSI driver.
  # default images "registry.k8s.io/sig-storage"
  ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.8.0"
  ROOK_CSI_REGISTRAR_IMAGE: "registry.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.7.0"
  ROOK_CSI_RESIZER_IMAGE: "registry.aliyuncs.com/google_containers/csi-resizer:v1.7.0"
  ROOK_CSI_PROVISIONER_IMAGE: "registry.aliyuncs.com/google_containers/csi-provisioner:v3.4.0"
  ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.aliyuncs.com/google_containers/csi-snapshotter:v6.2.1"
  ROOK_CSI_ATTACHER_IMAGE: "registry.aliyuncs.com/google_containers/csi-attacher:v4.1.0"

  # To indicate the image pull policy to be applied to all the containers in the csi driver pods.
  # ROOK_CSI_IMAGE_PULL_POLICY: "IfNotPresent"

  # (Optional) set user created priorityclassName for csi plugin pods.
  CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"

  # (Optional) set user created priorityclassName for csi provisioner pods.
  CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical"

  # CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
  # Default value is RollingUpdate.
  # CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY: "OnDelete"
  # CSI RBD plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
  # Default value is RollingUpdate.
  # CSI_RBD_PLUGIN_UPDATE_STRATEGY: "OnDelete"
  # A maxUnavailable parameter of CSI RBD plugin daemonset update strategy.
  # Default value is 1.
  # CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE: "1"

  # CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.
  # Default value is RollingUpdate.
  # CSI_NFS_PLUGIN_UPDATE_STRATEGY: "OnDelete"

  # kubelet directory path, if kubelet configured to use other than /var/lib/kubelet path.
  # ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/kubelet"

  # Labels to add to the CSI CephFS Deployments and DaemonSets Pods.
  # ROOK_CSI_CEPHFS_POD_LABELS: "key1=value1,key2=value2"
  # Labels to add to the CSI RBD Deployments and DaemonSets Pods.
  # ROOK_CSI_RBD_POD_LABELS: "key1=value1,key2=value2"
  # Labels to add to the CSI NFS Deployments and DaemonSets Pods.
  # ROOK_CSI_NFS_POD_LABELS: "key1=value1,key2=value2"

  # (Optional) CephCSI CephFS plugin Volumes
  # CSI_CEPHFS_PLUGIN_VOLUME: |
  #  - name: lib-modules
  #    hostPath:
  #      path: /run/current-system/kernel-modules/lib/modules/
  #  - name: host-nix
  #    hostPath:
  #      path: /nix

  # (Optional) CephCSI CephFS plugin Volume mounts
  # CSI_CEPHFS_PLUGIN_VOLUME_MOUNT: |
  #  - name: host-nix
  #    mountPath: /nix
  #    readOnly: true

  # (Optional) CephCSI RBD plugin Volumes
  # CSI_RBD_PLUGIN_VOLUME: |
  #  - name: lib-modules
  #    hostPath:
  #      path: /run/current-system/kernel-modules/lib/modules/
  #  - name: host-nix
  #    hostPath:
  #      path: /nix

  # (Optional) CephCSI RBD plugin Volume mounts
  # CSI_RBD_PLUGIN_VOLUME_MOUNT: |
  #  - name: host-nix
  #    mountPath: /nix
  #    readOnly: true

  # (Optional) CephCSI provisioner NodeAffinity (applied to both CephFS and RBD provisioner).
  # CSI_PROVISIONER_NODE_AFFINITY: "role=storage-node; storage=rook, ceph"
  # (Optional) CephCSI provisioner tolerations list(applied to both CephFS and RBD provisioner).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  # CSI_PROVISIONER_TOLERATIONS: |
  #   - effect: NoSchedule
  #     key: node-role.kubernetes.io/control-plane
  #     operator: Exists
  #   - effect: NoExecute
  #     key: node-role.kubernetes.io/etcd
  #     operator: Exists
  # (Optional) CephCSI plugin NodeAffinity (applied to both CephFS and RBD plugin).
  # CSI_PLUGIN_NODE_AFFINITY: "role=storage-node; storage=rook, ceph"
  # (Optional) CephCSI plugin tolerations list(applied to both CephFS and RBD plugin).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  # CSI_PLUGIN_TOLERATIONS: |
  #   - effect: NoSchedule
  #     key: node-role.kubernetes.io/control-plane
  #     operator: Exists
  #   - effect: NoExecute
  #     key: node-role.kubernetes.io/etcd
  #     operator: Exists

  # (Optional) CephCSI RBD provisioner NodeAffinity (if specified, overrides CSI_PROVISIONER_NODE_AFFINITY).
  # CSI_RBD_PROVISIONER_NODE_AFFINITY: "role=rbd-node"
  # (Optional) CephCSI RBD provisioner tolerations list(if specified, overrides CSI_PROVISIONER_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  # CSI_RBD_PROVISIONER_TOLERATIONS: |
  #   - key: node.rook.io/rbd
  #     operator: Exists
  # (Optional) CephCSI RBD plugin NodeAffinity (if specified, overrides CSI_PLUGIN_NODE_AFFINITY).
  # CSI_RBD_PLUGIN_NODE_AFFINITY: "role=rbd-node"
  # (Optional) CephCSI RBD plugin tolerations list(if specified, overrides CSI_PLUGIN_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  # CSI_RBD_PLUGIN_TOLERATIONS: |
  #   - key: node.rook.io/rbd
  #     operator: Exists

  # (Optional) CephCSI CephFS provisioner NodeAffinity (if specified, overrides CSI_PROVISIONER_NODE_AFFINITY).
  # CSI_CEPHFS_PROVISIONER_NODE_AFFINITY: "role=cephfs-node"
  # (Optional) CephCSI CephFS provisioner tolerations list(if specified, overrides CSI_PROVISIONER_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  # CSI_CEPHFS_PROVISIONER_TOLERATIONS: |
  #   - key: node.rook.io/cephfs
  #     operator: Exists
  # (Optional) CephCSI CephFS plugin NodeAffinity (if specified, overrides CSI_PLUGIN_NODE_AFFINITY).
  # CSI_CEPHFS_PLUGIN_NODE_AFFINITY: "role=cephfs-node"
  # NOTE: Support for defining NodeAffinity for operators other than "In" and "Exists" requires the user to input a
  # valid v1.NodeAffinity JSON or YAML string. For example, the following is valid YAML v1.NodeAffinity:
  # CSI_CEPHFS_PLUGIN_NODE_AFFINITY: |
  #   requiredDuringSchedulingIgnoredDuringExecution:
  #     nodeSelectorTerms:
  #       - matchExpressions:
  #         - key: myKey
  #           operator: DoesNotExist
  # (Optional) CephCSI CephFS plugin tolerations list(if specified, overrides CSI_PLUGIN_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  # CSI_CEPHFS_PLUGIN_TOLERATIONS: |
  #   - key: node.rook.io/cephfs
  #     operator: Exists

  # (Optional) CephCSI NFS provisioner NodeAffinity (overrides CSI_PROVISIONER_NODE_AFFINITY).
  # CSI_NFS_PROVISIONER_NODE_AFFINITY: "role=nfs-node"
  # (Optional) CephCSI NFS provisioner tolerations list (overrides CSI_PROVISIONER_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI provisioner would be best to start on the same nodes as other ceph daemons.
  # CSI_NFS_PROVISIONER_TOLERATIONS: |
  #   - key: node.rook.io/nfs
  #     operator: Exists
  # (Optional) CephCSI NFS plugin NodeAffinity (overrides CSI_PLUGIN_NODE_AFFINITY).
  # CSI_NFS_PLUGIN_NODE_AFFINITY: "role=nfs-node"
  # (Optional) CephCSI NFS plugin tolerations list (overrides CSI_PLUGIN_TOLERATIONS).
  # Put here list of taints you want to tolerate in YAML format.
  # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
  # CSI_NFS_PLUGIN_TOLERATIONS: |
  #   - key: node.rook.io/nfs
  #     operator: Exists

  # (Optional) CEPH CSI RBD provisioner resource requirement list, Put here list of resource
  # requests and limits you want to apply for provisioner pod
  #CSI_RBD_PROVISIONER_RESOURCE: |
  #  - name : csi-provisioner
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-resizer
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-attacher
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-snapshotter
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-rbdplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : csi-omap-generator
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  # (Optional) CEPH CSI RBD plugin resource requirement list, Put here list of resource
  # requests and limits you want to apply for plugin pod
  #CSI_RBD_PLUGIN_RESOURCE: |
  #  - name : driver-registrar
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  #  - name : csi-rbdplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  # (Optional) CEPH CSI CephFS provisioner resource requirement list, Put here list of resource
  # requests and limits you want to apply for provisioner pod
  #CSI_CEPHFS_PROVISIONER_RESOURCE: |
  #  - name : csi-provisioner
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-resizer
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-attacher
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-snapshotter
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-cephfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  # (Optional) CEPH CSI CephFS plugin resource requirement list, Put here list of resource
  # requests and limits you want to apply for plugin pod
  #CSI_CEPHFS_PLUGIN_RESOURCE: |
  #  - name : driver-registrar
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  #  - name : csi-cephfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : liveness-prometheus
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m

  # (Optional) CEPH CSI NFS provisioner resource requirement list, Put here list of resource
  # requests and limits you want to apply for provisioner pod
  # CSI_NFS_PROVISIONER_RESOURCE: |
  #  - name : csi-provisioner
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  #  - name : csi-nfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m
  #  - name : csi-attacher
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 100m
  #      limits:
  #        memory: 256Mi
  #        cpu: 200m
  # (Optional) CEPH CSI NFS plugin resource requirement list, Put here list of resource
  # requests and limits you want to apply for plugin pod
  # CSI_NFS_PLUGIN_RESOURCE: |
  #  - name : driver-registrar
  #    resource:
  #      requests:
  #        memory: 128Mi
  #        cpu: 50m
  #      limits:
  #        memory: 256Mi
  #        cpu: 100m
  #  - name : csi-nfsplugin
  #    resource:
  #      requests:
  #        memory: 512Mi
  #        cpu: 250m
  #      limits:
  #        memory: 1Gi
  #        cpu: 500m

  # Configure CSI Ceph FS grpc and liveness metrics port
  # Set to true to enable Ceph CSI liveness container.
  CSI_ENABLE_LIVENESS: "false"
  # CSI_CEPHFS_GRPC_METRICS_PORT: "9091"
  # CSI_CEPHFS_LIVENESS_METRICS_PORT: "9081"
  # Configure CSI RBD grpc and liveness metrics port
  # CSI_RBD_GRPC_METRICS_PORT: "9090"
  # CSI_RBD_LIVENESS_METRICS_PORT: "9080"
  # CSIADDONS_PORT: "9070"

  # Set CephFS Kernel mount options to use https://docs.ceph.com/en/latest/man/8/mount.ceph/#options
  # Set to "ms_mode=secure" when connections.encrypted is enabled in CephCluster CR
  # CSI_CEPHFS_KERNEL_MOUNT_OPTIONS: "ms_mode=secure"

  # Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used
  ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"

  # Whether to start the discovery daemon to watch for raw storage devices on nodes in the cluster.
  # This daemon does not need to run if you are only going to create your OSDs based on StorageClassDeviceSets with PVCs.
  ROOK_ENABLE_DISCOVERY_DAEMON: "false"
  # The timeout value (in seconds) of Ceph commands. It should be >= 1. If this variable is not set or is an invalid value, it's default to 15.
  ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
  # Enable the csi addons sidecar.
  CSI_ENABLE_CSIADDONS: "false"
  # ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.5.0"
  # The CSI GRPC timeout value (in seconds). It should be >= 120. If this variable is not set or is an invalid value, it's default to 150.
  CSI_GRPC_TIMEOUT_SECONDS: "150"

  ROOK_DISABLE_ADMISSION_CONTROLLER: "true"

  # Enable topology based provisioning.
  CSI_ENABLE_TOPOLOGY: "false"
  # Domain labels define which node labels to use as domains
  # for CSI nodeplugins to advertise their domains
  # NOTE: the value here serves as an example and needs to be
  # updated with node labels that define domains of interest
  # CSI_TOPOLOGY_DOMAIN_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"

  # Enable read affinity for RBD volumes. Recommended to
  # set to true if running kernel 5.8 or newer.
  CSI_ENABLE_READ_AFFINITY: "false"
  # CRUSH location labels define which node labels to use
  # as CRUSH location. This should correspond to the values set in
  # the CRUSH map.
  # Defaults to all the labels mentioned in
  # https://rook.io/docs/rook/latest/CRDs/Cluster/ceph-cluster-crd/#osd-topology
  # CSI_CRUSH_LOCATION_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"

  # Whether to skip any attach operation altogether for CephCSI PVCs.
  # See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
  # If set to false it skips the volume attachments and makes the creation of pods using the CephCSI PVC fast.
  # **WARNING** It's highly discouraged to use this for RWO volumes. for RBD PVC it can cause data corruption,
  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false
  # since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
  CSI_CEPHFS_ATTACH_REQUIRED: "true"
  CSI_RBD_ATTACH_REQUIRED: "true"
  CSI_NFS_ATTACH_REQUIRED: "true"
---
# OLM: BEGIN OPERATOR DEPLOYMENT
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-operator
  namespace: rook-ceph # namespace:operator
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/name: rook-ceph
    app.kubernetes.io/instance: rook-ceph
    app.kubernetes.io/component: rook-ceph-operator
    app.kubernetes.io/part-of: rook-ceph-operator
spec:
  selector:
    matchLabels:
      app: rook-ceph-operator
  strategy:
    type: Recreate
  replicas: 1
  template:
    metadata:
      labels:
        app: rook-ceph-operator
    spec:
      serviceAccountName: rook-ceph-system
      containers:
        - name: rook-ceph-operator
          image: rook/ceph:v1.11.9
          args: ["ceph", "operator"]
          securityContext:
            runAsNonRoot: true
            runAsUser: 2016
            runAsGroup: 2016
          volumeMounts:
            - mountPath: /var/lib/rook
              name: rook-config
            - mountPath: /etc/ceph
              name: default-config-dir
            - mountPath: /etc/webhook
              name: webhook-cert
          ports:
            - containerPort: 9443
              name: https-webhook
              protocol: TCP
          env:
            # If the operator should only watch for cluster CRDs in the same namespace, set this to "true".
            # If this is not set to true, the operator will watch for cluster CRDs in all namespaces.
            - name: ROOK_CURRENT_NAMESPACE_ONLY
              value: "false"
            # Rook Discover toleration. Will tolerate all taints with all keys.
            # Choose between NoSchedule, PreferNoSchedule and NoExecute:
            # - name: DISCOVER_TOLERATION
            #   value: "NoSchedule"
            # (Optional) Rook Discover toleration key. Set this to the key of the taint you want to tolerate
            # - name: DISCOVER_TOLERATION_KEY
            #   value: ""
            # (Optional) Rook Discover tolerations list. Put here list of taints you want to tolerate in YAML format.
            # - name: DISCOVER_TOLERATIONS
            #   value: |
            #     - effect: NoSchedule
            #       key: node-role.kubernetes.io/control-plane
            #       operator: Exists
            #     - effect: NoExecute
            #       key: node-role.kubernetes.io/etcd
            #       operator: Exists
            # (Optional) Rook Discover priority class name to set on the pod(s)
            # - name: DISCOVER_PRIORITY_CLASS_NAME
            #   value: ""
            # (Optional) Discover Agent NodeAffinity.
            # - name: DISCOVER_AGENT_NODE_AFFINITY
            #   value: "role=storage-node; storage=rook, ceph"
            # (Optional) Discover Agent Pod Labels.
            # - name: DISCOVER_AGENT_POD_LABELS
            #   value: "key1=value1,key2=value2"

            # The duration between discovering devices in the rook-discover daemonset.
            - name: ROOK_DISCOVER_DEVICES_INTERVAL
              value: "60m"

            # Whether to start pods as privileged that mount a host path, which includes the Ceph mon and osd pods.
            # Set this to true if SELinux is enabled (e.g. OpenShift) to workaround the anyuid issues.
            # For more details see https://github.com/rook/rook/issues/1314#issuecomment-355799641
            - name: ROOK_HOSTPATH_REQUIRES_PRIVILEGED
              value: "false"

            # Disable automatic orchestration when new devices are discovered
            - name: ROOK_DISABLE_DEVICE_HOTPLUG
              value: "false"

            # Provide customised regex as the values using comma. For eg. regex for rbd based volume, value will be like "(?i)rbd[0-9]+".
            # In case of more than one regex, use comma to separate between them.
            # Default regex will be "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+"
            # Add regex expression after putting a comma to blacklist a disk
            # If value is empty, the default regex will be used.
            - name: DISCOVER_DAEMON_UDEV_BLACKLIST
              value: "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+"

            # - name: DISCOVER_DAEMON_RESOURCES
            #   value: |
            #     resources:
            #       limits:
            #         cpu: 500m
            #         memory: 512Mi
            #       requests:
            #         cpu: 100m
            #         memory: 128Mi

            # Time to wait until the node controller will move Rook pods to other
            # nodes after detecting an unreachable node.
            # Pods affected by this setting are:
            # mgr, rbd, mds, rgw, nfs, PVC based mons and osds, and ceph toolbox
            # The value used in this variable replaces the default value of 300 secs
            # added automatically by k8s as Toleration for
            # 
            # The total amount of time to reschedule Rook pods in healthy nodes
            # before detecting a  condition will be the sum of:
            #  --> node-monitor-grace-period: 40 seconds (k8s kube-controller-manager flag)
            #  --> ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS: 5 seconds
            - name: ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS
              value: "5"

            # The name of the node to pass with the downward API
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            # The pod name to pass with the downward API
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # The pod namespace to pass with the downward API
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          # Recommended resource requests and limits, if desired
          #resources:
          #  limits:
          #    cpu: 500m
          #    memory: 512Mi
          #  requests:
          #    cpu: 100m
          #    memory: 128Mi

          #  Uncomment it to run lib bucket provisioner in multithreaded mode
          #- name: LIB_BUCKET_PROVISIONER_THREADS
          #  value: "5"

      # Uncomment it to run rook operator on the host network
      #hostNetwork: true
      volumes:
        - name: rook-config
          emptyDir: {}
        - name: default-config-dir
          emptyDir: {}
        - name: webhook-cert
          emptyDir: {}
# OLM: END OPERATOR DEPLOYMENT

创建ceph-cluster

#################################################################################################################
# Define the settings for the rook-ceph cluster with common settings for a production cluster.
# All nodes with available raw devices will be used for the Ceph cluster. At least three nodes are required
# in this example. See the documentation for more details on storage settings available.

# For example, to create the cluster:
#   kubectl create -f crds.yaml -f common.yaml -f operator.yaml
#   kubectl create -f cluster.yaml
#################################################################################################################

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph # namespace:cluster
spec:
  cephVersion:
    # The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
    # v16 is Pacific, and v17 is Quincy.
    # RECOMMENDATION: In production, use a specific version tag instead of the general v17 flag, which pulls the latest release and could result in different
    # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
    # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v17.2.6-20230410
    # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
    image: quay.io/ceph/ceph:v17.2.6
    # Whether to allow unsupported versions of Ceph. Currently `pacific` and `quincy` are supported.
    # Future versions such as `reef` (v18) would require this to be set to `true`.
    # Do not set to true in production.
    allowUnsupported: false
  # The path on the host where configuration files will be persisted. Must be specified.
  # Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
  # In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
  dataDirHostPath: /var/lib/rook
  # Whether or not upgrade should continue even if a check fails
  # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise
  # Use at your OWN risk
  # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/latest/ceph-upgrade.html#ceph-version-upgrades
  skipUpgradeChecks: false
  # Whether or not continue if PGs are not clean during an upgrade
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  # WaitTimeoutForHealthyOSDInMinutes defines the time (in minutes) the operator would wait before an OSD can be stopped for upgrade or restart.
  # If the timeout exceeds and OSD is not ok to stop, then the operator would skip upgrade for the current OSD and proceed with the next one
  # if `continueUpgradeAfterChecksEvenIfNotHealthy` is `false`. If `continueUpgradeAfterChecksEvenIfNotHealthy` is `true`, then operator would
  # continue with the upgrade of an OSD even if its not ok to stop after the timeout. This timeout won't be applied if `skipUpgradeChecks` is `true`.
  # The default wait timeout is 10 minutes.
  waitTimeoutForHealthyOSDInMinutes: 10
  mon:
    # Set the number of mons to be started. Generally recommended to be 3.
    # For highest availability, an odd number of mons should be specified.
    count: 3
    # The mons should be on unique nodes. For production, at least 3 nodes are recommended for this reason.
    # Mons should only be allowed on the same node for test environments where data loss is acceptable.
    allowMultiplePerNode: false
  mgr:
    # When higher availability of the mgr is needed, increase the count to 2.
    # In that case, one mgr will be active and one in standby. When Ceph updates which
    # mgr is active, Rook will update the mgr services to match the active mgr.
    count: 2
    allowMultiplePerNode: false
    modules:
      # Several modules should not need to be included in this list. The "dashboard" and "monitoring" modules
      # are already enabled by other settings in the cluster CR.
      - name: pg_autoscaler
        enabled: true
  # enable the ceph dashboard for viewing cluster status
  dashboard:
    enabled: true
    # serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
    # urlPrefix: /ceph-dashboard
    # serve the dashboard at the given port.
    # port: 8443
    # serve the dashboard using SSL
    ssl: true
  # enable prometheus alerting for cluster
  monitoring:
    # requires Prometheus to be pre-installed
    enabled: false
    # Whether to disable the metrics reported by Ceph. If false, the prometheus mgr module and Ceph exporter are enabled.
    # If true, the prometheus mgr module and Ceph exporter are both disabled. Default is false.
    metricsDisabled: false
  network:
    connections:
      # Whether to encrypt the data in transit across the wire to prevent eavesdropping the data on the network.
      # The default is false. When encryption is enabled, all communication between clients and Ceph daemons, or between Ceph daemons will be encrypted.
      # When encryption is not enabled, clients still establish a strong initial authentication and data integrity is still validated with a crc check.
      # IMPORTANT: Encryption requires the 5.11 kernel for the latest nbd and cephfs drivers. Alternatively for testing only,
      # you can set the "mounter: rbd-nbd" in the rbd storage class, or "mounter: fuse" in the cephfs storage class.
      # The nbd and fuse drivers are *not* recommended in production since restarting the csi driver pod will disconnect the volumes.
      encryption:
        enabled: false
      # Whether to compress the data in transit across the wire. The default is false.
      # Requires Ceph Quincy (v17) or newer. Also see the kernel requirements above for encryption.
      compression:
        enabled: false
      # Whether to require communication over msgr2. If true, the msgr v1 port (6789) will be disabled
      # and clients will be required to connect to the Ceph cluster with the v2 port (3300).
      # Requires a kernel that supports msgr v2 (kernel 5.11 or CentOS 8.4 or newer).
      requireMsgr2: false
    # enable host networking
    #provider: host
    # enable the Multus network provider
    #provider: multus
    #selectors:
      # The selector keys are required to be `public` and `cluster`.
      # Based on the configuration, the operator will do the following:
      #   1. if only the `public` selector key is specified both public_network and cluster_network Ceph settings will listen on that interface
      #   2. if both `public` and `cluster` selector keys are specified the first one will point to 'public_network' flag and the second one to 'cluster_network'
      #
      # In order to work, each selector value must match a NetworkAttachmentDefinition object in Multus
      #
      #public: public-conf --> NetworkAttachmentDefinition object name in Multus
      #cluster: cluster-conf --> NetworkAttachmentDefinition object name in Multus
    # Provide internet protocol version. IPv6, IPv4 or empty string are valid options. Empty string would mean IPv4
    #ipFamily: "IPv6"
    # Ceph daemons to listen on both IPv4 and Ipv6 networks
    #dualStack: false
    # Enable multiClusterService to export the mon and OSD services to peer cluster.
    # This is useful to support RBD mirroring between two clusters having overlapping CIDRs.
    # Ensure that peer clusters are connected using an MCS API compatible application, like Globalnet Submariner.
    #multiClusterService:
    #  enabled: false

  # enable the crash collector for ceph daemon crash collection
  crashCollector:
    disable: false
    # Uncomment daysToRetain to prune ceph crash entries older than the
    # specified number of days.
    #daysToRetain: 30
  # enable log collector, daemons will log on files and rotate
  logCollector:
    enabled: true
    periodicity: daily # one of: hourly, daily, weekly, monthly
    maxLogSize: 500M # SUFFIX may be 'M' or 'G'. Must be at least 1M.
  # automate [data cleanup process](https://github.com/rook/rook/blob/master/Documentation/Storage-Configuration/ceph-teardown.md#delete-the-data-on-hosts) in cluster destruction.
  cleanupPolicy:
    # Since cluster cleanup is destructive to data, confirmation is required.
    # To destroy all Rook data on hosts during uninstall, confirmation must be set to "yes-really-destroy-data".
    # This value should only be set when the cluster is about to be deleted. After the confirmation is set,
    # Rook will immediately stop configuring the cluster and only wait for the delete command.
    # If the empty string is set, Rook will not destroy any data on hosts during uninstall.
    confirmation: ""
    # sanitizeDisks represents settings for sanitizing OSD disks on cluster deletion
    sanitizeDisks:
      # method indicates if the entire disk should be sanitized or simply ceph's metadata
      # in both case, re-install is possible
      # possible choices are 'complete' or 'quick' (default)
      method: quick
      # dataSource indicate where to get random bytes from to write on the disk
      # possible choices are 'zero' (default) or 'random'
      # using random sources will consume entropy from the system and will take much more time then the zero source
      dataSource: zero
      # iteration overwrite N times instead of the default (1)
      # takes an integer value
      iteration: 1
    # allowUninstallWithVolumes defines how the uninstall should be performed
    # If set to true, cephCluster deletion does not wait for the PVs to be deleted.
    allowUninstallWithVolumes: false
  # To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
  # The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
  # tolerate taints with a key of 'storage-node'.
  # placement:
  #   all:
  #     nodeAffinity:
  #       requiredDuringSchedulingIgnoredDuringExecution:
  #         nodeSelectorTerms:
  #         - matchExpressions:
  #           - key: role
  #             operator: In
  #             values:
  #             - storage-node
  #     podAffinity:
  #     podAntiAffinity:
  #     topologySpreadConstraints:
  #     tolerations:
  #     - key: storage-node
  #       operator: Exists
  # The above placement information can also be specified for mon, osd, and mgr components
  #   mon:
  # Monitor deployments may contain an anti-affinity rule for avoiding monitor
  # collocation on the same node. This is a required rule when host network is used
  # or when AllowMultiplePerNode is false. Otherwise this anti-affinity rule is a
  # preferred rule with weight: 50.
  #   osd:
  #    prepareosd:
  #    mgr:
  #    cleanup:
  annotations:
  #   all:
  #   mon:
  #   osd:
  #   cleanup:
  #   prepareosd:
  # clusterMetadata annotations will be applied to only `rook-ceph-mon-endpoints` configmap and the `rook-ceph-mon` and `rook-ceph-admin-keyring` secrets.
  # And clusterMetadata annotations will not be merged with `all` annotations.
  #    clusterMetadata:
  #       kubed.appscode.com/sync: "true"
  # If no mgr annotations are set, prometheus scrape annotations will be set by default.
  #   mgr:
  labels:
  #   all:
  #   mon:
  #   osd:
  #   cleanup:
  #   mgr:
  #   prepareosd:
  # monitoring is a list of key-value pairs. It is injected into all the monitoring resources created by operator.
  # These labels can be passed as LabelSelector to Prometheus
  #   monitoring:
  #   crashcollector:
  resources:
  #The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
  #   mgr:
  #     limits:
  #       cpu: "500m"
  #       memory: "1024Mi"
  #     requests:
  #       cpu: "500m"
  #       memory: "1024Mi"
  # The above example requests/limits can also be added to the other components
  #   mon:
  #   osd:
  # For OSD it also is a possible to specify requests/limits based on device class
  #   osd-hdd:
  #   osd-ssd:
  #   osd-nvme:
  #   prepareosd:
  #   mgr-sidecar:
  #   crashcollector:
  #   logcollector:
  #   cleanup:
  #   exporter:
  # The option to automatically remove OSDs that are out and are safe to destroy.
  removeOSDsIfOutAndSafeToRemove: false
  priorityClassNames:
    #all: rook-ceph-default-priority-class
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
    #crashcollector: rook-ceph-crashcollector-priority-class
  storage: # 集群级存储配置和选择
    useAllNodes: true # 使用所有节点
    useAllDevices: true # 使用所有设备
    #deviceFilter:
    config:
      # crushRoot: "custom-root" # specify a non-default root label for the CRUSH map
      # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
      # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
      # journalSizeMB: "1024"  # uncomment if the disks are 20 GB or smaller
      # osdsPerDevice: "1" # this value can be overridden at the node or device level
      # encryptedDevice: "true" # the default value for this option is "false"
    # Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
    # nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    # 指定节点
    # nodes:
    #   - name: "172.17.4.201"
    #     devices: # specific devices to use for storage can be specified for each node
    #       - name: "sdb"
    #       - name: "nvme01" # multiple osds can be created on high performance devices
    #         config:
    #           osdsPerDevice: "5"
    #       - name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths
    #     config: # configuration can be specified at the node level which overrides the cluster level config
    #   - name: "172.17.4.301"
    #     deviceFilter: "^sd."
    # when onlyApplyOSDPlacement is false, will merge both placement.All() and placement.osd
    onlyApplyOSDPlacement: false
  # The section for configuring management of daemon disruptions during upgrade or fencing.
  disruptionManagement:
    # If true, the operator will create and manage PodDisruptionBudgets for OSD, Mon, RGW, and MDS daemons. OSD PDBs are managed dynamically
    # via the strategy outlined in the [design](https://github.com/rook/rook/blob/master/design/ceph/ceph-managed-disruptionbudgets.md). The operator will
    # block eviction of OSDs by default and unblock them safely when drains are detected.
    managePodBudgets: true
    # A duration in minutes that determines how long an entire failureDomain like `region/zone/host` will be held in `noout` (in addition to the
    # default DOWN/OUT interval) when it is draining. This is only relevant when  `managePodBudgets` is `true`. The default value is `30` minutes.
    osdMaintenanceTimeout: 30
    # A duration in minutes that the operator will wait for the placement groups to become healthy (active+clean) after a drain was completed and OSDs came back up.
    # Operator will continue with the next drain if the timeout exceeds. It only works if `managePodBudgets` is `true`.
    # No values or 0 means that the operator will wait until the placement groups are healthy before unblocking the next drain.
    pgHealthCheckTimeout: 0

  # healthChecks
  # Valid values for daemons are 'mon', 'osd', 'status'
  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    # Change pod liveness probe timing or threshold values. Works for all mon,mgr,osd daemons.
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    # Change pod startup probe timing or threshold values. Works for all mon,mgr,osd daemons.
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false

创建service

apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-mgr-dashboard-external-https
  namespace: rook-ceph # namespace:cluster
  labels:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph # namespace:cluster
spec:
  ports:
    - name: dashboard
      port: 8443
      protocol: TCP
      targetPort: 8443
  selector:
    app: rook-ceph-mgr
    mgr_role: active
    rook_cluster: rook-ceph
  sessionAffinity: None
  type: NodePort

创建ceph存储

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: ceph
  namespace: rook-ceph
spec:
  metadataPool:
   replicated:
     size: 3
  dataPools:
    - replicated:  
        size: 3
  preservePoolsOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true

创建集群storageclass

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    k8s.kuboard.cn/storageType: cephfs_rook
  name: ceph
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  fsName: ceph
  pool: ceph-data0
provisioner: rook-ceph.cephfs.csi.ceph.com
reclaimPolicy: Retain
volumeBindingMode: Immediate

创建ceph工具箱

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-tools
  namespace: rook-ceph # namespace:cluster
  labels:
    app: rook-ceph-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rook-ceph-tools
  template:
    metadata:
      labels:
        app: rook-ceph-tools
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: rook-ceph-tools
          image: quay.io/ceph/ceph:v17.2.6
          command:
            - /bin/bash
            - -c
            - |
              # Replicate the script from toolbox.sh inline so the ceph image
              # can be run directly, instead of requiring the rook toolbox
              CEPH_CONFIG="/etc/ceph/ceph.conf"
              MON_CONFIG="/etc/rook/mon-endpoints"
              KEYRING_FILE="/etc/ceph/keyring"

              # create a ceph config file in its default location so ceph/rados tools can be used
              # without specifying any arguments
              write_endpoints() {
                endpoints=$(cat ${MON_CONFIG})

                # filter out the mon names
                # external cluster can have numbers or hyphens in mon names, handling them in regex
                # shellcheck disable=SC2001
                mon_endpoints=$(echo "${endpoints}"| sed 's/[a-z0-9_-]\+=//g')

                DATE=$(date)
                echo "$DATE writing mon endpoints to ${CEPH_CONFIG}: ${endpoints}"
                  cat < ${CEPH_CONFIG}
              [global]
              mon_host = ${mon_endpoints}

              [client.admin]
              keyring = ${KEYRING_FILE}
              EOF
              }

              # watch the endpoints config file and update if the mon endpoints ever change
              watch_endpoints() {
                # get the timestamp for the target of the soft link
                real_path=$(realpath ${MON_CONFIG})
                initial_time=$(stat -c %Z "${real_path}")
                while true; do
                  real_path=$(realpath ${MON_CONFIG})
                  latest_time=$(stat -c %Z "${real_path}")

                  if [[ "${latest_time}" != "${initial_time}" ]]; then
                    write_endpoints
                    initial_time=${latest_time}
                  fi

                  sleep 10
                done
              }

              # read the secret from an env var (for backward compatibility), or from the secret file
              ceph_secret=${ROOK_CEPH_SECRET}
              if [[ "$ceph_secret" == "" ]]; then
                ceph_secret=$(cat /var/lib/rook-ceph-mon/secret.keyring)
              fi

              # create the keyring file
              cat < ${KEYRING_FILE}
              [${ROOK_CEPH_USERNAME}]
              key = ${ceph_secret}
              EOF

              # write the initial config file
              write_endpoints

              # continuously update the mon endpoints if they fail over
              watch_endpoints
          imagePullPolicy: IfNotPresent
          tty: true
          securityContext:
            runAsNonRoot: true
            runAsUser: 2016
            runAsGroup: 2016
          env:
            - name: ROOK_CEPH_USERNAME
              valueFrom:
                secretKeyRef:
                  name: rook-ceph-mon
                  key: ceph-username
          volumeMounts:
            - mountPath: /etc/ceph
              name: ceph-config
            - name: mon-endpoint-volume
              mountPath: /etc/rook
            - name: ceph-admin-secret
              mountPath: /var/lib/rook-ceph-mon
              readOnly: true
      volumes:
        - name: ceph-admin-secret
          secret:
            secretName: rook-ceph-mon
            optional: false
            items:
              - key: ceph-secret
                path: secret.keyring
        - name: mon-endpoint-volume
          configMap:
            name: rook-ceph-mon-endpoints
            items:
              - key: data
                path: mon-endpoints
        - name: ceph-config
          emptyDir: {}
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5

本文2023-08-15首发于我的博客，最后修改于2023-08-15