Etcd 数据备份与恢复

一、单节点 etcd 备份恢复

1.1 单节点 etcd 部署

在备份前我们先部署一个单节点的etcd服务

1
2
[root@localhost ~]# yum install etcd-3.3.11
[root@localhost ~]# systemctl start etcd

查看etcd状态

1
2
3
4
5
6
7
8
9
10
11
12
[root@localhost ~]# ETCDCTL_API=3 etcdctl member list -w table
+------------------+---------+---------+-----------------------+-----------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+---------+-----------------------+-----------------------+
| 8e9e05c52164694d | started | default | http://localhost:2380 | http://localhost:2379 |
+------------------+---------+---------+-----------------------+-----------------------+
[root@localhost ~]# ETCDCTL_API=3 etcdctl endpoint status -w table
+----------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------+------------------+---------+---------+-----------+-----------+------------+
| 127.0.0.1:2379 | 8e9e05c52164694d | 3.3.11 | 20 kB | true | 2 | 4 |
+----------------+------------------+---------+---------+-----------+-----------+------------+

接着我们写入一条测试数据

1
2
3
4
5
[root@localhost ~]# ETCDCTL_API=3 etcdctl put ok 123
OK
[root@localhost ~]# ETCDCTL_API=3 etcdctl get ok
ok
123

1.2 备份单节点 etcd

如果etcd使用了证书,需要在etcdctl后面添加--cacert=ca.crt --cert=etcd-client.crt --key=etcd-client.key参数正确引用证书即可
将etcd备份到用户~/etcd-snapshot.db宿主目录下

1
2
3
4
5
6
[root@localhost ~]# ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 snapshot save ~/etcd-snapshot.db
Snapshot saved at /root/etcd-snapshot.db

#通过 snapshot status 查看etcd备份数据是否完整
[root@localhost ~]# ETCDCTL_API=3 etcdctl snapshot status ~/etcd-snapshot.db
388944a0, 2, 5, 20 kB

1.3 恢复单节点 etcd

开始恢复前使用 ps -ef |grep etcd查看etcd的运行用户,后面恢复数据后需要改数据目录的属主

1
2
3
4
5
6
7
8
9
10
[root@localhost ~]# systemctl stop etcd
[root@localhost ~]# rm -rf /var/lib/etcd/default.etcd
[root@localhost ~]# ETCDCTL_API=3 etcdctl --data-dir=/var/lib/etcd/default.etcd snapshot restore ~/etcd-snapshot.db
[root@localhost ~]# chown -R etcd.etcd /var/lib/etcd/default.etcd
[root@localhost ~]# systemctl start etcd

#最后再次查询之前创建的测试数据
[root@localhost ~]# ETCDCTL_API=3 etcdctl get ok
ok
123

二、etcd 集群备份恢复

2.1 etcd 集群部署

etcd 默认使用端口 2379 和 2380,2379用于接收客户端请求,2380 用于 etcd 节点之间的对等通信

主机名IP地址备注
etcd1192.168.223.131新建集群
etcd2192.168.223.137新建集群
etcd3192.168.223.140加入集群
参数说明
ETCD_DATA_DIR数据存储目录
ETCD_LISTEN_PEER_URLS监听地址,用于节点之间通信的url,可多个,集群内数据交互(如选举,数据同步等)
ETCD_LISTEN_CLIENT_URLS监听地址,的用于客户端通信的url,同样可以监听多个
ETCD_NAMEetcd集群中的节点名,这里可以随意,不重复就行
ETCD_INITIAL_ADVERTISE_PEER_URLS在集群中宣告用于节点之间通信的url,节点间将以该值进行通信
ETCD_ADVERTISE_CLIENT_URLS在集群中宣告使用的客户端通信url,该值用于etcd代理或etcd成员与etcd节点通信
ETCD_INITIAL_CLUSTER集群中的成员集合
ETCD_INITIAL_CLUSTER_STATE初始化集群的标志,新建使用 new,加入一个存在的集群为 existing
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
# 配置所有节点hosts
[root@etcd1 ~]# cat /etc/hosts
192.168.223.131 etcd1
192.168.223.137 etcd2
192.168.223.140 etcd3

# 配置 etcd1 节点
# 清空数据,修改配置文件,组建 2 节点集群
[root@etcd1 ~]# systemctl stop etcd
[root@etcd1 ~]# rm -rf /var/lib/etcd/default.etcd
[root@etcd1 ~]# vim /etc/etcd/etcd.conf
# Member
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.223.131:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.223.131:2379"
# Cluster
ETCD_ADVERTISE_CLIENT_URLS="http://etcd1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

[root@etcd1 ~]# systemctl start etcd
# 会卡住,等待第二节点加入

# 配置 etcd2 节点
[root@etcd2 ~]# yum install etcd-3.3.11
[root@etcd2 ~]# vim /etc/etcd/etcd.conf
# Member
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.223.137:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.223.137:2379"
# Cluster
ETCD_ADVERTISE_CLIENT_URLS="http://etcd2:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd2:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

[root@etcd2 ~]# systemctl enable --now etcd
[root@etcd2 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..2}:2379 member list -w table
+------------------+---------+-------+-------------------+-------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+-------------------+-------------------+
| ade526d28b1f92f7 | started | etcd1 | http://etcd1:2380 | http://etcd1:2379 |
| d282ac2ce600c1ce | started | etcd2 | http://etcd2:2380 | http://etcd2:2379 |
+------------------+---------+-------+-------------------+-------------------+
[root@etcd2 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..2}:2379 endpoint status -w table
+------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------+------------------+---------+---------+-----------+-----------+------------+
| etcd1:2379 | ade526d28b1f92f7 | 3.3.11 | 20 kB | true | 5 | 6 |
| etcd2:2379 | d282ac2ce600c1ce | 3.3.11 | 20 kB | false | 5 | 6 |
+------------+------------------+---------+---------+-----------+-----------+------------+

#--------------------节点 etcd3 加入集群------------------#
[root@etcd3 ~]# yum install etcd-3.3.11
[root@etcd3 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd1:2379 member list -w table
+------------------+---------+-------+-------------------+-------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+-------------------+-------------------+
| ade526d28b1f92f7 | started | etcd1 | http://etcd1:2380 | http://etcd1:2379 |
| d282ac2ce600c1ce | started | etcd2 | http://etcd2:2380 | http://etcd2:2379 |
+------------------+---------+-------+-------------------+-------------------+
[root@etcd3 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..2}:2379 member add etcd3 --peer-urls=http://etcd3:2380
Member ada76a05ac66b5ef added to cluster 8dd6792a88149148

ETCD_NAME="etcd3"
ETCD_INITIAL_CLUSTER="etcd3=http://etcd3:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[root@etcd3 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd1:2379 member list -w table
+------------------+-----------+-------+-------------------+-------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+-----------+-------+-------------------+-------------------+
| ade526d28b1f92f7 | started | etcd1 | http://etcd1:2380 | http://etcd1:2379 |
| d282ac2ce600c1ce | started | etcd2 | http://etcd2:2380 | http://etcd2:2379 |
| c931e7bd0b007b9d | unstarted | | http://etcd3:2380 | |
+------------------+-----------+-------+-------------------+-------------------+
[root@etcd3 ~]# vim /etc/etcd/etcd.conf
# Member
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.255.13:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.255.13:2379"
# Cluster
ETCD_ADVERTISE_CLIENT_URLS="http://etcd3:2379"
ETCD_INITIAL_CLUSTER="etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

[root@etcd3 ~]# rm -rf /var/lib/etcd/default.etcd
[root@etcd3 ~]# systemctl enable --now etcd
[root@etcd3 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..3}:2379 endpoint status -w table
+------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------+------------------+---------+---------+-----------+-----------+------------+
| etcd1:2379 | ade526d28b1f92f7 | 3.3.11 | 20 kB | true | 5 | 9 |
| etcd2:2379 | d282ac2ce600c1ce | 3.3.11 | 20 kB | false | 5 | 9 |
| etcd3:2379 | c931e7bd0b007b9d | 3.3.11 | 20 kB | false | 5 | 9 |
+------------+------------------+---------+---------+-----------+-----------+------------+

2.2 etcd 集群数据备份

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#这里为了测试,向etcd中写入测试数据,生产环境直接备份
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..3}:2379 put test 123
OK
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..3}:2379 get test
test
123
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..3}:2379 snapshot save ~/etcd-snapshot.db
Snapshot saved at /root/etcd-snapshot.db

[root@etcd1 ~]# ETCDCTL_API=3 etcdctl snapshot status ~/etcd-snapshot.db -w table
+----------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| f3b7530f | 4 | 9 | 20 kB |
+----------+----------+------------+------------+

2.3 etcd 集群数据恢复

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# 删除集群数据
[root@etcd1 ~]# systemctl stop etcd && rm -rf /var/lib/etcd/default.etcd/*
[root@etcd2 ~]# systemctl stop etcd && rm -rf /var/lib/etcd/default.etcd/*
[root@etcd3 ~]# systemctl stop etcd && rm -rf /var/lib/etcd/default.etcd/*

# 恢复集群数据,在任意节点上恢复etcd各节点数据目录
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl snapshot restore ~/etcd-snapshot.db --initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 --name=etcd1 --initial-advertise-peer-urls="http://etcd1:2380" --data-dir="etcd1"
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl snapshot restore ~/etcd-snapshot.db --initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 --name=etcd2 --initial-advertise-peer-urls="http://etcd2:2380" --data-dir="etcd2"
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl snapshot restore ~/etcd-snapshot.db --initial-cluster etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380 --name=etcd3 --initial-advertise-peer-urls="http://etcd3:2380" --data-dir="etcd3"

# 将恢复的etcd数据目录拷贝到对应的节点
[root@etcd1 ~]# ll -d etcd*
drwx------ 3 root root 20 2月 27 09:36 etcd1
drwx------ 3 root root 20 2月 27 09:37 etcd2
drwx------ 3 root root 20 2月 27 09:37 etcd3
-rw-r--r-- 1 root root 20512 2月 27 09:29 etcd-snapshot.db
[root@etcd1 ~]# rsync -aXSH --delete etcd1/ etcd1:/var/lib/etcd/default.etcd/
[root@etcd1 ~]# rsync -aXSH --delete etcd2/ etcd2:/var/lib/etcd/default.etcd/
[root@etcd1 ~]# rsync -aXSH --delete etcd3/ etcd3:/var/lib/etcd/default.etcd/

# 启动etcd集群
[root@etcd1 ~]# chown -R etcd.etcd /var/lib/etcd && systemctl start etcd
[root@etcd2 ~]# chown -R etcd.etcd /var/lib/etcd && systemctl start etcd
[root@etcd3 ~]# chown -R etcd.etcd /var/lib/etcd && systemctl start etcd

# 验证集群状态
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..3}:2379 endpoint status -w table
+------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------+------------------+---------+---------+-----------+-----------+------------+
| etcd1:2379 | ade526d28b1f92f7 | 3.3.11 | 20 kB | true | 3 | 9 |
| etcd2:2379 | d282ac2ce600c1ce | 3.3.11 | 20 kB | false | 3 | 9 |
| etcd3:2379 | c931e7bd0b007b9d | 3.3.11 | 20 kB | false | 3 | 9 |
+------------+------------------+---------+---------+-----------+-----------+------------+
[root@etcd1 ~]# ETCDCTL_API=3 etcdctl --endpoints=etcd{1..3}:2379 get test
test
123

2.4 使用证书部署 etcd 集群

生成etcd证书

1
2
3
4
5
6
7
8
9
etcd下载地址:https://github.com/etcd-io/etcd/releases
cfssl下载地址:https://github.com/cloudflare/cfssl/releases

[root@etcd1 ~]# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 -o cfssl
wget [root@etcd1 ~]# https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 -o cfssl
[root@etcd1 ~]# mv cfssl_1.6.1_linux_amd64 cfssl
[root@etcd1 ~]# mv cfssljson_1.6.1_linux_amd64 cfssljson
[root@etcd1 ~]# chmod +x cfssl cfssljson
[root@etcd1 ~]# mv cfssl cfssljson /usr/bin/
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# 生成etcd证书和etcd证书的key
# 创建CA
[root@etcd1 ~]# mkdir ssl
[root@etcd1 ~]# cd ssl/
[root@etcd1 ~]# cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF

# 创建证书签名请求
[root@etcd1 ~]# cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF

生成CA证书和私钥

1
[root@etcd1 ~]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# 如果以后可能会扩容,可以在ip那多写几个ip预留出来,或者扩容时添加ip重新生成证书,重启etcd集群
[root@etcd1 ~]# cat > etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"localhost",
"192.168.223.131",
"192.168.223.137",
"192.168.223.140"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF

[root@etcd1 ~]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd

# 将ca.pem和etcd-key.pem,etcd.pem三个证书拷贝到其它etcd节点
[root@etcd1 ~]# ll *.pem
-rw-------. 1 root root 1679 2月 27 10:34 ca-key.pem
-rw-r--r--. 1 root root 1216 2月 27 10:34 ca.pem
-rw-------. 1 root root 1675 2月 27 10:40 etcd-key.pem
-rw-r--r--. 1 root root 1342 2月 27 10:40 etcd.pem

[root@etcd1 ~]# scp ca.pem etcd-key.pem etcd.pem etcd1:/etc/etcd/
[root@etcd1 ~]# scp ca.pem etcd-key.pem etcd.pem etcd2:/etc/etcd/
[root@etcd1 ~]# scp ca.pem etcd-key.pem etcd.pem etcd3:/etc/etcd/
[root@etcd1 ~]# chown etcd.etcd -R /etc/etcd/
[root@etcd2 ~]# chown etcd.etcd -R /etc/etcd/
[root@etcd3 ~]# chown etcd.etcd -R /etc/etcd/

修改各etcd节点配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 修改所有etcd节点/etc/etcd/etcd.conf配置文件(把http换成https)
[root@etcd1 ~]# vim /etc/etcd/etcd.conf
# Member
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.223.131:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.223.131:2379"
# Cluster
ETCD_ADVERTISE_CLIENT_URLS="http://etcd1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

# [Security]
ETCD_CERT_FILE="/etc/etcd/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ca.pem"
#ETCD_AUTO_TLS="false"
ETCD_PEER_CERT_FILE="/etc/etcd/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ca.pem"

# 启用所有etcd服务

查看etcd集群状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@etcd1 etcd]# ETCDCTL_API=3 etcdctl --cacert=ca.pem --cert=etcd.pem --key=etcd-key.pem --endpoints=https://etcd1:2379 member list -w table
+------------------+---------+-------+--------------------+--------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+--------------------+--------------------+
| 1f6fd35e3327767a | started | etcd1 | https://etcd1:2380 | https://etcd1:2379 |
| 2a6277f8728ef760 | started | etcd3 | https://etcd3:2380 | https://etcd3:2379 |
| 4acd0a1e9189cd7a | started | etcd2 | https://etcd2:2380 | https://etcd2:2379 |
+------------------+---------+-------+--------------------+--------------------+

[root@etcd1 etcd]# ETCDCTL_API=3 etcdctl --cacert=ca.pem --cert=etcd.pem --key=etcd-key.pem --endpoints=https://etcd{1..3}:2379 endpoint status -w table
+--------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------+------------------+---------+---------+-----------+-----------+------------+
| https://etcd1:2379 | 1f6fd35e3327767a | 3.3.11 | 20 kB | true | 3 | 9 |
| https://etcd2:2379 | 4acd0a1e9189cd7a | 3.3.11 | 20 kB | false | 3 | 9 |
| https://etcd3:2379 | 2a6277f8728ef760 | 3.3.11 | 20 kB | false | 3 | 9 |
+--------------------+------------------+---------+---------+-----------+-----------+------------+
-------------本文结束感谢您的阅读-------------