环境:MySQL 8.0.35 GTID 一主两从(141 主 + 142/143 从)+ Orchestrator 3.2.6 raft 三节点(141/142/143)+ 元数据库(143:3307 独立实例)+ VIP(192.168.195.200)+ 最小化 SMTP 邮件服务(143:25)
本文覆盖 Orchestrator.pdf 全部内容:架构原理、安装、配置文件逐项讲解、运行、监控、企业级场景模拟、常见故障模拟与解决、邮件与 VIP、知识点补充。
所有命令均注明执行节点,可直接跟随复现。
一、架构总览(生产级规划)
1被监控 MySQL 集群(GTID 一主两从): 2 192.168.195.141 (Master, rw) ──> 192.168.195.142 (Slave, ro) 3 └──> 192.168.195.143 (Slave, ro) 4 5Orchestrator raft 集群(组件自身高可用,三节点互相发现,leader 处理写操作): 6 141:3000 + 142:3000 + 143:3000 (raft 通信端口 10008) 7 8元数据库(Orchestrator 专属后端,PDF 建议"专属后端放远程"): 9 192.168.195.143:3307 独立 MySQL 实例 / orchestrator 库 10 11VIP: 192.168.195.200/24 绑定当前 Master(ens32:0),故障切换由 Hook 脚本漂移 12邮件: 143:25 最小化 SMTP(smtp_server.py),告警存 /var/mailbox/*.eml 13
为什么这样规划(对应 PDF"架构"章节):
- raft 多点非共享架构:Orchestrator 自身高可用,leader 宕机 follower 秒级接管;
- 元数据库独立实例:不与业务库混用,从库 read_only 会导致 Orchestrator 写元数据失败(本次实际踩坑验证);
- VIP+邮件通过 Hook 脚本实现:对应 PDF"通过 Hook 实现故障切换"章节。
二、环境清理(三节点执行)
在 141/142/143 上清理旧 MySQL、MHA、VIP:
1# 执行节点: 141/142/143 分别执行 2ip addr del 192.168.195.200/24 dev ens32 2>/dev/null # 清理残留VIP 3systemctl stop mysqld 2>/dev/null; systemctl disable mysqld 2>/dev/null 4rm -rf /usr/local/mysql* /data/mysql /etc/my.cnf /etc/sysconfig/mysql \ 5 /etc/systemd/system/mysqld.service /etc/masterha /var/log/masterha 6rpm -e mha4mysql-manager 2>/dev/null; userdel -r mysql 2>/dev/null 7systemctl daemon-reload 8
三、GTID 最小化主从搭建(依据《MySQL_8.0.35_GTID主从复制搭建笔记.md》)
3.1 二进制包安装(141 执行,内网分发)
1# 执行节点: 141 2cd /usr/local 3wget -q https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.35-linux-glibc2.17-x86_64.tar.xz 4tar xf mysql-8.0.35-linux-glibc2.17-x86_64.tar.xz && ln -sf mysql-8.0.35-linux-glibc2.17-x86_64 mysql 5scp mysql-8.0.35-linux-glibc2.17-x86_64.tar.xz root@192.168.195.142:/usr/local/ 6scp mysql-8.0.35-linux-glibc2.17-x86_64.tar.xz root@192.168.195.143:/usr/local/ 7 8# 执行节点: 142/143 分别执行 9cd /usr/local && tar xf mysql-8.0.35-linux-glibc2.17-x86_64.tar.xz && ln -sf mysql-8.0.35-linux-glibc2.17-x86_64 mysql 10
3.2 用户与目录(三节点)
1# 执行节点: 141/142/143 2groupadd -f mysql && useradd -g mysql -s /sbin/nologin mysql 3mkdir -p /data/mysql/3306/data && chown -R mysql:mysql /data/mysql /usr/local/mysql* 4
3.3 配置文件(GTID 最小化 + report_host 供发现)
1# 执行节点: 141 ── 主库 2cat > /etc/my.cnf << 'EOF' 3[client] 4socket = /data/mysql/3306/data/mysql.sock 5[mysqld] 6basedir = /usr/local/mysql 7datadir = /data/mysql/3306/data 8user = mysql 9port = 3306 10socket = /data/mysql/3306/data/mysql.sock 11log_error = /data/mysql/3306/data/mysqld.err 12log_timestamps = system 13log-bin = mysql-bin # 主库必须开binlog 14server-id = 1 # 三节点唯一 15gtid_mode = ON # GTID核心参数(最小化两条) 16enforce_gtid_consistency = ON 17report_host = 192.168.195.141 # 供Orchestrator通过show slave hosts自动发现(关键!) 18EOF 19 20# 执行节点: 142 / 143 ── 从库(server-id 分别为 2/3,report_host 对应修改) 21# 从库额外加只读保护: 22read_only = ON 23super_read_only = ON 24
⚠️ 重要教训(本次踩坑):从库 my.cnf 里写死
read_only=ON后,主从切换角色变化时,新提升的库重启会自动变回只读。生产做法:所有节点 my.cnf 都不写 read_only,由 Orchestrator 在提升时动态设置。切换演练后需手工同步 my.cnf 与运行角色。
3.4 systemd 服务(三节点)
1# 执行节点: 141/142/143 2cat > /etc/systemd/system/mysqld.service << 'EOF' 3[Unit] 4Description=MySQL Server 5After=network.target syslog.target 6[Install] 7WantedBy=multi-user.target 8[Service] 9User=mysql 10Group=mysql 11Type=forking 12PIDFile=/data/mysql/3306/data/mysqld.pid 13TimeoutSec=0 14ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --pid-file=/data/mysql/3306/data/mysqld.pid --daemonize $MYSQLD_OPTS 15EnvironmentFile=-/etc/sysconfig/mysql 16LimitNOFILE=65535 17Restart=on-failure 18RestartPreventExitStatus=1 19PrivateTmp=false 20EOF 21echo 'MYSQLD_OPTS=' > /etc/sysconfig/mysql 22systemctl daemon-reload && systemctl start mysqld && systemctl enable mysqld 23
⚠️ 演进坑:
Restart=on-failure会在演练 kill -9 后自动拉起 mysqld,掩盖故障检测。演练故障请用systemctl stop(clean stop 不触发 restart)。
3.5 初始化与密码(三节点)
1# 执行节点: 141/142/143 2/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --initialize 3grep 'temporary password' /data/mysql/3306/data/mysqld.err # 取临时密码 4 5# 改密(从库因super_read_only会报ERROR 1290,需init-file绕过): 6# 执行节点: 142/143 7echo 'SET GLOBAL super_read_only=0; SET GLOBAL read_only=0;' > /tmp/init.sql 8chown mysql:mysql /tmp/init.sql 9systemctl stop mysqld 10/usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --init-file=/tmp/init.sql --daemonize 11sleep 2 12/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3306/data/mysql.sock -p'<临时密码>' --connect-expired-password \ 13 -e "alter user user() identified by 'Root@123456';" 14systemctl stop mysqld 2>/dev/null; pkill mysqld 2>/dev/null; sleep 5 15systemctl start mysqld # 交回systemd管理(重要!否则演练时状态不一致) 16rm -f /tmp/init.sql 17
3.6 复制用户与 GTID 主从建立
1# 执行节点: 141 ── 创建复制+探测账号 2/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3306/data/mysql.sock -p'Root@123456' -e " 3CREATE USER 'repl'@'%' IDENTIFIED WITH mysql_native_password BY '123456'; 4GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%'; 5CREATE USER 'orch_client'@'%' IDENTIFIED WITH mysql_native_password BY 'Orch@123456'; 6GRANT RELOAD, PROCESS, SUPER, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'orch_client'@'%'; 7CREATE USER 'root'@'192.168.195.%' IDENTIFIED WITH mysql_native_password BY 'Root@123456'; 8GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.195.%' WITH GRANT OPTION; 9FLUSH PRIVILEGES;" 10 11# 执行节点: 142/143 ── 建立 GTID 复制 12/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3306/data/mysql.sock -p'Root@123456' -e " 13SET GLOBAL super_read_only=0; 14CHANGE MASTER TO MASTER_HOST='192.168.195.141', MASTER_USER='repl', MASTER_PASSWORD='123456', 15 MASTER_AUTO_POSITION=1, GET_MASTER_PUBLIC_KEY=1; 16START SLAVE; 17SET GLOBAL super_read_only=1;" 18# 验证: Slave_IO_Running: Yes / Slave_SQL_Running: Yes / Auto_Position: 1 19
3.7 测试数据
1# 执行节点: 141 2/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3306/data/mysql.sock -p'Root@123456' -e " 3CREATE DATABASE orch_test; 4USE orch_test; 5CREATE TABLE t1 (id INT PRIMARY KEY AUTO_INCREMENT, name VARCHAR(50), ts DATETIME DEFAULT CURRENT_TIMESTAMP); 6INSERT INTO t1(name) VALUES ('init_row_1'),('init_row_2'),('init_row_3');" 7# 执行节点: 142/143 验证: SELECT * FROM orch_test.t1; 三行一致即成功 8
四、元数据库准备(143 独立 3307 实例)
对应 PDF:“Orchestrator 的专属后端可以放到远程服务器上”。
⚠️ 本次踩坑:最初元数据库放在 143:3306(业务从库),其super_read_only阻止 Orchestrator 写元数据,报Error 1290 ... --read-only option,服务反复 activating。解决方案:143 增开独立 3307 实例专职元数据。
1# 执行节点: 143 2mkdir -p /data/mysql/3307/data && chown -R mysql:mysql /data/mysql/3307 3cat > /etc/my3307.cnf << 'EOF' 4[client] 5socket = /data/mysql/3307/data/mysql.sock 6[mysqld] 7basedir = /usr/local/mysql 8datadir = /data/mysql/3307/data 9user = mysql 10port = 3307 11socket = /data/mysql/3307/data/mysql.sock 12log_error = /data/mysql/3307/data/mysqld.err 13log_timestamps = system 14server-id = 337 15EOF 16/usr/local/mysql/bin/mysqld --defaults-file=/etc/my3307.cnf --initialize 17# 取临时密码改密后建库授权: 18/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3307/data/mysql.sock -p'Root@123456' -e " 19CREATE DATABASE orchestrator; 20CREATE USER 'orchestrator'@'%' IDENTIFIED WITH mysql_native_password BY 'Orch@123456'; 21GRANT ALL PRIVILEGES ON orchestrator.* TO 'orchestrator'@'%'; 22FLUSH PRIVILEGES;" 23
五、Orchestrator 3.2.6 安装(三节点)
1# 执行节点: 141(下载后内网分发) 2cd /tmp && wget -q https://github.com/openark/orchestrator/releases/download/v3.2.6/orchestrator-3.2.6-linux-amd64.tar.gz 3scp orchestrator-3.2.6-linux-amd64.tar.gz root@192.168.195.142:/tmp/ 4scp orchestrator-3.2.6-linux-amd64.tar.gz root@192.168.195.143:/tmp/ 5 6# 执行节点: 141/142/143 7mkdir -p /home/orchestrator /var/lib/orchestrator 8tar xf /tmp/orchestrator-3.2.6-linux-amd64.tar.gz -C /home/orchestrator 9cp /home/orchestrator/usr/local/orchestrator/orchestrator /home/orchestrator/ 10cp -r /home/orchestrator/usr/local/orchestrator/resources /home/orchestrator/ 11chmod +x /home/orchestrator/orchestrator 12/home/orchestrator/orchestrator --version # 输出 3.2.6 13 14# orchestrator-client(只需在常用管理机装,本次三台都装) 15cp /home/orchestrator/usr/local/orchestrator/resources/bin/orchestrator-client /usr/local/bin/ 16chmod +x /usr/local/bin/orchestrator-client 17
5.1 client 环境变量(关键,本次踩坑)
1# 执行节点: 141/142/143 2cat > /etc/profile.d/orchestrator-client.sh << 'EOF' 3export ORCHESTRATOR_API="http://127.0.0.1:3000/api" # 必须带 /api 后缀! 4export ORCHESTRATOR_AUTH_USER="orch_api" # 变量名是 AUTH_USER 不是 USER 5export ORCHESTRATOR_AUTH_PASSWORD="Orch@123456" 6EOF 7. /etc/profile.d/orchestrator-client.sh 8# 验证: orchestrator-client -c clusters 9
⚠️ 排错记录:变量名写
ORCHESTRATOR_USER无效(脚本读ORCHESTRATOR_AUTH_USER);API 不带/api后缀时部分命令 404。ORCHESTRATOR_API支持多节点空格分隔,client 自动探测 leader。
六、配置文件逐项讲解(orchestrator.conf.json)
完整配置(三节点仅 RaftBind 不同,其余一致):
1{ 2 "Debug": false, // 调试模式,生产关闭 3 "EnableSyslog": false, // 是否输出到系统日志 4 "ListenAddress": ":3000", // Web/API 监听端口 5 "HTTPAuthUser": "orch_api", // Web/API Basic认证用户 6 "HTTPAuthPassword": "Orch@123456", 7 8 "MySQLTopologyUser": "orch_client", // 探测被监控集群的账号(所有实例都要有) 9 "MySQLTopologyPassword": "Orch@123456", 10 "MySQLTopologyUseMutualTLS": false, 11 "MySQLTopologySSLSkipVerify": true, 12 "MySQLTopologyMaxPoolConnections": 3, 13 14 "MySQLOrchestratorHost": "192.168.195.143", // 元数据库地址(专属后端,可远程) 15 "MySQLOrchestratorPort": 3307, 16 "MySQLOrchestratorDatabase": "orchestrator", 17 "MySQLOrchestratorUser": "orchestrator", 18 "MySQLOrchestratorPassword": "Orch@123456", 19 20 "MySQLConnectTimeoutSeconds": 1, // 连接 MySQL 超时 21 "DefaultInstancePort": 3306, // 被监控实例默认端口 22 "DiscoverByShowSlaveHosts": true, // 优先 show slave hosts 发现(依赖report_host) 23 "InstancePollSeconds": 5, // 探测间隔(敏感度核心参数,见第十一章) 24 "SkipMaxScaleCheck": true, // 无 MaxScale binlog server 设 true 25 "UnseenInstanceForgetHours": 240, // 消失实例保留时长 26 "SnapshotTopologiesIntervalHours": 0, // 拓扑快照间隔,0禁用 27 "InstanceBulkOperationsWaitTimeoutSeconds": 10, 28 "HostnameResolveMethod": "none", // 不做DNS解析(用IP环境) 29 "MySQLHostnameResolveMethod": "@@report_host", // 用 report_host 识别主机 30 "SkipBinlogServerUnresolveCheck": true, 31 "ExpiryHostnameResolvesMinutes": 60, 32 "RejectHostnameResolvePattern": "", 33 "ReasonableReplicationLagSeconds": 10, // 延迟>10s视为异常 34 "ProblemIgnoreHostnameFilters": [], 35 "VerifyReplicationFilters": false, 36 "ReasonableMaintenanceReplicationLagSeconds": 20, // 上移/下移维护阈值 37 "CandidateInstanceExpireMinutes": 60, 38 "AuditLogFile": "", // 审计日志文件(空=写元数据库audit表) 39 "AuditToSyslog": false, 40 "RemoveTextFromHostnameDisplay": ":3306", 41 "ReadOnly": false, // 全局只读模式(false才能执行变更) 42 "AuthenticationMethod": "basic", 43 44 "FailMasterPromotionIfSQLThreadNotUpToDate": true, // SQL线程未追平禁止提升(数据安全) 45 "MasterFailoverLostInstancesDowntimeMinutes": 0, 46 "MasterFailoverDetachSlaveMasterHost": false, 47 "ApplyMySQLPromotionAfterMasterFailover": true, // 提升后自动设 read_only=0 等 48 "PreventCrossDataCenterMasterFailover": false, 49 "DetachLostSlavesAfterMasterFailover": true, // 故障中丢失的从库自动detach 50 51 "RecoverMasterClusterFilters": ["*"], // 允许自动主库恢复的集群 52 "RecoverIntermediateMasterClusterFilters": ["*"], // 允许自动中间主库恢复 53 "RecoveryPeriodBlockSeconds": 3600, // 恢复阻塞期:1小时内同集群不重复自动切换(防抖) 54 "FailureDetectionPeriodBlockMinutes": 60, // 故障检测阻塞期 55 56 "OnFailureDetectionProcesses": [ // 故障检测到时触发(报警) 57 "/home/orchestrator/hooks/notify.sh OnFailureDetection {failureType} {failureCluster} {failedHost}:{failedPort}" 58 ], 59 "PreFailoverProcesses": [ // 故障转移前触发(最后检查/杀旧主) 60 "/home/orchestrator/hooks/notify.sh PreFailover {failureType} {failureCluster} {failedHost}:{failedPort} -> {successorHost}:{successorPort}" 61 ], 62 "PostFailoverProcesses": [ // 故障转移后触发(VIP漂移+通知) 63 "/home/orchestrator/hooks/vip_failover.sh {failedHost} {successorHost}", 64 "/home/orchestrator/hooks/notify.sh PostFailover {failureType} {failureCluster} {failedHost}:{failedPort} -> {successorHost}:{successorPort}" 65 ], 66 "PostMasterFailoverProcesses": [ 67 "/home/orchestrator/hooks/notify.sh PostMasterFailover {failureCluster} promoted {successorHost}:{successorPort}" 68 ], 69 "PostIntermediateMasterFailoverProcesses": [ 70 "/home/orchestrator/hooks/notify.sh PostIntermediateMasterFailover {failureCluster} {successorHost}:{successorPort}" 71 ], 72 "PostUnsuccessfulFailoverProcesses": [ // 转移失败时触发(回退+人工介入报警) 73 "/home/orchestrator/hooks/notify.sh PostUnsuccessfulFailover FAILED {failureCluster} {failedHost}:{failedPort}" 74 ], 75 "PreGracefulTakeoverProcesses": [ 76 "/home/orchestrator/hooks/notify.sh PreGracefulTakeover {failureCluster} master {failedHost}:{failedPort}" 77 ], 78 "PostGracefulTakeoverProcesses": [ 79 "/home/orchestrator/hooks/vip_failover.sh {failedHost} {successorHost}", 80 "/home/orchestrator/hooks/notify.sh PostGracefulTakeover {failureCluster} new master {successorHost}:{successorPort}" 81 ], 82 83 "RaftEnabled": true, // raft 多点模式(生产推荐) 84 "RaftBind": "192.168.195.141", // 本节点IP(三节点各不同!) 85 "RaftDataDir": "/var/lib/orchestrator", 86 "DefaultRaftPort": 10008, // raft 通信端口(三节点一致) 87 "RaftNodes": [ // 全部节点列表 88 "192.168.195.141", "192.168.195.142", "192.168.195.143" 89 ], 90 "BackendDB": "mysql" 91} 92
七、systemd 服务与启动
1# 执行节点: 141/142/143 2cat > /etc/systemd/system/orchestrator.service << 'EOF' 3[Unit] 4Description=orchestrator: MySQL replication management and visualization 5Documentation=https://github.com/openark/orchestrator 6After=syslog.target network.target mysqld.service 7 8[Service] 9User=root 10Group=root 11Type=simple 12WorkingDirectory=/home/orchestrator 13ExecStart=/home/orchestrator/orchestrator --config=/home/orchestrator/orchestrator.conf.json http 14EnvironmentFile=-/etc/sysconfig/orchestrator 15ExecReload=/bin/kill -HUP $MAINPID 16Restart=on-failure 17RestartSec=5 18 19[Install] 20WantedBy=multi-user.target 21EOF 22systemctl daemon-reload && systemctl start orchestrator && systemctl enable orchestrator 23
7.1 raft 集群验证
1# 执行节点: 任意 2curl -s -u orch_api:Orch@123456 http://127.0.0.1:3000/api/leader-check 3# Leader 节点返回 "OK",Follower 返回 "Not leader" 4# 也可看日志: journalctl -u orchestrator | grep -iE 'raft.*(leader|follower)' 5# 本次结果: 141=Leader, 142/143=Follower 6
7.2 接入被监控集群(发现)
1# 执行节点: 任意(自动转发到leader) 2curl -s -u orch_api:Orch@123456 -X POST 'http://127.0.0.1:3000/api/discover/192.168.195.141/3306' 3# 成功返回 {"Code":"OK","Message":"Instance discovered: 192.168.195.141:3306",...} 4 5# 查看拓扑(自动发现整个集群): 6orchestrator-client -c topology -i 192.168.195.141:3306 7192.168.195.141:3306 [0s,ok,8.0.35,rw,ROW,>>,GTID] 8+ 192.168.195.142:3306 [0s,ok,8.0.35,ro,ROW,>>,GTID] 9+ 192.168.195.143:3306 [0s,ok,8.0.35,ro,ROW,>>,GTID] 10
⚠️ 本次踩坑:改密码等本地事务会让从库产生 errant GTID(拓扑显示
GTID:errant),会阻碍后续故障切换。解决:从库STOP SLAVE; RESET SLAVE ALL; RESET MASTER;后重搭复制(见 PDF"集群 GTID 复制不统一"同源问题)。
八、邮件与 VIP(第三方最小化服务)
8.1 最小化 SMTP 服务(143)
1# 执行节点: 143 —— /opt/smtp_server.py 2#!/usr/bin/env python3 3# -*- coding: utf-8 -*- 4"""最小化SMTP邮件服务器:监听25端口,收到的邮件存为.eml文件""" 5import smtpd, asyncore, os, time 6MAILDIR = '/var/mailbox' 7os.makedirs(MAILDIR, exist_ok=True) 8class MailServer(smtpd.SMTPServer): 9 def process_message(self, peer, mailfrom, rcpttos, data, **kwargs): 10 ts = time.strftime('%Y%m%d_%H%M%S') 11 fname = os.path.join(MAILDIR, 'mail_%s.eml' % ts) 12 with open(fname, 'wb') as f: 13 f.write(data if isinstance(data, bytes) else data.encode()) 14 print('[SMTP] saved %s from=%s to=%s' % (fname, mailfrom, rcpttos), flush=True) 15if __name__ == '__main__': 16 MailServer(('0.0.0.0', 25), None) 17 asyncore.loop() 18 19# systemd 服务: 20cat > /etc/systemd/system/orch-smtp.service << 'EOF' 21[Unit] 22Description=Minimal SMTP server for orchestrator alerts 23After=network.target 24[Service] 25Type=simple 26ExecStart=/usr/bin/python3 /opt/smtp_server.py 27Restart=on-failure 28RestartSec=3 29[Install] 30WantedBy=multi-user.target 31EOF 32systemctl daemon-reload && systemctl start orch-smtp && systemctl enable orch-smtp 33ss -tlnp | grep ':25 ' # 验证监听 34
8.2 Hook 脚本(三节点分发至 /home/orchestrator/hooks/)
notify.sh(通用通知:日志+邮件):
1#!/bin/bash 2# 用法: notify.sh <事件类型> <附加信息...> 3EVENT="$1"; shift 4MSG="[$(date '+%F %T')] [$EVENT] $*" 5mkdir -p /var/log/orchestrator 6echo "$MSG" >> /var/log/orchestrator/hooks.log # 本地hook日志 7python3 - << PYEOF 2>/dev/null # 发邮件到143 SMTP 8import smtplib 9from email.mime.text import MIMEText 10from email.header import Header 11msg = MIMEText("""$MSG""", 'plain', 'utf-8') 12msg['Subject'] = Header('[Orchestrator] %s 告警通知' % '$EVENT', 'utf-8') 13msg['From'] = 'orchestrator@orch.local' 14msg['To'] = 'dba-alert@orch.local' 15s = smtplib.SMTP('192.168.195.143', 25, timeout=5) 16s.send_message(msg); s.quit() 17PYEOF 18exit 0 19
vip_failover.sh(VIP 漂移):
1#!/bin/bash 2# 用法: vip_failover.sh <旧主IP> <新主IP> 3OLD_MASTER="$1"; NEW_MASTER="$2" 4VIP="192.168.195.200/24"; DEV="ens32"; LABEL="ens32:0" 5# 1.所有节点先清残留VIP(避免双VIP脑裂) 6for h in 192.168.195.141 192.168.195.142 192.168.195.143; do 7 ssh -o ConnectTimeout=3 root@${h} "ip addr del ${VIP} dev ${DEV} label ${LABEL}" 2>/dev/null 8done 9# 2.新主绑定VIP 10ssh -o ConnectTimeout=3 root@${NEW_MASTER} "ip addr add ${VIP} dev ${DEV} label ${LABEL}" 11# 3.验证并记录 12sleep 1 13CHECK=$(ssh -o ConnectTimeout=3 root@${NEW_MASTER} "ip addr show ${DEV} | grep '${VIP}'" 2>/dev/null) 14if [ -n "$CHECK" ]; then 15 echo "[$(date '+%F %T')] [VIP] ${VIP} moved ${OLD_MASTER} -> ${NEW_MASTER} OK" >> /var/log/orchestrator/hooks.log 16else 17 echo "[$(date '+%F %T')] [VIP] WARNING: ${VIP} not found on ${NEW_MASTER}" >> /var/log/orchestrator/hooks.log 18fi 19exit 0 20
1# 分发与权限(三节点): 2chmod +x /home/orchestrator/hooks/*.sh 3# 手动验证: 4/home/orchestrator/hooks/notify.sh TEST_MAIL 手动测试 # 143:/var/mailbox 出现新邮件 5/home/orchestrator/hooks/vip_failover.sh 192.168.195.141 192.168.195.142 # VIP漂到142 6
8.3 初始 VIP 绑定
1# 执行节点: 141(初始主) 2ip addr add 192.168.195.200/24 dev ens32 label ens32:0 3ip addr show ens32 | grep 192.168.195.200 # 验证 4
九、企业级场景全验证(实测记录)
场景A:拓扑重构(Refactoring)—— 级联与还原 ✅
A1. move-below:把 143 挂到 142 下(141->142->143 级联)
1# 执行节点: 141 2orchestrator-client -c move-below -i 192.168.195.143:3306 -d 192.168.195.142:3306 3# 输出: 192.168.195.143:3306<192.168.195.142:3306 4 5orchestrator-client -c topology -i 192.168.195.141:3306 6192.168.195.141:3306 [0s,ok,8.0.35,rw,ROW,>>,GTID] 7+ 192.168.195.142:3306 [0s,ok,8.0.35,ro,ROW,>>,GTID] 8 + 192.168.195.143:3306 [0s,ok,8.0.35,ro,ROW,>>,GTID] # 缩进=级联层级 9
验证(143 上 Master_Host 变为 142,双 Yes):
1# 执行节点: 143 2/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3306/data/mysql.sock -p'Root@123456' -e 'show slave status\G' | grep -E '(Master_Host|Slave_IO_Running|Slave_SQL_Running)' 3# Master_Host: 192.168.195.142 / Slave_IO_Running: Yes / Slave_SQL_Running: Yes 4
级联下写入同步验证:141 插入 2 行 -> 142/143 均 5 行,一致。
A2. move-up:还原一主两从
1# 执行节点: 141 2orchestrator-client -c move-up -i 192.168.195.143:3306 3# 输出: 192.168.195.143:3306<192.168.195.141:3306 4
知识点:
move-below要求两实例同主(兄弟关系);relocate是通用移动;GTID 拓扑下这些操作安全无损。
场景B:手动在线切换(graceful-master-takeover,141->142)✅
1# 执行节点: 141 2orchestrator-client -c graceful-master-takeover -i 192.168.195.141:3306 -d 192.168.195.142:3306 3# 输出: 192.168.195.142:3306 (成功) 4
切换后实测状态:
1192.168.195.142:3306 [0s,ok,8.0.35,rw,ROW,>>,GTID] # 142成为新主(rw) 2- 192.168.195.141:3306 [null,nonreplicating,...,downtimed] # 旧主暂时downtimed 3+ 192.168.195.143:3306 [0s,ok,...,GTID] # 143跟随新主 4
Hook 链路(141:/var/log/orchestrator/hooks.log 实录):
1[15:40:56] [PreFailover] DeadMaster 192.168.195.141:3306 ... 2[15:40:56] [PostMasterFailover] 192.168.195.141:3306 promoted 192.168.195.142:3306 3[15:40:58] [VIP] 192.168.195.200/24 moved 192.168.195.141 -> 192.168.195.142 OK 4[15:41:00] [PostGracefulTakeover] 192.168.195.141:3306 new master 192.168.195.142:3306 5
旧主 141 手动加回(PDF:“DBA 对旧主故障处理完成后,手动将旧主加回到集群”):
1# 执行节点: 141 -- takeover后旧主复制账号字段被清空,需补上再启动 2/usr/local/mysql/bin/mysql -uroot -S /data/mysql/3306/data/mysql.sock -p'Root@123456' -e " 3SET GLOBAL super_read_only=0; 4CHANGE MASTER TO MASTER_USER='repl', MASTER_PASSWORD='123456'; 5START SLAVE;" 6# 验证: 双Yes, 数据追平(orch_test.t1 三台都=6行) 7 8# 清除downtime标记: 9orchestrator-client -c end-downtime -i 192.168.195.141:3306 10
⚠️ 首次切换报错
Relocating 1 replicas ... turns to be too complex:因 143 与 142 不满足直接迁移条件。解法:先move-below把 143 挂到 142 下再切换,或直接指定目标。这是实际生产中规划切换路径的典型案例。
场景C:主库故障自动 failover(142 主宕机)✅
1# 执行节点: 142 -- 预埋测试数据 2/usr/local/mysql/bin/mysql ... -e "CREATE TABLE orch_test.failover_log (id INT PRIMARY KEY AUTO_INCREMENT, event VARCHAR(100), ts DATETIME DEFAULT CURRENT_TIMESTAMP); 3INSERT INTO orch_test.failover_log(event) VALUES ('pre_failover_1'),('pre_failover_2'),('pre_failover_3');" 4 5# 模拟宕机(注意用stop,kill -9会被systemd Restart=on-failure拉起,掩盖故障): 6systemctl stop mysqld 7
故障检测(约 2 个探测周期后):
1orchestrator-client -c replication-analysis 2# 192.168.195.142:3306 (cluster 192.168.195.142:3306): DeadMaster 3
ack 阻塞机制(PDF 核心章节,本次完整触发):
自动恢复被阻塞(142 之前被提升过,处于 active period):
1ERROR AttemptRecoveryRegistration: instance 192.168.195.142:3306 has recently been promoted 2(by failover of 192.168.195.141:3306) and is in active period. It will not be failed over. 3You may acknowledge the failure ... (-c ack-cluster-recoveries) 4
解除阻塞并确认(RecoveryPeriodBlockSeconds=3600 防抖的官方途径):
1# 执行节点: 141 2orchestrator-client -c ack-cluster-recoveries -a 192.168.195.141:3306 -reason "drill ack" 3# 或 API: curl -u orch_api:Orch@123456 "http://127.0.0.1:3000/api/ack-recovery/cluster/192.168.195.141:3306?comment=drill" 4
failover 执行(hook 日志实录):
1[15:44:16] [OnFailureDetection] DeadMaster 192.168.195.142:3306 2[15:46:17] [PreFailover] DeadMaster 192.168.195.142:3306 3[15:46:17] [PostMasterFailover] 192.168.195.142:3306 promoted 192.168.195.141:3306 4[15:46:19] [VIP] 192.168.195.200/24 moved -> 141 5[15:46:19] [PostFailover] DeadMaster 192.168.195.142:3306 6
数据零丢失验证:141 提升后 SELECT COUNT(*) FROM orch_test.failover_log = 3(宕机前 3 条全在)——GTID 复制 failover 数据完整的实证。
恢复 142/143 加回(PDF 手动流程):142 修复后 RESET SLAVE ALL; CHANGE MASTER TO MASTER_HOST='192.168.195.141' ... MASTER_AUTO_POSITION=1; START SLAVE;,143 同理重定向。
场景D:中间主库故障(IntermediateMaster)⚠️ 部分自动+手动恢复
级联 141->142->143 后停 142:
1orchestrator-client -c replication-analysis 2# 192.168.195.141:3306: MasterSingleReplicaDead 3# 192.168.195.142:3306: DeadIntermediateMasterWithSingleReplica <- IM故障正确识别 4
自动恢复未触发的根因排查(重要实战经验):
curl /api/recover/192.168.195.142/3306返回Recovery not attempted;- 查分析详情:143 的
Slave_IO_Running=false(60 秒重连退避中),orchestrator 读到UsingOracleGTID=false,将 IM 子树判定为非 GTID 拓扑,要求 Pseudo-GTID(未配置)而拒绝自动恢复; - 这正是 PDF"集群 gtid 复制不统一"案例的变体:故障瞬间的从库 IO 退避状态会误导 GTID 判定。
手动恢复路径(生产 SOP):
1# 方法1: orchestrator relocate-replicas(适合从库状态正常时) 2orchestrator-client -c relocate-replicas -i 192.168.195.142:3306 -d 192.168.195.141:3306 3# 方法2: 直接MySQL层把143重定向(本次采用,简单可靠) 4# 执行节点: 143 5STOP SLAVE; CHANGE MASTER TO MASTER_HOST='192.168.195.141', ..., MASTER_AUTO_POSITION=1; START SLAVE; 6
场景E:Orchestrator 自身高可用(raft leader 切换)✅
1# 执行节点: 141(当前leader) 2systemctl stop orchestrator 3sleep 20 4# 执行节点: 142/143 分别检查: 5curl -s -o /dev/null -w "%{http_code}" -u orch_api:Orch@123456 http://127.0.0.1:3000/api/leader-check 6# 142 -> 200 (新leader) 143 -> 404(Not leader响应) 7# 新leader的拓扑视图完整,client自动跟随: 8orchestrator-client -c topology -i 192.168.195.141:3306 # 正常输出 9 10# 恢复141后: 11systemctl start orchestrator # 141回到集群,成为Follower 12curl .../api/health | jq '.Details | {IsActiveNode, ActiveNodeHostname}' 13# {"IsActiveNode": false, "ActiveNodeHostname": "192.168.195.142:10008"} <- 142继续服务 14
场景F:auto_position=0 引发 Pseudo-GTID 需求(PDF 案例)✅
1# 执行节点: 143 -- 模拟某从库被改成位置点复制 2STOP SLAVE; 3CHANGE REPLICATION SOURCE TO SOURCE_AUTO_POSITION=0, SOURCE_LOG_FILE='mysql-bin.000004', SOURCE_LOG_POS=4; 4START SLAVE; 5
PDF 同款定位 SQL(元数据库):
1-- 执行节点: 143(元数据库3307) 2SELECT hostname, port, oracle_gtid FROM orchestrator.database_instance; 3+-----------------+-------------+ 4| hostname | oracle_gtid | 5+-----------------+-------------+ 6| 192.168.195.141 | 0 | 7| 192.168.195.142 | 1 | 8| 192.168.195.143 | 1 | <- auto_position=1的从库标记为1 9+-----------------+-------------+ 10-- oracle_gtid=0 的节点会导致后续拓扑操作走 Pseudo-GTID 分支(未配置则失败) 11
修复(PDF:去对应节点 change master 把 auto_position 改回 1):
1# 执行节点: 143 2STOP SLAVE; CHANGE REPLICATION SOURCE TO SOURCE_AUTO_POSITION=1; START SLAVE; 3
注:主库 oracle_gtid=0 是正常现象(主库不使用复制),只关注从库。
十、Web UI 与 API 操作指引(需手动页面操作部分)
10.1 Web 访问
浏览器打开(三个节点任一,只有 leader 可写):
1http://192.168.195.141:3000 (认证: orch_api / Orch@123456) 2http://192.168.195.142:3000 3http://192.168.195.143:3000 4
10.2 手动页面操作步骤(老板请按此操作)
- 登录:输入 Basic 认证
orch_api / Orch@123456 - 查看集群:首页 Clusters 列表点击
192.168.195.141:3306 - 看拓扑图:Web 页面显示 141 主 + 142/143 从的树状图,绿色=正常
- 常用页面操作:
- 拖拽从库节点到另一个主库节点 = relocate(对应命令行 move-below)
- 点击节点 -> “Properties” 查看实例详情
- 顶部 “Audit” 页面查看所有操作审计
- “Audit / Recovery” 页面可对故障恢复进行 ack 确认(对应 ack-cluster-recoveries)
- 模拟演练(页面版):发现 -> 拖拽重构 -> 观察拓扑变化
10.3 API 速查(实测全部可用)
1A="orch_api:Orch@123456"; B="http://127.0.0.1:3000/api" 2curl -s -u $A $B/clusters # 集群列表 3curl -s -u $A $B/topology/192.168.195.141/3306 # 拓扑JSON 4curl -s -u $A $B/instance/192.168.195.143/3306 # 实例详情 5curl -s -u $A $B/replication-analysis # 故障分析 6curl -s -u $A $B/audit-recovery # 恢复审计 7curl -s -u $A $B/health # 节点健康/raft状态 8curl -s -u $A -X POST $B/discover/192.168.195.141/3306 # 发现实例 9curl -s -u $A -X POST $B/forget/192.168.195.142/3306 # 忘记实例 10curl -s -u $A "$B/ack-recovery/cluster/192.168.195.141:3306?comment=xx" # ack 11
10.4 orchestrator-client 常用命令(实测)
1orchestrator-client -c clusters # 集群列表 2orchestrator-client -c topology -i <host>:3306 # 拓扑 3orchestrator-client -c all-instances # 全部实例 4orchestrator-client -c which-cluster -i <host>:3306 # 实例归属 5orchestrator-client -c replication-analysis # 故障分析 6orchestrator-client -c discover -i <host>:3306 # 发现 7orchestrator-client -c forget -i <host>:3306 # 忘记 8orchestrator-client -c move-below -i <从>:3306 -d <新主>:3306 # 级联挂载 9orchestrator-client -c move-up -i <从>:3306 # 上移一层 10orchestrator-client -c relocate-replicas -i <实例>:3306 -d <目标>:3306 11orchestrator-client -c graceful-master-takeover -i <主>:3306 -d <新主>:3306 # 在线切换 12orchestrator-client -c recover -i <故障实例>:3306 # 手动恢复(忽略阻塞) 13orchestrator-client -c force-master-failover -i <主>:3306 # 强制切换 14orchestrator-client -c begin-downtime -i <实例>:3306 -reason "维护" -duration 30m 15orchestrator-client -c end-downtime -i <实例>:3306 16orchestrator-client -c ack-cluster-recoveries -a <集群> -reason "xx" 17orchestrator-client -c register-candidate -i <实例>:3306 --promotion-rule=prefer 18
十一、故障敏感度调整(PDF 章节)
- InstancePollSeconds(默认5):探测间隔。调大可过滤短抖动(如30秒间隔可忽略大部分10秒级抖动)。
- ReasonableInstanceCheckSeconds:单次探测允许的最长查询时间。实例能连但查询慢时,调大此参数容忍。
- MySQLDiscoveryReadTimeoutSeconds(默认10):discover 查询超时,应 >= ReasonableInstanceCheckSeconds。
- RecoveryPeriodBlockSeconds(本次=3600):同集群自动恢复阻塞期,防级联故障抖动。
- FailureDetectionPeriodBlockMinutes(本次=60):故障重复检测阻塞期。
生产建议:小集群 InstancePollSeconds=5 保持快速检测;抖动频繁的环境调到 10~15;配合 VIP 健康检查阈值一起调。
十二、常见故障案例与解决(PDF 全部案例 + 本次新增实测)
| # | 故障 | 现象 | 解决 |
|---|---|---|---|
| 1 | 元数据库用只读从库 | orchestrator 起不来, journalctl 报 Error 1290 --read-only option | 元数据库用独立可写实例(本次143:3307) |
| 2 | errant GTID | 拓扑显示 GTID:errant, failover 被阻 | 从库 STOP SLAVE; RESET SLAVE ALL; RESET MASTER; 重搭复制 |
| 3 | 主从版本不一致 | 切换失败(版本校验不匹配) | 同集群保持同版本(PDF案例) |
| 4 | 复制异常+主宕机 | 数据不一致风险 | 切换前检查 SQL 线程 Yes(FailMasterPromotionIfSQLThreadNotUpToDate=true) |
| 5 | 从库 auto_position=0 | 拓扑操作走 Pseudo-GTID 失败 | 元数据库 SELECT * FROM database_instance WHERE oracle_gtid=0 定位, CHANGE MASTER 改回 |
| 6 | IM 故障但从库 IO 退避 | 自动恢复返回 not attempted | 手动 relocate-replicas 或 MySQL 层重定向; 预防:Pseudo-GTID 或保持从库健康 |
| 7 | 演练时 kill -9 被 systemd 拉起 | 故障被掩盖,无法触发检测 | 演练用 systemctl stop; 或临时 systemctl edit 去掉 Restart |
| 8 | 旧主重启变只读 | my.cnf 遗留 read_only=ON, 重启后新主只读 | 所有节点 my.cnf 不写 read_only, Orchestrator 动态管理 |
| 9 | 阻塞期内二次故障 | 恢复被 active period 阻塞 | ack-cluster-recoveries -a <集群> -reason "xx" 解除 |
| 10 | orchestrator-client 连不上 | Cannot access orchestrator | 检查 ORCHESTRATOR_API 带 /api; 变量名 ORCHESTRATOR_AUTH_USER/PASSWORD |
| 11 | graceful 切换报 too complex | 兄弟从库不满足直接迁移 | 先 move-below 调整结构再切换, 或手动分步 |
十三、知识点补充(超出 PDF 的实战总结)
- raft 写转发:任何节点的 API/写操作自动转发给 leader,client 配多个地址自动探测。Follower 上
leader-check返回非200。 - 拓扑发现机制(PDF 原理验证):
DiscoverByShowSlaveHosts=true时靠从库report_host参数(show slave hosts);否则走information_schema.processlist的 Binlog Dump。本次 report_host 直配 IP,发现秒级。 - GTID errant 的危害链:改密码等本地事务 → 从库 gtid_executed 多出主库没有的事务 → orchestrator 标记 errant → failover 候选被排除或恢复失败。预防:从库管理操作一律走 SQL_TIMEOUT 会长事务的方式,或操作前
SET SESSION sql_log_bin=0。 - failover 数据零丢失的条件:GTID + 从库 SQL 线程追平(FailMasterPromotionIfSQLThreadNotUpToDate=true 强制)。本次 failover_log 3条全在实证。
- VIP Hook 时序:PostFailoverProcesses 数组按序执行,vip_failover.sh 放第一个保证业务尽快恢复;脚本里"先全网清理再绑定"避免双 VIP 脑裂。
- 元数据库表速查:
database_instance(实例状态) /topology_recovery(恢复记录) /topology_failure_detection(故障检测) /database_instance_downtime(维护标记) /audit(审计) /cluster_alias(集群别名)。 - 注册优先候选:
register-candidate --promotion-rule=prefer可指定故障切换优先提升某从库(prefer/neutral/exclude)。
十四、本次验证结果总表
| 场景 | 验证内容 | 结果 |
|---|---|---|
| 安装 | 三节点 orchestrator 3.2.6 + systemd + client | ✅ |
| 配置 | raft 模式 + Hook + 认证 + 元数据库 | ✅ |
| 运行 | leader 选举(141)、写转发、发现集群 | ✅ |
| 监控 | 拓扑实时刷新、replication-analysis、problems | ✅ |
| 场景A | move-below 级联 + move-up 还原 + 数据同步 | ✅ |
| 场景B | graceful-master-takeover 全链路(VIP/邮件/旧主回收) | ✅ |
| 场景C | DeadMaster 自动 failover + ack 阻塞解除 + 数据零丢失 | ✅ |
| 场景D | IM 故障识别 + GTID 判定陷阱 + 手动恢复 | ⚠️ 自动恢复受限(已分析根因,给出SOP) |
| 场景E | raft leader 宕机 follower 接管 + 回归 | ✅ |
| 场景F | auto_position=0 定位与修复(元数据库SQL) | ✅ |
| 邮件 | 9封告警邮件实测送达 | ✅ |
| VIP | 4次漂移实测(含hook自动) | ✅ |
环境 141+142+143 当前状态:141 主库 + 142/143 从库,全部双 Yes;orchestrator 三节点 active(leader=142);VIP 在 141;元数据库 143:3307。
说明:场景C后 leader 为 141→场景E后为 142,最终 142 为 leader,141 为 follower( IsActiveNode:false )。业务主库仍为 141。

