salt-minion 和salt-master 认证机制如下:

1 minion端生成一个秘钥对,并产生一个ID值,minion服务会安装ID值命名的公钥发送给 master ,直到接受为止

2 master认证完毕后,会将minion 端发送来的,以ID值命名的公钥存放在 /etc/salt/pki/master/minions 目录中

3 master认证完毕后,会将自身的公钥发送给 minion,并存储为 /etc/salt/pki/minion/minion_master.pub.

•master 秘钥对默认存储在/etc/salt/pki/master/master.pub /etc/salt/pki/master/master.pem(salt master私钥)

•master 端认证的公钥存储在:/etc/salt/pki/master/minions/

•minion 秘钥对默认存储在/etc/salt/pki/minion/minion.pub /etc/salt/pki/minion/minion.pem(salt minion私钥)

•minion 存放的master公钥/etc/salt/pki/minion/minion_master.pub

实际遇到的问题:

minion --> master 认证成功,pubkey成功写入

master--> minion 认证失败,无法生成minion_master.pub

查看日志内容如下:

# cat /var/log/salt/minion

2022-04-18 06:47:31,907 [tornado.application:640 ][ERROR   ][4767] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f88980802f0>, <s
alt.ext.tornado.concurrent.Future object at 0x7f88980ed1d0>)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 654, in _authenticate
    creds = yield self.sign_in(channel=channel)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 780, in sign_in
    ret = self.handle_signin_response(sign_in_payload, payload)
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 792, in handle_signin_response
    clear_signature = payload["sig"]
KeyError: 'sig'
2022-04-18 06:48:31,500 [salt.minion      :1095][ERROR   ][4767] Minion unable to successfully connect to a Salt Master.
2022-04-18 06:48:50,692 [salt.utils.verify:591 ][WARNING ][5032] Insecure logging configuration detected! Sensitive data may be logged.

使用systemctl status salt-minion 报错如下

Apr 18 06:12:42 xxxxx salt-minion[31331]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Apr 18 06:12:42 xxxxx salt-minion[31331]: raise_exc_info(self._exc_info)
Apr 18 06:12:42 xxxxx salt-minion[31331]: File "<string>", line 4, in raise_exc_info
Apr 18 06:12:42 xxxxx salt-minion[31331]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Apr 18 06:12:42 xxxxx salt-minion[31331]: yielded = self.gen.send(value)
Apr 18 06:12:42 xxxxx salt-minion[31331]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 780, in sign_in
Apr 18 06:12:42 xxxxx salt-minion[31331]: ret = self.handle_signin_response(sign_in_payload, payload)
Apr 18 06:12:42 xxxxx salt-minion[31331]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 792, in handle_signin_response
Apr 18 06:12:42 xxxxx salt-minion[31331]: clear_signature = payload["sig"]
Apr 18 06:12:42 xxxxx salt-minion[31331]: KeyError: 'sig'

手动删除minion公钥私钥,并删除master上的认证,重新在master上认证

salt-key -L

salt-key -a xxxx -y

salt-key -L

认证成功,

查看公钥内容是否和minion一致,切换到minion服务器,确认公钥pub内容一致

查看minion_master.pub 生成情况,依旧未生成,只有minion的公钥,私钥

查看salt-minion版本

[root@xxx minion]# salt-minion --versions-report
Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-957.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core
 

查看salt-master 版本

[root@xxxxx master]# salt-master --versions-report
Salt Version:
          Salt: 3004
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4
 
System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-957.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

版本号不一致

minion  是3004.1-1

master是3004.1

查看salt-minion有问题的服务器rpm文件版本

salt-minion.noarch                   3004.1-1.el7            salt-latest-repo

其他正常的服务器版本:

yum list | grep salt-minion

salt-minion.noarch                          3004-1.el7                 @salt-latest-repo

发现无法回滚到当前3004.1版本

由于是线上环境,无法升级master到最新版本,所以将其他服务器的相关路径拷贝到有问题的salt-minion服务器上(按照向下兼容理论,3004.1-1应该兼容3004-1 所以非生产环境,可以升级master来处理该问题)

yum remove salt-minion

处理方法:

1. 将旧版本的内容传输到minion服务器上

     传输目录有:

      /etc/salt

      /usr/bin/salt-minion

      /usr/lib/python3.6

2. master上手动删除相关认证,salt-key -d xxxx -y

3. 完成后,重启salt-minion

4. 再次再master上认证

     salt-key -L

     salt-key -a xxxx -y

5. 再到minion服务器确认,minion_master.pub 生成成功

注意:

1. master确保4505 和4506 端口没有被墙

echo exit | telnet master_ip 4505
echo exit | telnet master_ip 4506

2. minion删除公钥想要自动生成新的公钥,需要删除公钥,私钥,单独删除公钥,不会重新生成

没有pub报错内容如下

# systemctl status salt-minion
Apr 18 05:46:26 xxxxx salt-minion[27741]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
Apr 18 05:46:26 xxxxx salt-minion[27741]: yielded = next(result)
Apr 18 05:46:26 xxxxx salt-minion[27741]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 763, in sign_in
Apr 18 05:46:26 xxxxx salt-minion[27741]: sign_in_payload = self.minion_sign_in_payload()
Apr 18 05:46:26 xxxxx salt-minion[27741]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 948, in minion_sign_in_payload
Apr 18 05:46:26 xxxxx salt-minion[27741]: with salt.utils.files.fopen(self.pub_path) as f:
Apr 18 05:46:26 xxxxx salt-minion[27741]: File "/usr/lib/python3.6/site-packages/salt/utils/files.py", line 385, in fopen
Apr 18 05:46:26 xxxxx salt-minion[27741]: f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage
Apr 18 05:46:26 xxxxx salt-minion[27741]: FileNotFoundError: [Errno 2] No such file or directory: '/etc/salt/pki/minion/minion.pub'
Apr 18 05:47:26 xxxxx salt-minion[27741]: [ERROR   ] Minion unable to successfully connect to a Salt Master.

3. 传输前,需要将源路径备份或删除,建议备份

    传输后,需要修改/etc/salt/minion目录内容为指定服务器的内容

4. 使用salt-minion的debug,显示的内容如下,看不出问题的

[root@xxxx salt]#     salt-minion -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.log_handlers.sentry_mod' from '/usr/lib/python3.6/site-packages/salt/log/handlers/sentry_mod.py'>
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Minion "xxxxx"
[INFO    ] An instance is already running. Exiting the Salt Minion
[INFO    ] Shutting down the Salt Minion
[DEBUG   ] Stopping the multiprocessing logging queue listener
[DEBUG   ] closing multiprocessing queue
[DEBUG   ] joining multiprocessing queue thread
[DEBUG   ] Stopped the multiprocessing logging queue listener
The Salt Minion is shutdown.

5. 手动生成minion_master.pub文件的话,minion的log报错如下

[salt.crypt       :788 ][ERROR   ][28430] Sign-in attempt failed: {'enc': 'pub', 'pub_key': '-----BEGIN PUBLIC KEY-----\nMIIBxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx……

[salt.minion      :1149][ERROR   ][28430] Error while bringing up minion for multi-master. Is master at 10.28.194.84 responding?