自动化运维管理fabric

钱佩锌 發表於 2023-9-16 00:00:00

<h2 id="如何使用fabric_<a href=" http: target="_blank">自动化日常管理任务和部署">如何使用fabric 自动化日常管理任务和部署</h2>
<p>自动化，批量化是作为管理员，或者运维人员必须面临的问题。自动化和批量化也有很多方式，可以用单一工具也可以自己写shell脚本，甚至可以开发出来一套完备的任务管理系统。其实我们大多时候可以在一台主机上面通过ssh来控制所有机器，来完成我们的任务工作。是否有这样的工具来支持我们呢？</p>
<p><img src="https://zhuji.jb51.net/uploads/img/20230519/f46c930c4cafdf17b35ca426dcc966e5.jpg" width="150" height="117"></p>
<h3 id="fabric_常用接口">fabric 常用接口</h3>
<p>fabric是对ssh的一个集成工具，对我们而言只需要使用相应的接口，来高效的完成工作，我们常用到的功能基本是：本地或者远端执行命令，分发文件，收集文件，还有一些权限相关的操作。这些fabric都给我们提供了对应的接口。<br>
如下所示：</p><pre class="brush:bash;toolbar:false">run (fabric.operations.run)
sudo (fabric.operations.sudo)
local (fabric.operations.local)
get (fabric.operations.get)
put (fabric.operations.put)
prompt (fabric.operations.prompt)
reboot (fabric.operations.reboot)</pre><p></p>
<h3 id="fabric_还提供了上下文管理器">fabric 还提供了上下文管理器</h3>
<p>接口部分提供了命令运行的方式，不过都无法保持上下文关系，为了解决这个问题，fabric的context manager 就派上了用场：</p><pre class="brush:bash;toolbar:false">cd (fabric.context_managers.cd)
lcd (fabric.context_managers.lcd)
path (fabric.context_managers.path)
settings (fabric.context_managers.settings)
prefix (fabric.context_managers.prefix)</pre><p></p>
<h3 id="fabric_安装">fabric 安装</h3>
<p></p><pre class="brush:bash;toolbar:false">easy_install fabric</pre><p></p>
<h2 id="fabric_编程模型介绍">fabric 编程模型介绍</h2>
<p>由于fabric是基于python的，所以写fabric脚本就是写python脚本，你可以像写python脚本一样，可以依赖其他模块或者其他工具来完成工作。Fabric 脚本，通过fab工具运行fabric python脚本。fab工具默认执行fabfile.py ,也可以通过-f 参数指定脚本文件名。fabric优势多多，简单，方便，日志输出清晰，命令<br>
中可以使用AWK 命令下面我们看一个 hello world 程序。</p><pre class="brush:bash;toolbar:false">from fabric.api import *
def helloworld(who='world'):
print"Hello {0}!".format(who)
def helloworld1(you='world',me='ruiaylin'):
print"Hello {0}! i am {1} ! ".format(you,me)</pre><p>执行命令(其中参数的传递直接跟在任务后跟变量名和参数)：</p><pre class="brush:bash;toolbar:false">➜fabricfab -f helloword.pyhelloworld
Hello world!
Done.
➜fabricfab -f helloword.pyhelloworld1:you='ruichao',me='ruiaylin'
Hello ruichao! i am ruiaylin !
Done.</pre><p></p>
<h2 id="fabric主要接口方法">fabric主要接口方法</h2>
<p>我们已经看了一个简单例子下面我们来看一下fabric的主要接口。</p>
<h3 id="run_(fabric-operations-run)">run (fabric.operations.run)</h3>
<p>Fabric 中使用最多的就是 run 方法了。run是用来在一台或者多台远程主机上面执行shell 命令。</p>
<ul>
<li>方法的返回值是可以通过变量来进行捕获</li>
<li>可以通过变量的.failed 和 .succeeded 来检查命令是否执行成功</li>
<li>还有一个很赞的就是 run 方法中执行命令的时候，可以支持awk 很给力</li>
</ul>
<p>使用方法：</p><pre class="brush:bash;toolbar:false"># creat a directory
run(" mkdir /tmp/testdir/ -p ")
# check process
result = run("ps -ef |grep mysqld|grep -v safe |grep -v grep| wc -l "
#Check if command
result.failed</pre><p></p>
<h3 id="sudo_(fabric-operations-sudo)">sudo (fabric.operations.sudo)</h3>
<p>使用 sudo 命令执行对顶的命令。使用方法与run 类似。</p>
<h3 id="local_(fabric-operations-local)">local (fabric.operations.local)</h3>
<p>local 命令是执行本机的命令或者脚本.使用方法和run 还有sudo类似，但是有一个区别<br>
就是：捕获结果的时候，是通过指定 capture=False 或者capture=True来确定。来看<br>
实例：</p><pre class="brush:bash;toolbar:false"># example like this ：
def helloworld(who='world'):
print"Hello {0}!".format(who)
yy = local(" pwd ", capture=False)
print 'start :yy = ' , yy , ' : ::',yy.succeeded
zz = local(" pwd ", capture=True)
print 'start :zz = ' , zz , ' : ::',zz.succeeded
#result ：
➜fabricfab -f helloword.pyhelloworld-H 10.211.55.3 -u root
Executing task 'helloworld'
Hello world!
local:pwd
/Users/ruiaylin/Documents/workpython/fabric
start :yy = : :: True
local:pwd
start :zz =/Users/ruiaylin/Documents/workpython/fabric: :: True</pre><p></p>
<h3 id="get_(fabric-operations-get)">get (fabric.operations.get)</h3>
<p>get 方法是从远程主机 copy file 到本地,功能跟scp一样。可以从远程主机下载<br>
备份，或者日志文件等等。</p>
<ul>
<li>通过参数 remote_path 指定远程文件的路径</li>
<li>通过参数 local_path 指定远程文件的路径</li>
</ul>
<p>使用方法如下：</p><pre class="brush:bash;toolbar:false"># Download some logs
get(remote_path="/tmp/xxx.log", local_path="/tmp/xxx.log")
# Download a database back-up
get("/backup/db.gz", "./db.gz")</pre><p></p>
<h3 id="put_(fabric-operations-put)">put (fabric.operations.put)</h3>
<p>某些需要上传和分发文件的时候，put命令就派上了用场，使用方式类似 get。也同样可以<br>
通过.failed .succeeded进行命令是否执行成功的判断。</p>
<ul>
<li>local_path - 本地路径</li>
<li>remote_path - 远程路径</li>
<li>mode - 文件属性</li>
</ul>
<p>如下例子：</p><pre class="brush:bash;toolbar:false">upload = put("requirements.txt", "requirements.txt", mode=0664)</pre><p></p>
<h2 id="并行执行">并行执行</h2>
<p>目前官方来看 1.X 版本的fabric 并行执行的时候不是thread safe的。如果需要并行执行task。需要在方法上面使用注解 @parallel 为了防止管控机器上面过多的并发任务可以通过 @parallel(pool_size=5)来设置. 并行的执行输出都会输出到一个终端上面，比较混乱。最好是写到日志，以task为维度。跟下面的代码类似。</p>
<h2 id="MySQL_安装实例">MySQL 安装实例</h2>
<p>安装步骤如下</p>
<ul>
<li>获取主机ip</li>
<li>check主机可达性</li>
<li>检查linux平台详情</li>
<li>是否有运行的mysql实例</li>
<li>如果有获取对应的端口</li>
<li>检查是否和要安装的端口冲突</li>
<li>处理mysql用户以及属组</li>
<li>处理安装相关目录和权限</li>
<li>copy 安装包到目标机</li>
<li>解压处理，将主要软件工具软连接到path路径中</li>
<li>生成对应标准配置文件并分发到目标机对应目录</li>
<li>初始化数据库</li>
<li>启动数据库</li>
<li>基本步骤安装完毕</li>
</ul>
<p>基本脚本如下：</p>
<p>script 1 sub task :</p><pre class="brush:bash;toolbar:false">from fabric.api import *
from fabric.colors import green,red,blue,cyan,yellow
import os , sys
import socket
import datetime
import logging
import logging.handlers
#get logger for logging
def initLoggerWithRotate():
logname=''.join(env.host_string.split('.'))+'.log'
logFileName="logs/%s"%logname
logger = logging.getLogger("fabric")
formater = logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s","%Y-%m-%d %H:%M:%S")
file_handler = logging.handlers.RotatingFileHandler(logFileName, maxBytes=104857600, backupCount=5)
file_handler.setFormatter(formater)
stream_handler = logging.StreamHandler(sys.stderr)
logger.addHandler(file_handler)
logger.addHandler(stream_handler)
logger.setLevel(logging.INFO)
return logger
#mkdir
def runmkdir(dir):
run(''' mkdir -p %s '''%dir)
#stp 1 check host
def checkhost(logger):
host = env.host_string
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
flag_c = 0
try:
      s.connect((host, 22))
      flag_c = 1
      logger.info( green( ' --> host %s can be reachable ' %host ) )
except socket.error as e:
      logger.warning( yellow( ' --> Error on connect %s' %e ) )
s.close()
return flag_c
#stp 2 check alive instance on target host
def checkmysqlinstance(logger):
try:
   wc = run(''' ps -ef |grep mysqld|grep-v safe | grep -v grep | wc -l''')
   if int(wc) > 0:
         logger.warning(yellow( ' --> %sinstance exist on the target host'%wc ))
         portraw = run('''ps -ef |grep mysqld|grep -v safe |grep -v grep|awk ' {for(i=1;i<=NF;i++){if($i ~/--port/ ){print $i}}}' |awk -F '=' '{print $2}'
         ''')
         ports =
         logger.warning( yellow( ' --> existing instance port : [ %s ] '%( ','.join( ports ))))
         if port in ports:
            logger.error( red( ' --> Install port %s exist , install failed '%port))
            logger.error( red( ' <<<exit>>>>>task on host %s stop & exit() '%thost))
            sys.exit()
except Exception, e:
   logger.warning(yellow( ' --> checkmysqlinstance() exception : %s '%e ))
   raise e
#stp 3 initdir for installation
def createUser(logger,user='mysql',group='dba'):
try:
   if int(run('grep "^mysql" /etc/passwd|wc -l')) == 0 :
         run('groupadd dba ')
         run('useradd -c "mysql software owner" -g dba -G dba mysql')
         run('mkdir -p /home/mysql ; chown -R mysql.dba /home/mysql ')
         logger.info(cyan( ' --> create user [ mysql ] in group [ dba ]success ' ))
   else :
         logger.info(yellow ( ' --> user [ mysql ] in group [ dba ] exist & skip' ))
except Exception, e:
   logger.warning(yellow( ' --> createUser() exception : %s '%e ))
   raise e
#stp 4 initail directory for mysql
def initdir(logger,port=3306):
try :
   logger.info( green( ' --> begin to create dirs for installation '))
   datadir='/data/'
   logdir ='/log/'
   mandir = 'mysql%s'%port
   subddir ='/data/mysql%s/{data,log,run,tmp}'%(port)
   subldir ='/log/mysql%s/{binlog,iblog}'%(port)
   #data
   ck1 = run(' df -vh| grep/data | wc -l ')
   if ck1== 0 :
         logger.error(green(' --> no /data/ partition exist' ) )
         #sys.exit()
   if int( run(' ls /| grep/data | wc -l ')) == 0 or int( run(' ls /data/ | grep -w %s | wc -l '%mandir) ) == 0 :
         runmkdir(subddir)
         logger.info(green(' --> /data/*** create Ok ' ) )
   else :
         logger.info(green(' --> /data/mysql%s exsit '%port ))
         logger.info(green(' --> pls,handle it and restart this task '))
         sys.exit()
   #log
   ck2 = run(' df -vh | grep /log/| wc -l')
   if int( run(' df -vh | grep /log/| wc -l') ) == 0and int( run(' ls / | grep -w log| wc -l') ) == 0:
         logger.warning( yellow(' --> no /log/ partition exist') )
         logger.warning( yellow(' --> create link for /log/ --> /data/log/') )
         runmkdir('/data/log')
         run('ln -s /data/log/log ')
         runmkdir(subldir)
         logger.info(green(' --> /log/*** create Ok ' ) )
   else :
         ifint(run(' ls /log/ | grep -w %s | wc -l '%mandir)) == 0:
            runmkdir(subldir)
            logger.info(green(' --> /log/*** create Ok ' ) )
         else :
            logger.info(yellow(' --> /log/mysql%s exsit '%port ))
            logger.error(red(' --> pls,handle it and restart this task ' ))
            sys.exit()
   #change
   runmkdir('/data/tmp')
   logger.info(green(' --> change dirs owner&privs start'))
   run('chown -R mysql:dba /data/*')
   run('chown -R mysql:dba /log')
   logger.info(green(' --> change dirs owner&privs done'))
except Exception, e:
   logger.warning(yellow( ' --> initdir() exception : %s '%e ))
   raise e
#stp 5 put mysql install package
def copymysql(logger,version='5.7'):
try:
   dits = {
   'ubuntu':'mysql-server_5.6.21-1ubuntu12.04_amd64.deb-bundle.tar',
   'centos':'mysql-server.tar.gz'
   }
   issue = run ('cat /etc/issue')
   ss = issue.lower()
   logger.info( green( ' %s '%ss))
   if int ( run( ' ls /usr/local/ | grep mysql | wc -l ') ) > 0 :
         logger.info( yellow( ' --> mysql software installed , skip ' ))
         return
   plats = dits.keys()
   for x in plats:
         if ss.find(x) != -1:
            logger.info( green( ' --> the target host platform is %s'% x ) )
            put( local_path="configs/%s"%dits,remote_path="/tmp/%s"%dits )
            logger.info( green( ' --> tar the ball to prop dir '))
            run( 'tar zxvf /tmp/%s -C /usr/local/ '%dits )
            run( 'ln -s /usr/local/%s/usr/local/mysql'%dits[:-7] )
            break
except Exception, e:
   logger.warning(yellow( ' --> copymysql() exception : %s '%e ))
   raise e
#gen my.cnf file
def getnewServerId(logger,port):
host = env.host_string
print 'getnewServerId : ',host
pics = host.split('.')
a=int(pics)
b=int(pics)
c=int(pics)
d=int(pics)
suf = int(port) % 256
server_id =b * 256 * 256 * 256 + c * 256 * 256 + d * 256 + suf
logger.info( cyan( ' --> gen server_id done , %s %s is %s '%( host , port , server_id) ) )
return server_id
def genmycnf(logger,port=3306,itype='h'):
host = env.host_string
bps={
"a":"48|32|3100|3000",
"b":"62|40|4600|4500",
'c':'94|64|7600|7500',
'd':'94|32|3100|3000',
'e':'125|75|10100|10000',
'f':'188|120|15100|15000',
'g':'188|60|7600|7500',
'h':'1|256M|800|750'
}
try:
   myfile=''.join(host.split('.'))+'.cnf'
   cpmycnf="""cp configs/my.cnftmp/%s """%myfile
   local( 'rm -ftmp/%s'%myfile)
   local("cp configs/my.cnf tmp/%s "%myfile )
   sid=getnewServerId(logger,port)
   keys=bps.keys()
   bpxs=bps
   mem,bpsize,maxc,maxuc=bpxs.split('|')
   if bpsize[-1] != "M":
         bpsize = bpsize +'g'
   chrgcmd="""sed -i -e "s/3306/%s/g" -e "s/server_id=10000/server_id=%s/g" -e "s/=32g/=%s/g" -e "s/max_connections=3100/max_connections=%s/g" -e "s/max_user_connections=3000/max_user_connections=%s/g" tmp/%s """
   local( chrgcmd%(port,sid,bpsize,maxc,maxuc,myfile) )
   logger.info( green( ' --> gen my.cnf success') )
   logger.info( green( ' --> copy my.cnf to dist host ') )
   put( local_path="tmp/%s"%myfile, remote_path="/data/mysql%s/my.cnf"%(port) )
except Exception, e:
   logger.warning(yellow( ' --> genmycnf() exception : %s '%traceback.format_exc()) )
   raise e</pre><p>script 2 whole task :</p><pre class="brush:bash;toolbar:false">import inst_utils
from inst_utils import *
def install_mysql(port):
logger = initLoggerWithRotate()
thost = env.host_string
try:
   logger.info(green( 'stp 1 get the host %s '%thost ))
   #check host reachable
   rs1 = checkhost(logger )
   if int(rs1)== 0 :
         logger.info(red( 'stp 2 check the host is reachable failed ' ))
   logger.info(green( 'stp 2 check the host is reachable OK ' ))
   plat_type = run(''' uname -o ''')
   if plat_type !='GNU/Linux' :
         logger.warning(yellow('stp 3 target platform is not GNU/Linux & exit() '))
         sys.exit()
   logger.info(green('stp 3 target platform is GNU/Linux'))
   #check target host exsist mysql instance
   logger.info(green( 'stp 4 checkmysqlinstance' ))
   checkmysqlinstance(logger)
   #create MySQL user
   logger.info( green( 'stp 5 createUser ' ))
   createUser(logger)
   put(local_path="configs/bash_profile", remote_path="/home/mysql/.bash_profile")
   #checking dir
   logger.info( green( 'stp 6 initdir ' ))
   initdir(logger,port)
   #copy file
   logger.info( green( 'stp 7 copymysql ' ))
   copymysql(logger)
   logger.info( green( 'stp 8genmycnf') )
   genmycnf(logger,port,'h')
except Exception, e:
   print'main : exception : ' ,e</pre><p></p>

頁: [1]

琼殿技术论坛's Archiver

自动化运维管理fabric