Applying Yum Package Updates to Multiple Servers Using Fabric
Administrators of multiple servers know that updating software packages can be a tedious task. It requires a log in, followed by the execution of a "yum update" or something similar for each server. Although several tools exist for enabling administrators to update packages automatically, including yum-updatesd and yum-cron, updating the packages without confirmation possibly leads to further issues.
For instance, Yum completely ignores the dependencies of software compiled from the source. If the compiled software has strict version requirements against the update of packages, it may not function properly after the update. Furthermore, even if the package is installed using Source RPM with custom compile options, it is overwritten by a new package and the features specified in the compile options are disabled.
In this article, we introduce a Python-based short script that enables administrators to easily apply command-line package updates to multiple servers. Using this script, the administrator can manually confirm which packages are to be updated. Internally, this script executes a "yum check-update" in sequence in order to verify the presence of package updates and displays a prompt that allows the administrator to select whether the package updates are to be performed.
The proposed script uses a relatively new Python library called Fabric, which allows users to execute commands at multiple remote servers via SSH. Therefore, the script needs to connect to the user's remote servers via SSH.
Although we tested the script on CentOS 5.5 Linux, administrators using other RedHat-based distributions can also use this script.
Example of Error Caused by Yum
In this section, we will provide an actual example for demonstrating the issue that arises in the Yum package update. Postfix, which is a common mail server, implements MySQL support from recent versions. However, the Postfix package provided by the default Yum repository does not include this feature, and hence, many users install Postfix from the Source RPM.
Let us assume that the server has a previous version of Postfix installed from the Source RPM and uses the configuration given in Postfix MySQL Howto .
# sudo rpm -ivh PREVIOUS_POSTFIX_SOURCE_RPM_URL # cd /usr/src/redhat/ # sudo vim SPEC/postfix.spec < %define MYSQL 0 > %define MYSQL 1 # rpmbuild -ba SPEC/postfix.spec # rpm -ivh RPMS/x86_64/postfix-VERSION_NUM.x86_64.rpm
After executing the "yum -y update", Yum detects and installs the newer version of Postfix.
Loaded plugins: downloadonly, fastestmirror Loading mirror speeds from cached hostfile (snip) ======================================================================================================= Package Arch Version Repository Size ======================================================================================================= Updating: postfix x86_64 2:2.3.3-2.1.el5_2 base 3.7 M Transaction Summary ======================================================================================================= Install 0 Package(s) Upgrade 1 Package(s) (snip) Running Transaction Updating : postfix 1/2 warning: /etc/postfix/main.cf created as /etc/postfix/main.cf.rpmnew Cleanup : postfix 2/2 Updated: postfix.x86_64 2:2.3.3-2.1.el5_2 Complete!
Because the installed package does not contain the MySQL support, Postfix starts to output the following errors in /var/log/maillog after it is restarted.
Feb 18 00:31:29 jango postfix/master: warning: process /usr/libexec/postfix/local pid 6189 exit status 1 Feb 18 00:31:29 jango postfix/master: warning: /usr/libexec/postfix/local: bad command startup -- throttling Feb 18 00:32:29 jango postfix/local: fatal: unsupported dictionary type: mysql
If it is not confirmed whether the updated package works properly, the Yum package update may occasionally result in server failure. Therefore, we recommend the use of our proposed script instead of the automatic package update provided by yum-updatesd and yum-cron .
The proposed script requires several software packages. First, the EPEL software repository needs to be installed. Administrators using a Linux distribution based on RedHat Enterprise Linux 5 can install the repository using the following command. Installation instructions can also be found here.
# sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
Next, several software packages need to be installed.
# sudo yum -y install gcc python26 python26-devel python26-distribute
Now, Fabric can be installed using easy_install , which also installs the required Python libraries using the following command.
# sudo easy_install-2.6 fabric
The proposed script is given below. As Fabric uses fabfile.py as its default file name, the script should be entered into fabfile.py.
# cd INSTALL_DIR # vim fabfile.py #!/usr/bin/env python2.6 from fabric.api import env, run, sudo from fabric.contrib.console import confirm env.warn_only = True def update(): if run("yum check-update").return_code != 0: if confirm("execute yum update?", default=False) == True: sudo("yum -y update", pty=True)
As mentioned above, the script executes a yum check-update in each remote server and displays a prompt that enables the user to select whether to execute the yum update. Note that env.warn_only is a necessary component; this is because yum returns a non-zero status code if it finds any package updates, which Fabric incorrectly recognizes as an error and responds by shutting the execution down.
Now, the script can be executed. fab is the command that receives the target Python function name as an argument and runs the function. In this case, the function name is update , and thus, the command is as follows.
# fab -H HOST_NAME1,HOST_NAME2 update
After executing the above command, the script checks for the presence of package updates and outputs the following. As can be seen, if package updates exist, the user can choose whether to perform the updates after confirming the packages that are to be updated.
[qui-gon.ikuya.net] Executing task 'update' [qui-gon.ikuya.net] run: yum check-update [qui-gon.ikuya.net] out: Loaded plugins: fastestmirror [qui-gon.ikuya.net] out: [qui-gon.ikuya.net] out: bind-libs.x86_64 30:9.3.6-4.P1.el5_5.3 updates (snip) [qui-gon.ikuya.net] out: udev.x86_64 095-14.21.el5_5.1 updates Warning: run() encountered an error (return code 100) while executing 'yum check-update' execute yum update? [y/N]
The user name and password can be optionally specified using -u and -p options, respectively.
# fab -u USER_NAME -p PASSWORD -H HOST_NAME1,HOST_NAME2 update
The target remote server addresses can also be specified using the env.hosts variable in fabfile.py.
#!/usr/bin/env python2.6 from fabric.api import env, run, sudo from fabric.contrib.console import confirm env.warn_only = True env.hosts = [ "HOST_NAME1", "HOST_NAME2" ] def update(): if run("yum check-update").return_code != 0: if confirm("execute yum update?", default=False) == True: sudo("yum -y update", pty=True)
Using the above script, the user can run the update without specifying the remote server addresses.
# fab update
This article proposed a short script that enables server administrators to easily apply package updates to remote servers. The script uses a Python library called Fabric , and executes package updates in sequence.
Note that Fabric is a generic Python library and can be used to automate numerous other server administration tasks. If you want to learn more about Fabric, please refer to the official documentation .
Ikuya Yamada is an entrepreneur and an experienced software engineer. Currently, he is the founder and the CTO of Studio Ousia Inc., a software R&D company founded in 2007 in Tokyo. He is also a senior visiting researcher at the Keio Research Institute at SFC from 2010. Prior to Studio Ousia, he was the CTO of a listed Japanese software company named Fractalist Inc. and previously the founder and the CEO of a software R&D company called Newrong Inc., which was acquired by Fractalist Inc. in 2005. He obtained his B.S. (2006) and M.S. (2010) from Keio University.
Yoshiyasu Takefuji was heavily involved in developing a unix based color workstation in 1983 at University of South Florida. Recently he has been monitoring three Linux servers to see the behavior of DOS attacks. He is a chair of SecurityExpo in Japan since 2004 and also a chair of OECD TrustE security product evaluation committee chair in Japan, and advisor of Japan Network Security Association and CMU in Japan.