Skip to main content

Quick and simple parallel ssh command shell script

Why

Sometimes running commands across a range of servers is necessary. Yes, there is Ansible, Puppet and SaltStack etc., however in some cases this is overkill (and these usually require python, ruby or other languages installed.

The following shell script runs commands (via sudo if required) on a group of hosts in parallel. It is quite old and not very elegant but does the trick and hopefully helps somebody in the future. Please don't comment on syntax and bugs.

Configuration

The .run file in your home directory contains one line for user: and one line for pass:. Should this file not exist, the script will ask for the user and password interactively.

The servers file includes one line per host (or includes for additional host files) - or if the file does not exist it will be treated as single host name.

Usage

./run.sh server "uptime"

Script

#!/bin/bash
#
# About: Script to concurrently run commands on a list of servers (with or without sudo)
# Author: Lonestarr
# Last edit: 2012 05 24
# Version: 0.5
#
#####

# defaults
SLEEP=5
CONCURRENCY=0
RECURSIONS=6

# do not change below
user=`/usr/bin/id -un`
control="$HOME/.run"
file=""
command=""
MPIDS=""
PROCESSES=0
SERVERS=""
NUMSERVERS=0
RCOUNT=0
USESUDO=0
DEBUG=0
USECONFIRM=1

##############################################################################################
# FUNCTIONS
##############################################################################################

# running the actual command - via embedded expect script
run_on_server(){
  SHORT=`echo $host | cut -d. -f1`
  echo "$SHORT: running $command"
 
  # /usr/bin/expect -d -c " # debug mode
  /usr/bin/expect -c "
set timeout 10

# LOGIN to server
spawn ssh -q $user@$host

expect {
 # PASSWORD LOGIN - send password
 -re \"assword:\" {
    send \"$pass\r\n\"
   }
 # save new ssh key - accept - send yes
 -re \".*\(yes.*no\)\" {
    send \"yes\r\n\"
    exp_continue
  }
}

        sleep 2

# wait for prompt
expect \"*$user@$SHORT\"

# run commands requested
set timeout 200
send \"sudo $command\r\n\"

        sleep 2

# sudo password - if necessary - otherwise exit ssh and close
expect {
  -re \"assword for $user:\" {
            send \"$pass\r\"
    exp_continue
  }
  -re \"Password:|password for\" {
    send \"$pass\r\"
    exp_continue
  }
          -re \"$user@$SHORT\" {
    send \"exit\r\"
  }
}
  " | egrep -v "^spawn ssh| sudo |RHN Satellite kickstart| password for |^Password:|^Last login:|^[[:space:]]*$" | sed -e "s/^/$SHORT: /g"

 echo "$SHORT: completed."
}

# print usage and exit
print_usage(){
  echo >&2 ""
  echo >&2 "usage: $0 [-s delay] [-y] [-d] [-S] [-c concurrency] [-u user] <server name or server list file> <command>"
  echo >&2 " -s 5 : sleep between running the commands on each server in seconds, the default is 5"
  echo >&2 " -y   : usually you will ask for confirmation prior to executing the commands - use this option to disable confirmation"
  echo >&2 " -d   : enable debug mode"
  echo >&2 " -S   : run command using sudo - alternatively can also be supplied as part of the command"
  echo >&2 " -c 5 : use concurrency of 5, run a maximum of 5 servers in parallel. Use 0 to run on all servers in parallel"
  echo >&2 " -u x : use a different user name than the script is running under, can also be supplied via .run config file"
  echo >&2 ""
  echo >&2 "host / server file: either a single server name or the name of a file containing host names or include <file> statements."
  echo >&2 "command: command to run on the remote host. Needs to be quoted if it contains spaces or semicolons - also if multiple sudo"
  echo >&2 "commands need to be executed, add sudo before each additional command"
  echo >&2 ""
  echo >&2 "Example: $0 -S defiant.myhome.com \"uname -a\""
  echo >&2 ""
  exit 3
}

# recurse host files
include_server_file()
{
  # if not file, treat as server and add to list
  if [ ! -e "$1" ]; then
    SERVERS="$SERVERS
$1"
    NUMSERVERS=$[$NUMSERVERS+1]
    return
  fi

  # if it's a file, read hosts
  while read host
  do
    if [[ $host == *include* ]]; then
      FILE=`echo $host | cut -d" " -f2-`
      include_server_file "`dirname $1`/$FILE"
    else
      SERVERS="$SERVERS
$host"
      NUMSERVERS=$[$NUMSERVERS+1]
    fi
  done < "$1"
}

# get a list of running processes
getRunningJobs()
{
  # get a list of all running processes
  PROCS=`ps -e | awk '{ print $1 }' | grep -v PID`
  RUNNING=0

  # go through list of children and see if they are still running
  for MPID in $MPIDS; do
    IN=`echo $PROCS | grep -c " $MPID "`
    if [ $IN == "1" ]; then
      RUNNING=`expr $RUNNING + 1`
    fi
  done
  return $RUNNING
}

##############################################################################################
# MAIN
##############################################################################################

# make sure we have at least 2 parameters
if [ $# -lt 2 ]; then
  print_usage
fi

# read command line options
while [ $# -gt 0 ]
do
  # only two parameters left, use as file and command
  if [ $# -eq 2 ]; then
    serverfile="$1"
    command="$2"
    shift; shift
  else
    # more options to read, overwrite defaults where necessary
    case "$1" in
      -s) SLEEP="$2"; shift;;
      -u) user="$2"; shift;;
      -c) CONCURRENCY="$2"; shift;;
      -S) USESUDO=1;;
      -d) DEBUG=1;;
      -y) USECONFIRM=0;;
      -h) print_usage;;
      -*) print_usage;;
    esac
    shift
  fi
done

# building server list / handle includes
include_server_file "$serverfile"

# ask for confirmation
if [ "x$USECONFIRM" == "x1" ]; then
  if [ "x$USESUDO" == "x1" ]; then
    echo "Are you sure you want to run: \"$command\" on these $NUMSERVERS servers (as root)?"
  else
    echo "Are you sure you want to run: \"$command\" on these $NUMSERVERS servers?"
  fi
  echo "$SERVERS"
  echo
  read -p "Please confirm (y/N): " confirmation
  if [ "x$confirmation" != "xy" ]; then
    echo "aborted."
    exit 3
  fi
fi

# if user / pass config file exists and has user / pass, get it from there
if [ -r "$control" ]; then
  user=`grep user: $control | cut -d: -f2-`
  pass=`grep pass: $control | cut -d: -f2-`
else
  # ask for password
  read -s -p "Please enter $user's password: " pass
  echo
fi

# go through all servers in list
for host in $SERVERS; do
  # run command in background
  run_on_server &
  MPIDS="$MPIDS $! "
  PROCESSES=`expr $PROCESSES + 1`
  sleep $SLEEP

  # handle concurrency - and wait until some jobs have completed
  if [ "x$CONCURRENCY" != "x0" ]; then
    # wait until we can start more jobs again
    while true; do
      # get current number of running jobs
      getRunningJobs
      if [ "$?" -ge "$CONCURRENCY" ]; then
        echo "Currency of $CONCURRENCY reached - must wait";
        sleep 5
      else
        break
      fi
    done
  fi
done

# waiting for all processes to finish
while : ; do
  sleep 5

  # check running processes - count number
  getRunningJobs
  RUNNING=$?

  # are we finished yet?
  if [ "x$RUNNING" == "x0" ]; then
    echo "All $PROCESSES tasks completed."
    exit
  fi

  # not yet, let the user know
  echo "-> waiting for $RUNNING / $PROCESSES tasks to finish ... please be patient"
done
Labels:

Comments

Post a Comment

Popular posts from this blog

Manual Kubernetes TLS certificate renewal procedure

Intro Kubernetes utilizes TLS certificates to secure different levels of internal and external cluster communication.  This includes internal services like the apiserver, kubelet, scheduler and controller-manager etc. These TLS certificates are created during the initial cluster installation and are usually valid for 12 months. The cluster internal certificate authority (CA) certificate is valid for ten years. There are options available to automate certificate renewals, but they are not always utilised and these certs can become out of date. Updating certain certificates may require restarts of K8s components, which may not be fully automated either. If any of these certificates is outdated or expired, it will stop parts or all of your cluster from functioning correctly. Obviously this scenario should be avoided - especially in production environments. This blog entry focuses on manual renewals / re-creation of Kubernetes certificates. For example, the api-server certificate below...
Post a Comment
Read more

Analysing and replaying MySQL database queries using tcpdump

Why There are situations where you want to quickly enable query logging on a MySQL Database or trouble shoot queries hitting the Database server in real-time. Yes, you can enable the DB query log and there are other options available, however the script below has helped me in many cases as it is non intrusive and does not require changing the DB server, state or configuration in any way. Limitations The following only works if the DB traffic is not encrypted (no SSL/TLS transport enabled). Also this needs to be run directly on the DB server host (as root / admin). Please also be aware that this should be done on servers and data you own only. Script This script has been amended to suit my individual requirements. #!/bin/sh tcpdump -i any -s 0 -l -w - dst port 3306 | strings | perl -e ' while(<>) { chomp; next if /^[^ ]+[ ]*$/;   if(/^(ALTER|COMMIT|CREATE|DELETE|DROP|INSERT|SELECT|SET|UPDATE|ROLLBACK)/i) {     if (defined $q) { print "$q\n"; }     $q=$_; ...
Post a Comment
Read more

Deprecating Networking Ingress API version in Kubernetes 1.22

  Intro Kubernetes deprecates API versions over time. Usually this affects alpha and beta versions and only requires changing the apiVersion: line in your resource file to make it work. However with this Ingress object version change, additional changes are necessary. Basics For this post I am quickly creating a new cluster via Kind (Kubernetes in Docker) . Once done, we can see which API versions are supported by this cluster (version v1.21.1). $ kubectl api-versions | grep networking networking.k8s.io/v1 networking.k8s.io/v1beta1 Kubernetes automatically converts existing resources internally into different supported API versions. So if we create a new Ingress object with version v1beta1 on a recent cluster version, you will receive a deprecation warning - and the same Ingress object will exist both in version v1beta1 and v1. Create $ cat ingress_beta.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:   name: clusterpirate-ingress spec:   rules:  ...
Post a Comment
Read more