Send Alerts to Slack with Bash Shell Script

Slack has quickly become one of the standard messaging apps in the Enterprise. Many a companies use slack for communication amongst team members and IT departments like having slack channels for projects or critical system groups.

slack

Since most of the people who are supposed to take action on a event are in a slack channel it is sometimes better to have notifications send directly to slack for server events.

One of the requirement I had recently was to create server alerts for checking API gateways and if the webservice was found to be down, to send alert to a slack channel.

To create such event based notification to slack, first you need to create an app in slack and then a webhook to expose it to the internet. Once that is done the remaining magic is done in plain old shell scripting

1. Create an APP in Slack

a) Goto https://api.slack.com/slack-apps

b) Create an App

c) Sign-in with your workspace

2. Create the Slack App and a webhook

a) Go to incoming webhooks and Activate incoming webhooks

You can see the sample of your webhook cURLthen

b) Create  a channel in your Slack eg : testalerts and assign to step c below

c) Add new Webhook to workspacee Post To : #testalerts

d) Webhook URLS for Your Workspace

curl -X POST -H ‘Content-type: application/json’ –data ‘{“text”:”Hello, World!”}’ https://hooks.slack.com/services/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

3. Test by POST’ing data to your webhook and it should now come in your slack channel

$ curl -X POST -H ‘Content-type: application/json’ –data ‘{“text”:”Hello, World!”}’ https://hooks.slack.com/services/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

4. Write Shell Script to check a Webservice URL and check for HTTP 200 OK in the header

Shell Script

#!/bin/bash
#################################################################
# SHELL SCRIPT TO MONITOR PUBLIC URL AND REST API GATEWAY
# Created By : Shadab Mohammad
# Company : Whitehat Agency, Sydney
# Created Date : 25/10/2018
# Modified Date : 29/10/2018
#
#################################################################
## Check ASIC WebService ##
#################################################################
if curl -s –head –request GET https://abr.business.gov.au | grep “200 OK” > /dev/null; then
echo “https://abr.business.gov.au is UP”
else
echo “https://abr.business.gov.au is DOWN”
curl -X POST -H ‘Content-type: application/json’ –data ‘{“text”:”https://abr.business.gov.au is DOWN”}’ https://hooks.slack.com/services/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
fi

The above shell script checks the header for 200 OK message , if found it sends alert to the channel via the slack webhook. It can be added to crontab to check the status of the webservice every 5 minutes. If the Webservice is down it will send notification to the Slack channel.

 

A Very Cool Oracle Bash Profile


# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

# Oracle Settings
export TMP=/tmp

export ORACLE_HOSTNAME=easyoradba.com
export ORACLE_UNQNAME=ora12c
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/12.2.0/dbhome_1
export ORACLE_SID=ora12c
export CATALINA_HOME=/home/oracle/apache-tomcat

PATH=/usr/sbin:$PATH:$ORACLE_HOME/bin:$CATALINA_HOME/bin

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib;
export CLASSPATH=$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;

alias ob='cd $ORACLE_BASE'
alias oh='cd $ORACLE_HOME'
alias tns='cd $ORACLE_HOME/network/admin'
alias ch='cd $CATALINA_HOME'
alias envo='env | grep ORACLE'
alias sqld='rlwrap sqlplus "/as sysdba"'

umask 022

envo

Shell Script to Ping Hostnames and Send SMS if Host is not Available

We use the below procedure to ping hosts and based on availability send SMS to our team. The SMS is send by calling a web service from the BASH Shell script using curl 

The hosts are defined in a file called hosts.txt

— Shell Script —

#!/bin/bash
current_time=$(date “+%Y%m%d%H%M%S%3N”)
#### Above Date Format is in MilliSeconds Accuracy ###
cd /home/online
rm -rf /home/online/SMSHOST.TXT
date
#### Below Code Will Loop the Hosts.TXT file for the Ping Probes ####
cat /home/online/hosts.txt | while read output
do
ping -w 30 -c 1 $output > /dev/null
if [ $? -ne 0 ]
then
SMSTEXT=’The+Host+’$output’+is+not+reachable.+Please+Contact+the+Database+Team.’
echo “Host $output is Down”
curl “http://192.168.10.100:8080/SMSConnect/SendServlet?application=http_gw2&password=http_gw2&content=$SMSTEXT&destination=9998887190&source=97249&mask=COMPANY”
else
echo “Host $output is Up”
fi
done

 

— hosts.txt file containing list of hostnames —

proddb01

proddb02

proddb03

drdb01

drdb02

drdb03

RMAN Script for Tape Backup : Tivoli Storage Manager

[sourcecode language=”sql”]

run
{
allocate channel sbt1 type sbt parms ‘ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin64/tdpo.opt)’;
allocate channel sbt2 type sbt parms ‘ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin64/tdpo.opt)’;
CONFIGURE DEFAULT DEVICE TYPE TO ‘SBT_TAPE’;
BACKUP DATABASE FORMAT ‘MW_FULL_%s_%D_%M_%Y’ tag ‘MW_DAILY_FULL_DB_BKP’;
BACKUP archivelog all not backed up 1 times format ‘ARCH_%d_ALL_%s_%t’ tag ‘MW_ARCH_LOG_BKP’;
release channel sbt1;
release channel sbt2;
}

[/sourcecode]

Linux Matrix Effects Shell Script

Sometimes to have a funky time on the BASH shell you can run the below shell script

[code language=”bash”]
#!/bin/bash
clear
virtcent=$((`tput lines`/2))
horcent=$((`tput cols`/2-10))
tput setf 6
tput cup $virtcent $horcent && echo "entering the matrix"
sleep 1
tput cup $virtcent $horcent && echo " "
sleep 1
tput cup $virtcent $horcent && echo "entering the matrix"
sleep 1
tput cup $virtcent $horcent && echo " "
sleep 1
tput cup $virtcent $horcent && echo "connecting. "
sleep .1
tput cup $virtcent $horcent && echo "connecting.. "
sleep .1
tput cup $virtcent $horcent && echo "connecting… "
sleep .1
tput cup $virtcent $horcent && echo "connecting…. "
sleep .1
tput cup $virtcent $horcent && echo "connecting….. "
sleep .1
tput cup $virtcent $horcent && echo "connecting…… "
sleep .1
tput cup $virtcent $horcent && echo "connecting……. "
sleep .1
tput cup $virtcent $horcent && echo "connecting……. "
sleep .1
tput cup $virtcent $horcent && echo "connecting……. "
sleep .1
tput cup $virtcent $horcent && echo "connecting…….. "
sleep .1
tput cup $virtcent $horcent && echo "connecting………"
sleep 1
tput cup $virtcent $horcent && echo " CONNECTED "
sleep 1
clear

echo -e "\033[2J\033[?25l"; R=`tput lines` C=`tput cols`;: $[R–] ; while true
do ( e=echo\ -e s=sleep j=$[RANDOM%C] d=$[RANDOM%R];for i in `eval $e {1..$R}`;
do c=`printf ‘\\\\0%o’ $[RANDOM%57+33]` ### http://easyoradba.com ###
$e "\033[$[i-1];${j}H\033[32m$c\033[$i;${j}H\033[37m"$c; $s 0.1;if [ $i -ge $d ]
then $e "\033[$[i-d];${j}H ";fi;done;for i in `eval $e {$[i-d]..$R}`; #[mat!rix]
do echo -e "\033[$i;${j}f ";$s 0.1;done)& sleep 0.05;done #(c) 2016 — [ ShadabMohammad ]
[/code]

Just chmod 750 on the script and execute

$ vi matrix.sh
$ chmod 750 matrix.sh
$ matrix.sh

This is a re-worked version of the shell script available below.

http://bruxy.regnet.cz/web/linux/EN/matrix-sh/

Shell Script to Monitor AIX Filesystem and Send Email

The Below Shell Script checks the Filesystem mount points and using AWK outputs all filesystem exceeding 90% space to a file called diskspacepoll. Once that is done the sed command removes any special character like ‘%’ from the output file and cleans it to a file called output.log

The next important logic is in the AWK block. Here a variable called pattern is defined using the threshold of 90%. Another variable called var is defined. This is your baseline metric. So it value of pattern exceeds var then the mail is dispatched else the script does nothing. You can put this in crontab as a every 5 minute job to continuously poll the filesystems and incase the threshold is exceed it will dispatch an email immediately to the admin

[code language=”bash”]

#!/bin/ksh
df -P | grep -v Capacity | awk ‘{if ($5 >= 90) {print $5;}}’ > /home/root/diskspacepoll
sed ‘s/[!@#\$%^&*()]//g’ /home/root/diskspacepoll > /home/root/output.log
####### AWK LOGICAL BLOCK #########
pattern=$(awk ‘$1 > 90 {print $1}’ /home/root/output.log)
var=90
if [[ $pattern > $var ]]
then
echo "Please Check with System Administrator" | mailx -s "90% Threshold of DiskSpace exceeded on Server 1 (ESB1)" sysadmin@company.com
fi

[/code]

Running SQL Script for Multiple Databases SQL*PLUS | Mutiple Servers SQL*PLUS in Shell Script | BASH Script Looping SQL*PLUS commands

For running a single SQL Command via sqlplus on multiple servers, we have to first do a few pre-requisites

1. Oracle client should be installed on the Unix/Linux Server

2. Create Local TNS entry in the tnsnames.ora file for the databases where you will run the commands

3. All databases should have one common user with a common password (this method is not the safest method
since the password will be in plain text in your Shell script)
Now let us proceed to first create a text file called dbnodes.txt which will have the TNSNAMES for the
database we will connect to..

boston
chicago
newyork

Above is the entries in the dbnodes.txt file, these are the databases where we will loop the sql commands

Create a sql script with the commands you have to run on all the databases, lets call the file script.sql

set echo on
set linesize 200
set pages 0
select sysdate from dual;
select user from dual;
select instance_name,host_name from gv$instance;
alter system set cpu_count=8 scope=both sid='*';
show parameter cpu_count;
alter session set nls_date_format='DD/MM/YYYY';
exit;

I do this to dynamically to cap the cpu_count on some of our databases to contain any resouce hogging.

Now the final shell script which will call the dbnodes.txt and script.sql to loop the above sql commands
through all databases. The shell script is called dbloop.sh

#!/bin/bash
cat dbnodes.txt | while read line
do
sqlplus -s user/user123@$line @/u03/scripts/script.sql
done

Copy all 3 files dbnodes.txt, script.sql and dbloop.sh to one directiry and run it like below; to collect the log of the SQL commands.

$--> dbloop.sh > dbrun.log

This will spool the output to a logfile.

So there you see it, one of the easiest ,method to run a set ofcommon sql commands on hundreds of server. I use this script to do basics like checking a certain parameters on all our production databases, to check dataguard status for multiple production DB’s etc. The use cases of this script is unlimited. Hope you enjoyed it 🙂 Keep it Easy and Keep It Oracle !