Simple Shell Script to Check Diskspace and Send Email on AIX

The below simple shell script will check all filesystems and create output file for only filesystems which are more than 80% full and send the log by email.

The below has been checked on AIX7.1 , not too Sure if it works on Solaris. I suppose the -P flag does not work on Solaris.

df -P | grep -v Capacity | awk '{if ($5 >= 80) {print $5 $6;}}' > /home/root/diskspacelog
mailx -s "DiskSpace for Server Esb1" </home/root/diskspacelog

Shell Script to Monitor AIX Filesystem and Send Email

The Below Shell Script checks the Filesystem mount points and using AWK outputs all filesystem exceeding 90% space to a file called diskspacepoll. Once that is done the sed command removes any special character like ‘%’ from the output file and cleans it to a file called output.log

The next important logic is in the AWK block. Here a variable called pattern is defined using the threshold of 90%. Another variable called var is defined. This is your baseline metric. So it value of pattern exceeds var then the mail is dispatched else the script does nothing. You can put this in crontab as a every 5 minute job to continuously poll the filesystems and incase the threshold is exceed it will dispatch an email immediately to the admin

df -P | grep -v Capacity | awk '{if ($5 >= 90) {print $5;}}' > /home/root/diskspacepoll
sed 's/[!@#\$%^&*()]//g' /home/root/diskspacepoll > /home/root/output.log
####### AWK LOGICAL BLOCK #########
pattern=$(awk '$1 > 90 {print $1}' /home/root/output.log)
if [[ $pattern > $var ]]
echo "Please Check with System Administrator" | mailx -s "90% Threshold of DiskSpace exceeded on Server 1 (ESB1)"

Top command AIX

AIX doesn’t have a top command what it has a better tool called topas. Just write topas from a shell prompt and it opens a beautiful colored stats window where you can get the top consuming sessions as well as few other useful information.

# topas

Join AIX to DNS

# vi /etc/resolv.conf

— If file doesn’t exist, create it

Add the below entries into resolv.conf file of your DNS server and domain name


# nslookup nameofyourserver

Create Logical Volume Group and JFS2 filesystem AIX

We had a new Oracle installation on an AIX 6.1 Power7 server. Our organization doesn’t have an AIX admin. So to faciliate the lack of resources, I decided to be the makeshift Aix admin and create the mountpoints for the Oracle installation. It was easier than I thought. The Smitty tool is very easy and powerful in doing day-to-day tasks on Aix.

The local hard disks can be found at /dev/hdisk* location. First we will create a Volume group.

1. Create Logical Volume Group
mkvg -y oradata hdisk2

2. Create Logical Volume
prmdb[/dev] # mklv -t jfs2log oradata 1

3. List the volume group (our new volume group is called oradata)
prmdb[/dev] # lsvg

4. Check the status of the volume group
prmdb[/dev] # lsvg -l oradata
loglv01 jfs2log 1 1 1 closed/syncd N/A

5. Goto smitty and create the journal filesystem for the logical volume we just created.

prmdb[/dev] # smitty jfs2

– Add an Enhanced Journaled File System
– Select the Volume Group Name “oradata” we previously created
– Put the unit size in your choice of megabytes or gigabytes(to find out size command is “lsvg oradata”
– Select mount at restart as yes
– Save and Exit

prmdb[/dev] # lsvg

prmdb[/dev] # lsvg oradata
VOLUME GROUP: oradata VG IDENTIFIER: 00f7abf000004c000000013b046eb8c2
VG STATE: active PP SIZE: 128 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 799 (102272 megabytes)
MAX LVs: 256 FREE PPs: 798 (102144 megabytes)
LVs: 1 USED PPs: 1 (128 megabytes)
OPEN LVs: 0 QUORUM: 2 (Enabled)
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 256 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable

prmdb[/dev] # smitty jfs2

Create the file system as “Add a new enchanced journal filesystem”
New File System size is 207618048

Command: OK stdout: yes stderr: no

Before command completion, additional instructions may appear below.

File system created successfully.
103805652 kilobytes total disk space.
New File System size is 207618048

List all mounted file systems from smitty

Name Nodename Mount Pt VFS Size Options Auto Accounting
/dev/hd4 — / jfs2 4194304 rw yes no
/dev/hd1 — /home jfs2 1048576 rw yes no
/dev/hd2 — /usr jfs2 6291456 rw yes no
/dev/hd9var — /var jfs2 6291456 rw yes no
/dev/hd3 — /tmp jfs2 6291456 rw yes no
/dev/hd11admin — /admin jfs2 262144 rw yes no
/proc — /proc procfs — rw yes no
/dev/hd10opt — /opt jfs2 2097152 rw yes no
/dev/livedump — /var/adm/ras/livedump jfs2 524288 rw yes no
/dev/cd0 — /cdrom cdrfs — ro no no
/dev/locallv — /usr/local jfs2 1048576 rw yes no
/dev/fslv00 — /oracle jfs2 102760448 rw yes no
/dev/fslv01 — /oradata jfs2 207618048 rw yes no

6. Create a mount point and mount the filesystem on it (Check the device name from the above entry in smitty)

prmdb[/dev] # mkdir /oradata
prmdb[/dev] # mount /dev/fslv01 /oradata
prmdb[/dev] # df -g
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/hd4 2.00 1.80 11% 10247 3% /
/dev/hd2 3.00 0.43 86% 56419 36% /usr
/dev/hd9var 3.00 2.63 13% 8462 2% /var
/dev/hd3 3.00 0.74 76% 6017 4% /tmp
/dev/hd1 0.50 0.39 23% 60 1% /home
/dev/hd11admin 0.12 0.12 1% 5 1% /admin
/proc – – – – – /proc
/dev/hd10opt 1.00 0.69 32% 9968 6% /opt
/dev/livedump 0.25 0.25 1% 4 1% /var/adm/ras/livedump
/dev/locallv 0.50 0.50 1% 128 1% /usr/local
/dev/fslv00 49.00 38.92 21% 5661 1% /oracle
/dev/fslv01 99.00 98.98 1% 4 1% /oradata

7. Test the mountpoint by creating a text file

prmdb[/dev] # cd /oradata
prmdb[/oradata] # touch abc.txt
prmdb[/oradata] # rm -rf abc.txt

8. Check the entries in the filesystems file. if it doesnt exists then create it

prmdb[/oradata] # vi /etc/filesystems
prmdb[/oradata] #

dev = /dev/fslv01
vfs = jfs2
log = /dev/loglv02
mount = true
options = rw
account = false

And Voila your done configuring a persistent file system using the JFS2 filesystem on Aix for your oracle installation. From their on it is a standard Oracle installation with creating users, groups etc.


Prepare Raw Devices for ASM on AIX 6.1 (RAC 11gR2)

To Configure RAC 11gR2 on AIX 6.1 using ASM. You need to prepare the raw devices like below. The general location for raw devices on AIX is /dev. The disk names are like /dev/rhdisk*

# lsattr -E -l hdisk3 | grep reserve_
reserve_policy no_reserve
#chdev -l hdisk3 -a [ reserve_lock=no | reserve_policy=no_reserve ]
reserve_policy is for AIX storage, rreserve_lock is for EMC and other storage. You need to change the
reserve option on every storage device you will be using in ASM

prmdb[/dev] # /usr/sbin/chdev -l hdisk3 -a pv=clear
hdisk3 changed
prmdb[/dev] # /usr/sbin/chdev -l hdisk2 -a pv=clear
hdisk2 changed
prmdb[/dev] # /usr/sbin/chdev -l hdisk4 -a pv=clear
hdisk4 changed

prmdb[/dev] # chmod 660 rhdisk4
prmdb[/dev] # chmod 660 rhdisk3
prmdb[/dev] # chmod 660 rhdisk2

prmdb[/dev] # chown grid:asmadmin rhdisk2
prmdb[/dev] # chown grid:asmadmin rhdisk4
prmdb[/dev] # chown grid:asmadmin rhdisk3

If you get error that you cannnot create the disk group then clear the raw devices using below command( this generally happens when you had previously configured an ASM instance to use these raw devices and it was a failed installation and you are installing again)

chdev -l hdisk4 -a pv=yes
chdev -l hdisk3 -a pv=yes
chdev -l hdisk2 -a pv=yes

ORA-27300 ORA-27301 ORA-27302

ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sun Mar 27 08:45:52 2011

This problem is most likely due to swap space being less. Ask your system administrator to increase the swap space. The rule of thumb is swap space should be twice that of physical memory. We faced this issue on AIX 5L box running Oracle Database 10gR2

Alternately you can run the HCVE script which is now part of RDA to do health check of database and confirm this.

1. Download RDA package for your OS from metalink.

2. Unzip the RDA package.

3. Go to the RDA directory and type below command to run. For an AIX system you will see options. Select your database version.

$ ./ -T hcve

Processing HCVE tests …
Available Pre-Installation Rule Sets:
1. Oracle Database 10g R1 (10.1.0) Preinstall (AIX)
2. Oracle Database 10g R2 (10.2.0) Preinstall (AIX)
3. Oracle Database 11g R1 (11.1.0) Preinstall (AIX)
4. Oracle Database 11g R2 (11.2.0) Preinstall (AIX)
5. Oracle Application Server 10g (9.0.4) Preinstall (AIX)
6. Oracle Fusion Middleware 11g R1 (11.1.1) Preinstall (AIX)
7. Oracle Portal Preinstall (Generic)
8. Oracle Identity Management 10g (10.1.4) Preinstall (AIX)
9. Oracle E-Business Suite Release 11i (11.5.10) Preinstall (AIX)
10. Oracle E-Business Suite Release 12 (12.1.1) Preinstall (AIX)
Available Post-Installation Rule Sets:
11. Oracle Portal Postinstall (generic)
12. Data Guard Postinstall (Generic)
Enter the HCVE rule set number
Hit ‘Return’ to accept the default (1)
> 2

Enter value for < Planned ORACLE_HOME location >
Hit ‘Return’ to accept the default

4. Once you select a test the step above you will see the test results on the screen. The test results are also written to a HTML file that is located in the RDA output directory. You can review the test results by using a Web Browser to open the following file located in:

Test “Oracle Database 10g R1 (10.1.0)  Preinstall (AIX)” executed at 28-Mar-2011 14:33:03

Test Results

ID     NAME                 RESULT  VALUE
====== ==================== ======= ==========================================
A00010 OS Certified?        PASSED  Certified with Oracle Database 10g
A00020 User in “/etc/passw PASSED  userOK
A00030 Group in “/etc/grou PASSED  GroupOK
A00040 Enter ORACLE_HOME    RECORD  /oradb/u01/app/oracle/oracle/product/..>
A00050 ORACLE_HOME Valid?   PASSED  OHexists
A00060 O_H Permissions OK?  PASSED  CorrectPerms
A00070 Umask Set to 022?    PASSED  UmaskOK
A00090 LIBPATH Unset?       PASSED  UnSet
A00100 Other O_Hs in PATH?  FAILED  OratabEntryInPath
A00110 oraInventory Permiss PASSED  oraInventoryOK
A00120 /tmp Adequate?       PASSED  TempSpaceOK
A00130 Swap (in MB)         RECORD  5632
A00140 RAM (in MB)          PASSED  7712
A00150 SwapToRAM OK?        FAILED  SwapLessThanRAM
A00160 Disk Space OK?       PASSED  DiskSpaceOK
A00180 Got Software Tools?  PASSED  ld_nm_ar_make_found
A00190 ulimits OK?          PASSED  ulimitOK
A00200 Got OS Packages?     PASSED  All required OS packages are installed
A00210 Got OS Patches?      PASSED  PatchesFound
A00220 Other OUI Up?        PASSED  NoOtherOUI
Result file: /oradb/u01/app/oracle/rda/output/RDA_HCVE_A200DB10R1_aix_res.htm

Note: There are NO tests currently available for the Windows Platform.