[Ocfs2-tools-devel] [PATCH 1/1] Ocfs2-test: Add multiple_run.sh for ocfs2-test.
tristan.ye
tristan.ye at oracle.com
Wed Sep 24 20:06:56 PDT 2008
Marcos,
When you decide to test this in your cluster,please keep following
things in mind:)
* Please uninstall your lammpi first,then install the openmpi setup,it
can be done via ULN(up2date) or download the latest src then build by
yourself. never mix openmpi and lammpi together in one node.
*Should configure passwordless ssh/rsh connection for all nodes,this
may have been done by your former tests. it depends on your personal
favor to use ssh or rsh for remote execution. while ssh is recommended.
*Pull the latest src from ocfs2-test.git,and apply the following
patch.it should work.before your test,please umount your target volume
from all testing nodes.all what you will be asked for this test is a
umounted volume,node list and the mount point location.
*It may be common to hit errors in your first attempt,you can turn help
to the logfiles or the script itself for help.(it was written in a same
pattern as single_run.sh).
For the openmpi issues,there is a useful link here:
http://www.open-mpi.org/faq/
Almost all of my issues hit before get a fix there.
BTW,also i've attached the latest multiple_run.sh patch there:
http://oss.oracle.com/osswiki/OCFS2/XattrTest
Since i'll be on annual leave and holiday in coming 10 days, you can
leave the messages to me via this mail address:
tristanye.swjtu at yahoo.com.cn when you hit any problem.
Regards,
Tristan.
On Wed, 2008-09-24 at 13:17 -0700, Sunil Mushran wrote:
> Tristan, Excellent.
>
> Marcos, Please could you test this in your cluster.
>
> Tristan Ye wrote:
> > Like single_run.sh,we also need an automatic launcher for existing multi-nodes testcase
> > to organize and manipulate these cases,statistic their running result and logs in a graceful way.
> >
> > Currently,it includes following testcases,
> >
> > xattr_test
> > inline_test
> > write_append_truncate_test
> > multi_mmap_test
> > lvb_torture_test
> > create_racer_test
> > flock_tests
> > open_delete_test
> >
> > Need to run this wrapper script like this:
> >
> > multiple_run.sh -n node1.us.oracle.com,node2.us.oracle.com -d /dev/sdd1 /storage
> >
> > All its logs and behaviors kept a unification as single_run.sh did. More testcases were expected
> > to be integrated as we're going to develop more multi-nodes cases for ocfs2-test.
> >
> > Signed-off-by: Tristan Ye <tristan.ye at oracle.com>
> > ---
> > programs/python_common/multiple_run.sh | 403 ++++++++++++++++++++++++++++++++
> > 1 files changed, 403 insertions(+), 0 deletions(-)
> > create mode 100755 programs/python_common/multiple_run.sh
> >
> > diff --git a/programs/python_common/multiple_run.sh b/programs/python_common/multiple_run.sh
> > new file mode 100755
> > index 0000000..15e3998
> > --- /dev/null
> > +++ b/programs/python_common/multiple_run.sh
> > @@ -0,0 +1,403 @@
> > +#!/bin/bash
> > +# vi: set ts=8 sw=8 autoindent noexpandtab :
> > +################################################################################
> > +#
> > +# File : multiple_run.sh
> > +#
> > +# Description: The wrapper script help to organize all multi-nodes testcase
> > +# to perform a thorough test among multiple nodes.
> > +#
> > +#
> > +# Author: Tristan Ye, tristan.ye at oracle.com
> > +#
> > +# History: 22 Sep 2008
> > +#
> > +
> > +################################################################################
> > +# Global Variables
> > +################################################################################
> > +PATH=$PATH:/sbin # Add /sbin to the path for ocfs2 tools
> > +export PATH=$PATH:.
> > +
> > +USERNAME=`/usr/bin/whoami`
> > +DATE=`/bin/date +%F-%H-%M`
> > +DF=`which df`
> > +GREP=`which grep`
> > +AWK=`which awk`
> > +ECHO="`which echo` -e"
> > +
> > +SUDO="`which sudo` -u root"
> > +DEBUGFS_BIN="`which sudo` -u root `which debugfs.ocfs2`"
> > +TUNEFS_BIN="`which sudo` -u root `which tunefs.ocfs2`"
> > +MKFS_BIN="`which sudo` -u root `which mkfs.ocfs2`"
> > +
> > +REMOTE_MOUNT_BIN="`which sudo` -u root `which remote_mount.py`"
> > +REMOTE_UMOUNT_BIN="`which sudo` -u root `which remote_umount.py`"
> > +
> > +NODE_LIST=
> > +DEVICE_NAME=
> > +MOUNT_POINT=
> > +
> > +################################################################################
> > +# Utility Functions
> > +################################################################################
> > +f_usage()
> > +{
> > + echo "usage: `basename ${0}` <-n nodes> <-d device> <mountpoint path>"
> > + echo " -n nodelist,should be comma separated."
> > + echo " -d device name used for ocfs2 volume."
> > + echo " <mountpoint path> path of mountpoint where test will be performed."
> > + echo
> > + echo "Eaxamples:"
> > + echo " `basename ${0}` -n node1.us.oracle.com,node2.us.oracle.com -d /dev/sdd1 /storage"
> > + exit 1;
> > +
> > +}
> > +
> > +f_getoptions()
> > +{
> > + if [ $# -eq 0 ]; then
> > + f_usage;
> > + exit 1
> > + fi
> > +
> > + while getopts "n:d:h:" options; do
> > + case $options in
> > + n ) NODE_LIST="$OPTARG";;
> > + d ) DEVICE_NAME="$OPTARG";;
> > + h ) f_usage
> > + exit 1;;
> > + * ) f_usage
> > + exit 1;;
> > + esac
> > + done
> > + shift $(($OPTIND -1))
> > + MOUNT_POINT=${1}
> > +
> > +}
> > +
> > +f_setup()
> > +{
> > + f_getoptions $*
> > +
> > + if [ -z "${MOUNT_POINT}" ];then
> > + f_usage
> > + fi
> > +
> > + . ./config.sh
> > +
> > + LOGFILE=${O2TDIR}/log/single_run_${DATE}.log
> > +
> > + if [ ! -d ${MOUNT_POINT} -o ! -w ${MOUNT_POINT} ]; then
> > + ${ECHO} "Mount point ${MOUNT_POINT} does not exist or is not writable" \
> > + |tee -a ${LOGFILE};
> > + exit 1
> > + fi
> > +
> > + ${DF} -h|${GREP} -q ${DEVICE_NAME}
> > + if [ "$?" == "0" ];then
> > + ${ECHO} "Partition has been mounted,should umount first to perform test" \
> > + |tee -a ${LOGFILE};
> > + exit 1
> > + fi
> > +
> > +}
> > +
> > +LogRC()
> > +{
> > +if [ ${1} -ne 0 ]; then
> > + ${ECHO} "Failed." >> ${LOGFILE};
> > +else
> > + ${ECHO} "Passed." >> ${LOGFILE};
> > +fi;
> > +END=$(date +%s);
> > +DIFF=$(( ${END} - ${START} ));
> > +${ECHO} "Runtime ${DIFF} seconds.\n" >> ${LOGFILE};
> > +}
> > +
> > +LogMsg()
> > +{
> > +${ECHO} `date` >> ${LOGFILE};
> > +${ECHO} "${1}\c" >> ${LOGFILE};
> > +i=${#1};
> > +while (( i < 60 ))
> > +do
> > + ${ECHO} ".\c" >> ${LOGFILE};
> > + (( ++i ));
> > +done;
> > +}
> > +
> > +IsDeviceMounted()
> > +{
> > + local OCFS2_DEVICE=${1}
> > + ${DF} -h | ${GREP} -q ${OCFS2_DEVICE}
> > +
> > + if [ "$?" == "0" ]; then
> > + return 0
> > + else
> > + return 1
> > + fi
> > +}
> > +
> > +run_xattr_test()
> > +{
> > + LogMsg "xattr-test"
> > + ${BINDIR}/xattr-multi-run.sh -r 8 -f ${NODE_LIST} -a rsh -o ${O2TDIR}/log/xattr-tests-log -d ${DEVICE_NAME} ${MOUNT_POINT}
> > + LogRC $?
> > +}
> > +
> > +run_inline_test()
> > +{
> > + LogMsg "inline-test"
> > + ${BINDIR}/multi-inline-run.sh -r 4 -f ${NODE_LIST} -a rsh -o ${O2TDIR}/log/inline-tests-log -d ${DEVICE_NAME} ${MOUNT_POINT}
> > + LogRC $?
> > +
> > +}
> > +
> > +run_write_append_truncate_test()
> > +{
> > + LogMsg "write-append-truncate-test"
> > +
> > + local logdir=${O2TDIR}/log/write_append_truncate_log
> > + local logfile=${logdir}/write_append_truncate_${DATE}.log
> > +
> > + local testfile=${MOUNT_POINT}/write_append_truncate_test_file
> > +
> > + ${SUDO} mkdir -p ${logdir}
> > + #force to umount volume from all nodes
> > + ${ECHO} "Try to umount volume from all nodes before test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > +
> > + CLUSTERSIZE=32k
> > + BLOCKSIZE=4k
> > + SLOTS=4
> > + LABEL=oracle_home
> > +
> > + ${ECHO} "Format volume to launch new test"|tee -a ${logfile}
> > + echo y|${MKFS_BIN} -C ${CLUSTERSIZE} -b ${BLOCKSIZE} -N ${SLOTS} -L ${LABEL} ${DEVICE_NAME} || {
> > + ${ECHO} "Can not format ${DEVICE_NAME}"
> > + return 1
> > + }
> > +
> > + ${ECHO} "Mount volume to all nodes"|tee -a ${logfile}
> > + ${REMOTE_MOUNT_BIN} -l ${LABEL} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > + ${BINDIR}/run_write_append_truncate.py -i 20000 -l ${logfile} -n ${NODE_LIST} -f ${testfile}
> > +
> > + LogRC $?
> > +
> > + ${ECHO} "Umount volume from all nodes after test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +}
> > +
> > +run_multi_mmap_test()
> > +{
> > + LogMsg "multi-mmap-test"
> > +
> > + local logdir=${O2TDIR}/log/multi_mmap_log
> > + local logfile=${logdir}/multi_mmap_test_${DATE}.log
> > +
> > + local testfile=${MOUNT_POINT}/multi_mmap_test_file
> > +
> > + ${SUDO} mkdir -p ${logdir}
> > + #force to umount volume from all nodes
> > + ${ECHO} "Try to umount volume from all nodes before test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > +
> > + CLUSTERSIZE=32k
> > + BLOCKSIZE=4k
> > + SLOTS=4
> > + LABEL=oracle_home
> > +
> > + ${ECHO} "Format volume to launch new test"|tee -a ${logfile}
> > + echo y|${MKFS_BIN} -C ${CLUSTERSIZE} -b ${BLOCKSIZE} -N ${SLOTS} -L ${LABEL} ${DEVICE_NAME} || {
> > + ${ECHO} "Can not format ${DEVICE_NAME}"
> > + return 1
> > + }
> > +
> > + ${ECHO} "Mount volume to all nodes"|tee -a ${logfile}
> > + ${REMOTE_MOUNT_BIN} -l ${LABEL} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > + ${BINDIR}/run_multi_mmap.py -i 20000 -n ${NODE_LIST} -c -b 6000 --hole -f ${testfile} | tee -a ${logfile}
> > +
> > + LogRC $?
> > +
> > + ${ECHO} "Umount volume from all nodes after test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +}
> > +
> > +run_lvb_torture_test()
> > +{
> > + LogMsg "lvb_torture_test"
> > + :
> > + LogRC $?
> > +
> > +}
> > +
> > +run_create_racer_test()
> > +{
> > + LogMsg "create-racer-test"
> > +
> > + local logdir=${O2TDIR}/log/create_racer_log
> > + local logfile=${logdir}/create_racer_test_${DATE}.log
> > +
> > + local testpath=${MOUNT_POINT}
> > +
> > + ${SUDO} mkdir -p ${logdir}
> > + #force to umount volume from all nodes
> > + ${ECHO} "Try to umount volume from all nodes before test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > +
> > + CLUSTERSIZE=32k
> > + BLOCKSIZE=4k
> > + SLOTS=4
> > + LABEL=oracle_home
> > +
> > + ${ECHO} "Format volume to launch new test"|tee -a ${logfile}
> > + echo y|${MKFS_BIN} -C ${CLUSTERSIZE} -b ${BLOCKSIZE} -N ${SLOTS} -L ${LABEL} ${DEVICE_NAME} || {
> > + ${ECHO} "Can not format ${DEVICE_NAME}"
> > + return 1
> > + }
> > +
> > + ${ECHO} "Mount volume to all nodes"|tee -a ${logfile}
> > + ${REMOTE_MOUNT_BIN} -l ${LABEL} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > + ${BINDIR}/run_create_racer.py -i 40000 -l ${logfile} -n ${NODE_LIST} -p ${testpath}
> > +
> > + LogRC $?
> > +
> > + ${ECHO} "Umount volume from all nodes after test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +}
> > +
> > +run_flock_tests()
> > +{
> > + LogMsg "flock-tests"
> > +
> > + local logdir=${O2TDIR}/log/flock_log
> > + local flock_logfile=${logdir}/flock_test_${DATE}.log
> > + local fcntl_logfile=${logdir}/fcntl_test_${DATE}.log
> > +
> > + local testfile1=${MOUNT_POINT}/flock_test_file1
> > + local testfile2=${MOUNT_POINT}/flock_test_file2
> > +
> > + ${SUDO} touch ${testfile1}
> > + ${SUDO} touch ${testfile2}
> > +
> > + ${SUDO} mkdir -p ${logdir}
> > + #force to umount volume from all nodes
> > + ${ECHO} "Try to umount volume from all nodes before test."|tee -a ${flock_logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${flock_logfile} 2>&1
> > +
> > +
> > + CLUSTERSIZE=32k
> > + BLOCKSIZE=4k
> > + SLOTS=4
> > + LABEL=oracle_home
> > +
> > + ${ECHO} "Format volume to launch new test"|tee -a ${flock_logfile}
> > + echo y|${MKFS_BIN} -C ${CLUSTERSIZE} -b ${BLOCKSIZE} -N ${SLOTS} -L ${LABEL} ${DEVICE_NAME} || {
> > + ${ECHO} "Can not format ${DEVICE_NAME}"
> > + return 1
> > + }
> > +
> > + ${ECHO} "Mount volume to all nodes"|tee -a ${flock_logfile}
> > + ${REMOTE_MOUNT_BIN} -l ${LABEL} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${flock_logfile} 2>&1
> > +
> > + ${BINDIR}/run_flock_unit_test.py -l ${fcntl_logfile} -n ${NODE_LIST} -t fcntl -e ${testfile1} -f ${testfile2} \
> > + && \
> > + ${BINDIR}/run_flock_unit_test.py -l ${flock_logfile} -n ${NODE_LIST} -t flock -e ${testfile1} -f ${testfile2}
> > +
> > + LogRC $?
> > +
> > + ${ECHO} "Umount volume from all nodes after test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > +}
> > +
> > +run_open_delete_test()
> > +{
> > + LogMsg "open-delete-test"
> > +
> > + local logdir=${O2TDIR}/log/open_delete_log
> > + local logfile=${logdir}/open_delete_test_${DATE}.log
> > +
> > + local testfile=${MOUNT_POINT}/open_delete_test_file
> > +
> > + ${SUDO} mkdir -p ${logdir}
> > + #force to umount volume from all nodes
> > + ${ECHO} "Try to umount volume from all nodes before test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > +
> > + CLUSTERSIZE=32k
> > + BLOCKSIZE=4k
> > + SLOTS=4
> > + LABEL=oracle_home
> > +
> > + ${ECHO} "Format volume to launch new test"|tee -a ${logfile}
> > + echo y|${MKFS_BIN} -C ${CLUSTERSIZE} -b ${BLOCKSIZE} -N ${SLOTS} -L ${LABEL} ${DEVICE_NAME} || {
> > + ${ECHO} "Can not format ${DEVICE_NAME}"
> > + return 1
> > + }
> > +
> > + ${ECHO} "Mount volume to all nodes"|tee -a ${logfile}
> > + ${REMOTE_MOUNT_BIN} -l ${LABEL} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +
> > + ${BINDIR}/open_delete.py -f ${testfile} -i 10000 -l ${logfile} -n ${NODE_LIST}
> > +
> > + LogRC $?
> > +
> > + ${ECHO} "Umount volume from all nodes after test."|tee -a ${logfile}
> > + ${REMOTE_UMOUNT_BIN} -m ${MOUNT_POINT} -n ${NODE_LIST}>>${logfile} 2>&1
> > +}
> > +
> > +f_cleanup()
> > +{
> > + :
> > +}
> > +################################################################################
> > +# Main Entry
> > +################################################################################
> > +
> > +trap 'echo -ne "\n\n">>${LOGFILE};echo "Interrupted by Ctrl+C,Cleanuping... "|tee -a ${LOGFILE}; f_cleanup;exit 1' SIGINT
> > +trap ' : ' SIGTERM
> > +
> > +f_setup $*
> > +
> > +STARTRUN=$(date +%s)
> > +${ECHO} "`date` - Starting Multiple Nodes Regress test" > ${LOGFILE}
> > +
> > +START=$(date +%s)
> > +run_xattr_test
> > +
> > +START=$(date +%s)
> > +run_inline_test
> > +
> > +START=$(date +%s)
> > +run_write_append_truncate_test
> > +
> > +START=$(date +%s)
> > +run_multi_mmap_test
> > +
> > +START=$(date +%s)
> > +run_lvb_torture_test
> > +
> > +START=$(date +%s)
> > +run_create_racer_test
> > +
> > +START=$(date +%s)
> > +run_flock_tests
> > +
> > +START=$(date +%s)
> > +run_open_delete_test
> > +
> > +END=$(date +%s)
> > +DIFF=$(( ${END} - ${STARTRUN} ));
> > +${ECHO} "Total Runtime ${DIFF} seconds.\n" >> ${LOGFILE}
> > +${ECHO} "`date` - Ended Multiple Nodes Regress test" >> ${LOGFILE}
> > +
> > +
> >
>
More information about the Ocfs2-tools-devel
mailing list