### RUNNING '000-save_sysinfo' ### ========== /proc/buddyinfo: ========== ========== /proc/cgroups: ========== ========== /proc/cmdline: ========== ========== /proc/cpuinfo: ========== ========== /proc/crypto: ========== ========== /proc/devices: ========== ========== /proc/diskstats: ========== ========== /proc/dma: ========== ========== /proc/execdomains: ========== ========== /proc/fb: ========== ========== /proc/filesystems: ========== ========== /proc/hpqci: ========== ========== /proc/interrupts: ========== ========== /proc/iomem: ========== ========== /proc/ioports: ========== ========== /proc/keys: ========== ========== /proc/key-users: ========== ========== /proc/latency_stats: ========== ========== /proc/loadavg: ========== ========== /proc/locks: ========== ========== /proc/mdstat: ========== ========== /proc/meminfo: ========== ========== /proc/misc: ========== ========== /proc/modules: ========== ========== /proc/mounts: ========== ========== /proc/mtrr: ========== ========== /proc/pagetypeinfo: ========== ========== /proc/partitions: ========== ========== /proc/sched_debug: ========== ========== /proc/schedstat: ========== ========== /proc/slabinfo: ========== ========== /proc/stat: ========== ========== /proc/swaps: ========== ========== /proc/timer_list: ========== ========== /proc/timer_stats: ========== ========== /proc/uptime: ========== ========== /proc/version: ========== ========== /proc/vmallocinfo: ========== ========== /proc/vmstat: ========== ========== /proc/zoneinfo: ========== ========== /proc/config.gz: ========== proc files dumped ### RUNNING '010-kernel-tuning.sh' ### run kernel-tuning.sh: change IO scheduler: noop anticipatory deadline [cfq] ### RUNNING '020-install_packages' ### ca-ostest293: Attempting to install packages... Installing ltp-tstres... ltp-tstres not found, attempting to lookup /cruc/packages/ltp-tstres found, attempting to get latest pkg_dir: /cruc/packages Installing 'ltp-tstres' 'ltp-tstres-001.tar.gz' source /cruc/packages/package.fns source /cruc/packages/ltp-tstres/ltp-tstres.fns action_unpack /cruc/packages/ltp-tstres/ltp-tstres-001.tar.gz /usr/builds ltp-tstres-001 >> /cruc/runs/6428/logs/ca-ostest293/ltp-tstres-001.log action_patch action_configure /usr/builds/ltp-tstres-001 action_build action_check action_install ### RUNNING '030-hp_read_power' ### ### RUNNING '040-runtest_oltest_power' ### oltest 0 INFO : BEGIN PACKAGE oltest oltest 0 INFO : Running oltest on SUT: ca-ostest293 /cruc/runs/6428/RUNNING/ca-ostest293/040-runtest_oltest_power: line 62: kernel-tuning.sh: command not found Cleaning process containing - ora_ sqlplus make olt-archive-testlogs olt-execute run-test runtest gzip-scp.sh auto.sh shtriage status-listener iostat vmstat mpstat sar test-syncer sleep olt-synctestcases opmn httpd tns stressdb apache lockfile find diagnostic fifo tnslsnr rac-st rac-node rac-dest olt-check-schedule dest- mempressure nfs-hang nfs_hang dbt2 dbt3 oci oracledbt2 oradbt driver lsnr listener attach strace install silent bufferhead fio sar sadc aio_writetest customtest xm-stress xm-dom-test sv-ovs ... Cleaning up ipcs ... Cleaning up tests completed insmod: error inserting '/usr/builds/2.6.30-rc7/hpilof.ko': -1 File exists hpilof insmod failed Using /cruc/packages/oltest/validate-power.cmd Scheduled command file: validate-power.cmd_ca-ostest293.us.oracle.com! Schedule Log file: /opt/oracle/oltest/olt-schedule/log/validate-power.cmd_ca-ostest293.us.oracle.com.log The suite summary log file (csv format) for all nodes is: /opt/oracle/oltest/olt-schedule/log/olt-summary.csv To get the current the status of scheduled tests: olt-schedule -f -s oltest 0 INFO : using: /usr/bin/olt-schedule -f /cruc/packages/oltest/validate-power.cmd -r oltest 0 INFO : oltest >>> Node : ca-ostest293.us.oracle.com - testsuite "install-silent-oracle-11106" has PASSED >>> Node : ca-ostest293.us.oracle.com - 1 out of 6 testcases in testsuite "ft-aio-dio" have FAILED OLT log results file: cat: /opt/oracle/oltest/olt-schedule/log/validate-power.cmd_ca-ostest293.us.oracle.com.log: No such file or directory ========== --- --- --- Starting Log for (validate-power.cmd_ca-ostest293.us.oracle.com) @ Wed Jun 24 03:50:24 PDT 2009 --- --- --- INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Checking if nothing else is running on the remote node ... INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - [If it hangs here for more than 60secs then tests have been scheduled by someone else using QAMonitor (AE or olt-execute) INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - I will wait till these tests complete INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - To find out what is running on the remote box run: olt-get-status -n ca-ostest293.us.oracle.com] INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Nothing else is running now. Going to schedule... INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Obtain n/w lock: This can hang for a very long time in case of already scheduled long running tests on the node INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - [If a previous test hung or m/c restarted you may have to clean the lock file manually under /tmp/olt-node_schedule.lck on the remote node] INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - N/w lock Obtained cat: /make.pid: No such file or directory FORCE_EXECUTE_OPTION = 0 REMOTE_OLT_SCHEDULE_WORKQA_DIR=/home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/olt-schedule_work INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Scheduled tests install-silent-oracle-11106 INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Getting the pid to poll for test completion INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Polling on 22809... echo "Description: Verify if Oracle 11.1.0.6 installs and relinks" Description: Verify if Oracle 11.1.0.6 installs and relinks Executing Test install-silent-oracle-11106 Creating the QA_HOME work dir (/home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661) for test execution start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661 /home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661/results PRODUCT_NAME=oracle:PRODUCT_VERSION=11.1.0.6:UNIX_GROUP_NAME=g502:ORACLE_HOME_NAME=ohome_ora_11106:OUI_TIMEOUT=150:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661/results Default runtest not found in path - looking for a custom script - /opt/oracle/oltest/.srchome/si-install/si-install Setting TEST_SCRIPT to /opt/oracle/oltest/.srchome/si-install/si-install with params - /home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661 /home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661/results PRODUCT_NAME=oracle:PRODUCT_VERSION=11.1.0.6:UNIX_GROUP_NAME=g502:ORACLE_HOME_NAME=ohome_ora_11106:OUI_TIMEOUT=150:POSTREBOOT=0 1 tasks running: /home/oracle/work/install-silent-oracle-11106_Jun24_09-03-50-28/install-silent-oracle-11106_Jun-24-03-50-28-429879324-24661/results --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting install-silent-oracle-11106 ; exit 0 FORCE_EXECUTE_OPTION = 0 REMOTE_OLT_SCHEDULE_WORKQA_DIR=/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Scheduled tests ft-aio-dio INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Getting the pid to poll for test completion INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Polling on 28051... echo "Description: Verify combined aio and dio functionality with Oracle ( Can be used to check this across drivers )" Description: Verify combined aio and dio functionality with Oracle ( Can be used to check this across drivers ) make[1]: Entering directory `/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work' Executing Test ft-aio-dio-dbt2_1 Executing Test ft-aio-dio-dbt2_1-attach Configuring the Oracle Home ... Configuring the Oracle Home ... Create the necessary directory structures... Sucessfully created the QA Home - /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242 for test oltdbt2 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242 /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work single ORACLE_VERSION=11.1.0.6:DCG=9G:USERS=100:DURATION=300:ODIRECT=true:AIO=true:DBBLOCKSIZE=8192:STORAGETYPE=local:FILESYSTEMLOC=/olt-storage/local:numtests=2:MODE=io:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work Using already running instance of QA Monitor Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1-attach_Jun-24-03-55-05-119834405-32150 /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1-attach_Jun-24-03-55-05-119834405-32150/work aio_dio ORACLE_VERSION=11.1.0.6:numtests=2:FILESYSTEMLOC=/olt-storage/local:FILESYSTEMDETAILS=local:POSTREBOOT=0; 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work +++ Task added 2 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1-attach_Jun-24-03-55-05-119834405-32150/work 2 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1-attach_Jun-24-03-55-05-119834405-32150/work /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work--- Task removed 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work Done runtest Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_1-attach ; exit 0 Done excuting ft-aio-dio-dbt2_1-attach ; exit 0 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_1_Jun-24-03-55-05-119924124-31242/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_1 ; exit 0 make[1]: Leaving directory `/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work' make[1]: Entering directory `/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work' Executing Test ft-aio-dio-dbt2_2 Executing Test ft-aio-dio-dbt2_2-attach Configuring the Oracle Home ... Configuring the Oracle Home ... Create the necessary directory structures... Sucessfully created the QA Home - /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2_Jun-24-04-07-41-441706372-13563 for test oltdbt2 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2_Jun-24-04-07-41-441706372-13563 /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2_Jun-24-04-07-41-441706372-13563/work single ORACLE_VERSION=11.1.0.6:DCG=9G:USERS=100:DURATION=300:ODIRECT=true:AIO=true:DBBLOCKSIZE=8192:STORAGETYPE=ocfs:FILESYSTEMLOC=/olt-storage/local/ocfs:numtests=2:MODE=io:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2_Jun-24-04-07-41-441706372-13563/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2_Jun-24-04-07-41-441706372-13563/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_2 ; exit 0 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2-attach_Jun-24-04-07-41-452844059-31310 /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2-attach_Jun-24-04-07-41-452844059-31310/work aio_dio ORACLE_VERSION=11.1.0.6:numtests=2:FILESYSTEMLOC=/olt-storage/local/ocfs:FILESYSTEMDETAILS=ocfs:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2-attach_Jun-24-04-07-41-452844059-31310/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_2-attach_Jun-24-04-07-41-452844059-31310/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_2-attach ; exit 0 Done excuting ft-aio-dio-dbt2_2-attach ; exit 0 make[1]: Leaving directory `/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work' make[1]: Entering directory `/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work' Executing Test ft-aio-dio-dbt2_3 Executing Test ft-aio-dio-dbt2_3-attach Configuring the Oracle Home ... Configuring the Oracle Home ... Create the necessary directory structures... Sucessfully created the QA Home - /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3_Jun-24-04-08-42-308671202-2952 for test oltdbt2 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3_Jun-24-04-08-42-308671202-2952 /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3_Jun-24-04-08-42-308671202-2952/work single ORACLE_VERSION=11.1.0.6:DCG=9G:USERS=100:DURATION=300:ODIRECT=true:AIO=true:DBBLOCKSIZE=8192:STORAGETYPE=nas:FILESYSTEMLOC=/olt-storage/local/nas:numtests=2:MODE=io:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3_Jun-24-04-08-42-308671202-2952/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3_Jun-24-04-08-42-308671202-2952/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_3 ; exit 0 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3-attach_Jun-24-04-08-42-314283914-20699 /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3-attach_Jun-24-04-08-42-314283914-20699/work aio_dio ORACLE_VERSION=11.1.0.6:numtests=2:FILESYSTEMLOC=/olt-storage/local/nas:FILESYSTEMDETAILS=nas:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3-attach_Jun-24-04-08-42-314283914-20699/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/ft-aio-dio-dbt2_3-attach_Jun-24-04-08-42-314283914-20699/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_3-attach ; exit 0 Done excuting ft-aio-dio-dbt2_3-attach ; exit 0 make[1]: Leaving directory `/home/oracle/work/ft-aio-dio_Jun24_09-03-55-04/olt-schedule_work' INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Removing the network schedule lock file now ========== oltest 0 INFO : fini ### RUNNING '050-hp_read_power_kill' ### hp_read_power: no process killed ### RUNNING '060-power-avg.sh' ### power average: 141 with 115 samples ### RUNNING '070-save_sysinfo' ### Warning: Run directory '/cruc/runs/6428/sysinfo/ca-ostest293.1' already exists. Incrementing... ========== /proc/buddyinfo: ========== ========== /proc/cgroups: ========== ========== /proc/cmdline: ========== ========== /proc/cpuinfo: ========== ========== /proc/crypto: ========== ========== /proc/devices: ========== ========== /proc/diskstats: ========== ========== /proc/dma: ========== ========== /proc/execdomains: ========== ========== /proc/fb: ========== ========== /proc/filesystems: ========== ========== /proc/hpqci: ========== ========== /proc/interrupts: ========== ========== /proc/iomem: ========== ========== /proc/ioports: ========== ========== /proc/keys: ========== ========== /proc/key-users: ========== ========== /proc/latency_stats: ========== ========== /proc/loadavg: ========== ========== /proc/locks: ========== ========== /proc/mdstat: ========== ========== /proc/meminfo: ========== ========== /proc/misc: ========== ========== /proc/modules: ========== ========== /proc/mounts: ========== ========== /proc/mtrr: ========== ========== /proc/pagetypeinfo: ========== ========== /proc/partitions: ========== ========== /proc/sched_debug: ========== ========== /proc/schedstat: ========== ========== /proc/slabinfo: ========== ========== /proc/stat: ========== ========== /proc/swaps: ========== ========== /proc/timer_list: ========== ========== /proc/timer_stats: ========== ========== /proc/uptime: ========== ========== /proc/version: ========== ========== /proc/vmallocinfo: ========== ========== /proc/vmstat: ========== ========== /proc/zoneinfo: ========== ========== /proc/config.gz: ========== proc files dumped