### RUNNING '000-save_sysinfo' ### ========== /proc/buddyinfo: ========== ========== /proc/cgroups: ========== ========== /proc/cmdline: ========== ========== /proc/cpuinfo: ========== ========== /proc/crypto: ========== ========== /proc/devices: ========== ========== /proc/diskstats: ========== ========== /proc/dma: ========== ========== /proc/execdomains: ========== ========== /proc/fb: ========== ========== /proc/filesystems: ========== ========== /proc/interrupts: ========== ========== /proc/iomem: ========== ========== /proc/ioports: ========== ========== /proc/key-users: ========== ========== /proc/loadavg: ========== ========== /proc/lockdep: ========== ========== /proc/lockdep_stats: ========== ========== /proc/locks: ========== ========== /proc/meminfo: ========== ========== /proc/misc: ========== ========== /proc/modules: ========== ========== /proc/mounts: ========== ========== /proc/mtrr: ========== ========== /proc/pagetypeinfo: ========== ========== /proc/partitions: ========== ========== /proc/schedstat: ========== ========== /proc/slabinfo: ========== ========== /proc/stat: ========== ========== /proc/swaps: ========== ========== /proc/timer_list: ========== ========== /proc/timer_stats: ========== ========== /proc/uptime: ========== ========== /proc/version: ========== ========== /proc/vmallocinfo: ========== ========== /proc/vmstat: ========== ========== /proc/zoneinfo: ========== ========== /proc/config.gz: ========== proc files dumped ### RUNNING '010-kernel-tuning.sh' ### run kernel-tuning.sh: change IO scheduler: [noop] anticipatory deadline cfq ### RUNNING '020-install_packages' ### ca-ostest293: Attempting to install packages... Installing ltp-tstres... ltp-tstres not found, attempting to lookup /cruc/packages/ltp-tstres found, attempting to get latest pkg_dir: /cruc/packages Installing 'ltp-tstres' 'ltp-tstres-001.tar.gz' source /cruc/packages/package.fns source /cruc/packages/ltp-tstres/ltp-tstres.fns action_unpack /cruc/packages/ltp-tstres/ltp-tstres-001.tar.gz /usr/builds ltp-tstres-001 >> /cruc/runs/6384/logs/ca-ostest293/ltp-tstres-001.log action_patch action_configure /usr/builds/ltp-tstres-001 action_build action_check action_install Installing hpilof... hpilof not found, attempting to lookup /cruc/packages/hpilof found, attempting to get latest pkg_dir: /cruc/packages Installing 'hpilof' 'hpilof-8.0.0-144.OB.tar.gz' source /cruc/packages/package.fns source /cruc/packages/hpilof/hpilof.fns action_unpack /cruc/packages/hpilof/hpilof-8.0.0-144.OB.tar.gz /usr/builds hpilof-8.0.0-144.OB >> /cruc/runs/6384/logs/ca-ostest293/hpilof-8.0.0-144.OB.log action_patch action_configure /usr/builds/hpilof-8.0.0-144.OB action_build ERROR(2): Failed to build /usr/builds/hpilof-8.0.0-144.OB Error: hpilof did not install. Continuing with next... ### RUNNING '030-hp_read_power' ### ### RUNNING '040-runtest_oltest_power' ### oltest 0 INFO : BEGIN PACKAGE oltest oltest 0 INFO : Running oltest on SUT: ca-ostest293 /cruc/runs/6384/RUNNING/ca-ostest293/040-runtest_oltest_power: line 62: kernel-tuning.sh: command not found Cleaning process containing - ora_ sqlplus make olt-archive-testlogs olt-execute run-test runtest gzip-scp.sh auto.sh shtriage status-listener iostat vmstat mpstat sar test-syncer sleep olt-synctestcases opmn httpd tns stressdb apache lockfile find diagnostic fifo tnslsnr rac-st rac-node rac-dest olt-check-schedule dest- mempressure nfs-hang nfs_hang dbt2 dbt3 oci oracledbt2 oradbt driver lsnr listener attach strace install silent bufferhead fio sar sadc aio_writetest customtest xm-stress xm-dom-test sv-ovs ... Cleaning up ipcs ... Cleaning up tests completed insmod: can't read '/lib/modules/2.6.30-rc8/kernel/drivers/misc/hpilof.ko': No such file or directory hpilof insmod failed Using /cruc/packages/oltest/validate-power.cmd Scheduled command file: validate-power.cmd_ca-ostest293.us.oracle.com! Schedule Log file: /opt/oracle/oltest/olt-schedule/log/validate-power.cmd_ca-ostest293.us.oracle.com.log The suite summary log file (csv format) for all nodes is: /opt/oracle/oltest/olt-schedule/log/olt-summary.csv To get the current the status of scheduled tests: olt-schedule -f -s oltest 0 INFO : using: /usr/bin/olt-schedule -f /cruc/packages/oltest/validate-power.cmd -r oltest 0 INFO : oltest >>> Node : ca-ostest293.us.oracle.com - testsuite "install-silent-oracle-11106" has PASSED >>> Node : ca-ostest293.us.oracle.com - 1 out of 6 testcases in testsuite "ft-aio-dio" have FAILED OLT log results file: cat: /opt/oracle/oltest/olt-schedule/log/validate-power.cmd_ca-ostest293.us.oracle.com.log: No such file or directory ========== --- --- --- Starting Log for (validate-power.cmd_ca-ostest293.us.oracle.com) @ Wed Jun 03 08:55:31 PDT 2009 --- --- --- INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Checking if nothing else is running on the remote node ... INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - [If it hangs here for more than 60secs then tests have been scheduled by someone else using QAMonitor (AE or olt-execute) INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - I will wait till these tests complete INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - To find out what is running on the remote box run: olt-get-status -n ca-ostest293.us.oracle.com] INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Nothing else is running now. Going to schedule... INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Obtain n/w lock: This can hang for a very long time in case of already scheduled long running tests on the node INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - [If a previous test hung or m/c restarted you may have to clean the lock file manually under /tmp/olt-node_schedule.lck on the remote node] INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - N/w lock Obtained cat: /make.pid: No such file or directory FORCE_EXECUTE_OPTION = 0 REMOTE_OLT_SCHEDULE_WORKQA_DIR=/home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/olt-schedule_work INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Scheduled tests install-silent-oracle-11106 INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Getting the pid to poll for test completion INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Polling on 8279... echo "Description: Verify if Oracle 11.1.0.6 installs and relinks" Description: Verify if Oracle 11.1.0.6 installs and relinks Executing Test install-silent-oracle-11106 Creating the QA_HOME work dir (/home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813) for test execution start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813 /home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813/results PRODUCT_NAME=oracle:PRODUCT_VERSION=11.1.0.6:UNIX_GROUP_NAME=g502:ORACLE_HOME_NAME=ohome_ora_11106:OUI_TIMEOUT=150:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813/results Default runtest not found in path - looking for a custom script - /opt/oracle/oltest/.srchome/si-install/si-install Setting TEST_SCRIPT to /opt/oracle/oltest/.srchome/si-install/si-install with params - /home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813 /home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813/results PRODUCT_NAME=oracle:PRODUCT_VERSION=11.1.0.6:UNIX_GROUP_NAME=g502:ORACLE_HOME_NAME=ohome_ora_11106:OUI_TIMEOUT=150:POSTREBOOT=0 1 tasks running: /home/oracle/work/install-silent-oracle-11106_Jun03_09-08-55-35/install-silent-oracle-11106_Jun-03-08-55-35-588864641-19813/results --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting install-silent-oracle-11106 ; exit 0 FORCE_EXECUTE_OPTION = 0 REMOTE_OLT_SCHEDULE_WORKQA_DIR=/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Scheduled tests ft-aio-dio INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Getting the pid to poll for test completion INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Polling on 13591... echo "Description: Verify combined aio and dio functionality with Oracle ( Can be used to check this across drivers )" Description: Verify combined aio and dio functionality with Oracle ( Can be used to check this across drivers ) make[1]: Entering directory `/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work' Executing Test ft-aio-dio-dbt2_1 Executing Test ft-aio-dio-dbt2_1-attach Configuring the Oracle Home ... Configuring the Oracle Home ... Sucessfully created the QA Home - /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442 for test oltdbt2 Create the necessary directory structures... start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442 /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work single ORACLE_VERSION=11.1.0.6:DCG=9G:USERS=100:DURATION=300:ODIRECT=true:AIO=true:DBBLOCKSIZE=8192:STORAGETYPE=local:FILESYSTEMLOC=/olt-storage/local:numtests=2:MODE=io:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work Using already running instance of QA Monitor Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1-attach_Jun-03-09-01-12-380581840-14351 /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1-attach_Jun-03-09-01-12-380581840-14351/work aio_dio ORACLE_VERSION=11.1.0.6:numtests=2:FILESYSTEMLOC=/olt-storage/local:FILESYSTEMDETAILS=local:POSTREBOOT=0; 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work +++ Task added 2 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1-attach_Jun-03-09-01-12-380581840-14351/work 2 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1-attach_Jun-03-09-01-12-380581840-14351/work /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work--- Task removed 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work Done runtest Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_1-attach ; exit 0 Done excuting ft-aio-dio-dbt2_1-attach ; exit 0 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_1_Jun-03-09-01-12-378740441-13442/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_1 ; exit 0 make[1]: Leaving directory `/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work' make[1]: Entering directory `/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work' Executing Test ft-aio-dio-dbt2_2 Executing Test ft-aio-dio-dbt2_2-attach Configuring the Oracle Home ... Create the necessary directory structures... Configuring the Oracle Home ... Sucessfully created the QA Home - /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2_Jun-03-09-13-28-254973321-8282 for test oltdbt2 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2_Jun-03-09-13-28-254973321-8282 /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2_Jun-03-09-13-28-254973321-8282/work single ORACLE_VERSION=11.1.0.6:DCG=9G:USERS=100:DURATION=300:ODIRECT=true:AIO=true:DBBLOCKSIZE=8192:STORAGETYPE=ocfs:FILESYSTEMLOC=/olt-storage/local/ocfs:numtests=2:MODE=io:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2_Jun-03-09-13-28-254973321-8282/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2_Jun-03-09-13-28-254973321-8282/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_2 ; exit 0 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2-attach_Jun-03-09-13-28-242375520-23303 /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2-attach_Jun-03-09-13-28-242375520-23303/work aio_dio ORACLE_VERSION=11.1.0.6:numtests=2:FILESYSTEMLOC=/olt-storage/local/ocfs:FILESYSTEMDETAILS=ocfs:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2-attach_Jun-03-09-13-28-242375520-23303/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_2-attach_Jun-03-09-13-28-242375520-23303/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_2-attach ; exit 0 Done excuting ft-aio-dio-dbt2_2-attach ; exit 0 make[1]: Leaving directory `/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work' make[1]: Entering directory `/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work' Executing Test ft-aio-dio-dbt2_3 Executing Test ft-aio-dio-dbt2_3-attach Configuring the Oracle Home ... Configuring the Oracle Home ... Create the necessary directory structures... Sucessfully created the QA Home - /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3_Jun-03-09-14-28-173873321-16520 for test oltdbt2 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3_Jun-03-09-14-28-173873321-16520 /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3_Jun-03-09-14-28-173873321-16520/work single ORACLE_VERSION=11.1.0.6:DCG=9G:USERS=100:DURATION=300:ODIRECT=true:AIO=true:DBBLOCKSIZE=8192:STORAGETYPE=nas:FILESYSTEMLOC=/olt-storage/local/nas:numtests=2:MODE=io:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3_Jun-03-09-14-28-173873321-16520/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3_Jun-03-09-14-28-173873321-16520/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_3 ; exit 0 start-triage successful Calling /opt/oracle/oltest/olt-schedule/qamonitor/utils/auto.sh /x /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3-attach_Jun-03-09-14-28-179628040-1499 /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3-attach_Jun-03-09-14-28-179628040-1499/work aio_dio ORACLE_VERSION=11.1.0.6:numtests=2:FILESYSTEMLOC=/olt-storage/local/nas:FILESYSTEMDETAILS=nas:POSTREBOOT=0; 0 tasks running: +++ Task added 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3-attach_Jun-03-09-14-28-179628040-1499/work 1 tasks running: /home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/ft-aio-dio-dbt2_3-attach_Jun-03-09-14-28-179628040-1499/work --- Task removed 0 tasks running: Done runtest /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: stopped shtriage 0 tasks running: /opt/oracle/oltest/olt-schedule/qamonitor/stop-triage: Copying system logs Warning: Test logs will not be archived. Variable ARCHIVE_TEST_LOG_HOST is not set Enterprise Linux Enterprise Linux Server release 5 (Carthage) Done excuting ft-aio-dio-dbt2_3-attach ; exit 0 Done excuting ft-aio-dio-dbt2_3-attach ; exit 0 make[1]: Leaving directory `/home/oracle/work/ft-aio-dio_Jun03_09-09-01-12/olt-schedule_work' INFO: (validate-power.cmd_ca-ostest293.us.oracle.com) - Removing the network schedule lock file now ========== oltest 0 INFO : fini ### RUNNING '050-hp_read_power_kill' ### hp_read_power: no process killed ### RUNNING '060-power-avg.sh' ### power average: 0 with 0 samples ### RUNNING '070-save_sysinfo' ### Warning: Run directory '/cruc/runs/6384/sysinfo/ca-ostest293.1' already exists. Incrementing... ========== /proc/buddyinfo: ========== ========== /proc/cgroups: ========== ========== /proc/cmdline: ========== ========== /proc/cpuinfo: ========== ========== /proc/crypto: ========== ========== /proc/devices: ========== ========== /proc/diskstats: ========== ========== /proc/dma: ========== ========== /proc/execdomains: ========== ========== /proc/fb: ========== ========== /proc/filesystems: ========== ========== /proc/interrupts: ========== ========== /proc/iomem: ========== ========== /proc/ioports: ========== ========== /proc/key-users: ========== ========== /proc/loadavg: ========== ========== /proc/lockdep: ========== ========== /proc/lockdep_stats: ========== ========== /proc/locks: ========== ========== /proc/meminfo: ========== ========== /proc/misc: ========== ========== /proc/modules: ========== ========== /proc/mounts: ========== ========== /proc/mtrr: ========== ========== /proc/pagetypeinfo: ========== ========== /proc/partitions: ========== ========== /proc/schedstat: ========== ========== /proc/slabinfo: ========== ========== /proc/stat: ========== ========== /proc/swaps: ========== ========== /proc/timer_list: ========== ========== /proc/timer_stats: ========== ========== /proc/uptime: ========== ========== /proc/version: ========== ========== /proc/vmallocinfo: ========== ========== /proc/vmstat: ========== ========== /proc/zoneinfo: ========== ========== /proc/config.gz: ========== proc files dumped