2012-05-15 08:04:45,521 INFO repeater task wh300_lastensq starting 0.5 seconds late 2012-05-15 08:04:50,659 INFO repeater task wh300_lastensq ended after 5.1 seconds 2012-05-15 08:04:50,660 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:04:35 2012-05-15 08:04:50,662 INFO repeater task os75bb_lastensq starting 5.7 seconds late 2012-05-15 08:04:59,739 INFO repeater task os75bb_lastensq ended after 9.1 seconds 2012-05-15 08:04:59,739 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:04:35 2012-05-15 08:04:59,747 INFO repeater task os75nb_lastensq starting 14.7 seconds late 2012-05-15 08:05:04,179 INFO repeater task os75nb_lastensq ended after 4.4 seconds 2012-05-15 08:05:04,180 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:04:35 2012-05-15 08:06:49,949 INFO repeater task wh300_lastensq starting 4.9 seconds late 2012-05-15 08:06:54,657 INFO repeater task wh300_lastensq ended after 4.7 seconds 2012-05-15 08:06:54,657 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:06:35 2012-05-15 08:06:54,661 INFO repeater task wh300_quick starting 0.7 seconds late 2012-05-15 08:09:37,771 INFO repeater task wh300_quick ended after 163.1 seconds 2012-05-15 08:09:37,773 INFO repeater task wh300_3dayplots starting 163.8 seconds late 2012-05-15 08:10:32,398 INFO repeater task wh300_3dayplots ended after 54.6 seconds 2012-05-15 08:10:32,406 INFO repeater task os75bb_lastensq starting 47.4 seconds late 2012-05-15 08:10:42,059 INFO repeater task os75bb_lastensq ended after 9.7 seconds 2012-05-15 08:10:42,060 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:09:35 2012-05-15 08:10:42,061 INFO repeater task os75bb_quick starting 228.1 seconds late 2012-05-15 08:12:29,703 INFO repeater task os75bb_quick ended after 107.6 seconds 2012-05-15 08:12:29,707 INFO repeater task os75bb_3dayplots starting 335.7 seconds late 2012-05-15 08:13:25,137 INFO repeater task os75bb_3dayplots ended after 55.4 seconds 2012-05-15 08:13:25,138 INFO repeater task os75nb_lastensq starting 220.1 seconds late 2012-05-15 08:13:30,199 INFO repeater task os75nb_lastensq ended after 5.1 seconds 2012-05-15 08:13:30,200 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:09:35 2012-05-15 08:13:30,201 INFO repeater task os75nb_quick starting 396.2 seconds late 2012-05-15 08:15:05,383 INFO repeater task os75nb_quick ended after 95.2 seconds 2012-05-15 08:15:05,388 INFO repeater task os75nb_3dayplots starting 491.4 seconds late 2012-05-15 08:15:59,555 INFO repeater task os75nb_3dayplots ended after 54.2 seconds 2012-05-15 08:15:59,556 INFO repeater task wh300_beamplots starting 245.5 seconds late 2012-05-15 08:16:09,793 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:16:09,793 INFO repeater task wh300_beamplots ended after 10.2 seconds 2012-05-15 08:16:09,794 INFO repeater task wh300_beamplots starting 255.8 seconds late 2012-05-15 08:16:19,714 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:16:19,714 INFO repeater task wh300_beamplots ended after 9.9 seconds 2012-05-15 08:16:19,715 INFO repeater task os75bb_beamplots starting 265.7 seconds late 2012-05-15 08:16:25,069 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:16:25,069 INFO repeater task os75bb_beamplots ended after 5.4 seconds 2012-05-15 08:16:25,070 INFO repeater task os75bb_beamplots starting 271.1 seconds late 2012-05-15 08:16:30,119 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:16:30,119 INFO repeater task os75bb_beamplots ended after 5.0 seconds 2012-05-15 08:16:30,120 INFO repeater task os75nb_beamplots starting 276.1 seconds late 2012-05-15 08:16:35,094 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 08:16:35,095 INFO repeater task os75nb_beamplots ended after 5.0 seconds 2012-05-15 08:16:35,095 INFO repeater task os75nb_beamplots starting 281.1 seconds late 2012-05-15 08:16:40,003 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 08:16:40,003 INFO repeater task os75nb_beamplots ended after 4.9 seconds 2012-05-15 08:16:40,004 INFO repeater task phins_hcorrstats starting 586.0 seconds late 2012-05-15 08:16:46,112 INFO repeater task phins_hcorrstats ended after 6.1 seconds 2012-05-15 08:16:46,112 INFO repeater task phins_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 08:16:46,113 INFO repeater task phins2_hcorrstats starting 592.1 seconds late 2012-05-15 08:16:51,021 INFO repeater task phins2_hcorrstats ended after 4.9 seconds 2012-05-15 08:16:51,022 INFO repeater task phins2_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 08:16:56,026 INFO repeater task wh300_lastensq starting 491.0 seconds late 2012-05-15 08:17:04,615 INFO repeater task wh300_lastensq ended after 8.6 seconds 2012-05-15 08:17:04,616 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:16:35 2012-05-15 08:17:04,618 INFO repeater task os75bb_lastensq starting 139.6 seconds late 2012-05-15 08:17:14,186 INFO repeater task os75bb_lastensq ended after 9.6 seconds 2012-05-15 08:17:14,187 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:14:35 2012-05-15 08:17:14,189 INFO repeater task os75nb_lastensq starting 149.2 seconds late 2012-05-15 08:17:18,960 INFO repeater task os75nb_lastensq ended after 4.8 seconds 2012-05-15 08:17:18,961 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:14:35 2012-05-15 08:17:18,967 INFO repeater task phins_hcorrstats starting 24.9 seconds late 2012-05-15 08:17:24,017 INFO repeater task phins_hcorrstats ended after 5.1 seconds 2012-05-15 08:17:24,018 INFO repeater task phins2_hcorrstats starting 30.0 seconds late 2012-05-15 08:17:29,005 INFO repeater task phins2_hcorrstats ended after 5.0 seconds 2012-05-15 08:18:49,289 INFO repeater task wh300_lastensq starting 4.3 seconds late 2012-05-15 08:18:54,037 INFO repeater task wh300_lastensq ended after 4.7 seconds 2012-05-15 08:18:54,038 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:18:35 2012-05-15 08:19:49,373 INFO repeater task os75bb_lastensq starting 4.4 seconds late 2012-05-15 08:19:57,945 INFO repeater task os75bb_lastensq ended after 8.6 seconds 2012-05-15 08:19:57,945 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:19:35 2012-05-15 08:19:57,947 INFO repeater task os75nb_lastensq starting 12.9 seconds late 2012-05-15 08:20:02,664 INFO repeater task os75nb_lastensq ended after 4.7 seconds 2012-05-15 08:20:02,664 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:19:35 2012-05-15 08:20:47,773 INFO repeater task wh300_lastensq starting 2.8 seconds late 2012-05-15 08:20:52,869 INFO repeater task wh300_lastensq ended after 5.1 seconds 2012-05-15 08:20:52,870 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:20:35 2012-05-15 08:21:58,041 INFO repeater task wh300_quick starting 4.0 seconds late 2012-05-15 08:24:34,704 INFO repeater task wh300_quick ended after 156.7 seconds 2012-05-15 08:24:34,708 INFO repeater task wh300_3dayplots starting 160.7 seconds late 2012-05-15 08:25:29,399 INFO repeater task wh300_3dayplots ended after 54.7 seconds 2012-05-15 08:25:29,405 INFO repeater task os75bb_lastensq starting 44.4 seconds late 2012-05-15 08:25:38,782 INFO repeater task os75bb_lastensq ended after 9.4 seconds 2012-05-15 08:25:38,783 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:24:35 2012-05-15 08:25:38,783 INFO repeater task os75bb_quick starting 224.8 seconds late 2012-05-15 08:27:30,658 INFO repeater task os75bb_quick ended after 111.9 seconds 2012-05-15 08:27:30,659 INFO repeater task os75bb_3dayplots starting 336.7 seconds late 2012-05-15 08:28:26,840 INFO repeater task os75bb_3dayplots ended after 56.2 seconds 2012-05-15 08:28:26,841 INFO repeater task os75nb_lastensq starting 221.8 seconds late 2012-05-15 08:28:31,998 INFO repeater task os75nb_lastensq ended after 5.2 seconds 2012-05-15 08:28:31,999 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:24:35 2012-05-15 08:28:32,000 INFO repeater task os75nb_quick starting 398.0 seconds late 2012-05-15 08:30:16,557 INFO repeater task os75nb_quick ended after 104.6 seconds 2012-05-15 08:30:16,558 INFO repeater task os75nb_3dayplots starting 502.6 seconds late 2012-05-15 08:31:21,519 INFO repeater task os75nb_3dayplots ended after 65.0 seconds 2012-05-15 08:31:21,525 INFO repeater task wh300_beamplots starting 567.5 seconds late 2012-05-15 08:31:34,185 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:31:34,186 INFO repeater task wh300_beamplots ended after 12.7 seconds 2012-05-15 08:31:34,187 INFO repeater task wh300_beamplots starting 580.2 seconds late 2012-05-15 08:31:45,785 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:31:45,785 INFO repeater task wh300_beamplots ended after 11.6 seconds 2012-05-15 08:31:45,789 INFO repeater task os75bb_beamplots starting 591.8 seconds late 2012-05-15 08:31:53,230 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:31:53,231 INFO repeater task os75bb_beamplots ended after 7.4 seconds 2012-05-15 08:31:53,231 INFO repeater task os75bb_beamplots starting 599.2 seconds late 2012-05-15 08:31:59,301 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:31:59,302 INFO repeater task os75bb_beamplots ended after 6.1 seconds 2012-05-15 08:31:59,302 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 08:31:59,303 INFO repeater task os75nb_beamplots starting 605.3 seconds late 2012-05-15 08:32:05,004 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 08:32:05,004 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2012-05-15 08:32:05,005 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 08:32:05,005 INFO repeater task os75nb_beamplots starting 611.0 seconds late 2012-05-15 08:32:10,666 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 08:32:10,666 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2012-05-15 08:32:10,666 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 08:32:10,667 INFO repeater task phins_hcorrstats starting 616.6 seconds late 2012-05-15 08:32:18,100 INFO repeater task phins_hcorrstats ended after 7.4 seconds 2012-05-15 08:32:18,100 INFO repeater task phins_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 08:32:18,105 INFO repeater task phins2_hcorrstats starting 624.1 seconds late 2012-05-15 08:32:24,168 INFO repeater task phins2_hcorrstats ended after 6.1 seconds 2012-05-15 08:32:24,168 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 08:32:29,173 INFO repeater task wh300_lastensq starting 584.2 seconds late 2012-05-15 08:32:37,595 INFO repeater task wh300_lastensq ended after 8.4 seconds 2012-05-15 08:32:37,596 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:30:35 2012-05-15 08:32:37,611 INFO repeater task os75bb_lastensq starting 172.6 seconds late 2012-05-15 08:32:49,553 INFO repeater task os75bb_lastensq ended after 11.9 seconds 2012-05-15 08:32:49,553 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:29:35 2012-05-15 08:32:49,556 INFO repeater task os75nb_lastensq starting 184.6 seconds late 2012-05-15 08:32:55,265 INFO repeater task os75nb_lastensq ended after 5.7 seconds 2012-05-15 08:32:55,265 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:29:35 2012-05-15 08:32:55,276 INFO repeater task wh300_beamplots starting 61.3 seconds late 2012-05-15 08:33:06,922 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:33:06,923 INFO repeater task wh300_beamplots ended after 11.6 seconds 2012-05-15 08:33:06,924 INFO repeater task wh300_beamplots starting 72.9 seconds late 2012-05-15 08:33:19,037 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:33:19,038 INFO repeater task wh300_beamplots ended after 12.1 seconds 2012-05-15 08:33:19,040 INFO repeater task os75bb_beamplots starting 85.0 seconds late 2012-05-15 08:33:25,364 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:33:25,364 INFO repeater task os75bb_beamplots ended after 6.3 seconds 2012-05-15 08:33:30,371 INFO repeater task wh300_lastensq starting 45.4 seconds late 2012-05-15 08:33:36,525 INFO repeater task wh300_lastensq ended after 6.2 seconds 2012-05-15 08:33:36,526 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:32:35 2012-05-15 08:34:46,971 INFO repeater task wh300_lastensq starting 2.0 seconds late 2012-05-15 08:35:11,464 INFO repeater task wh300_lastensq ended after 24.5 seconds 2012-05-15 08:35:11,464 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:34:35 2012-05-15 08:35:11,473 INFO repeater task os75bb_lastensq starting 26.5 seconds late 2012-05-15 08:35:29,744 INFO repeater task os75bb_lastensq ended after 18.3 seconds 2012-05-15 08:35:29,745 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:34:35 2012-05-15 08:35:29,747 INFO repeater task os75nb_lastensq starting 44.7 seconds late 2012-05-15 08:35:43,630 INFO repeater task os75nb_lastensq ended after 13.9 seconds 2012-05-15 08:35:43,630 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:34:35 2012-05-15 08:36:49,141 INFO repeater task wh300_lastensq starting 4.1 seconds late 2012-05-15 08:36:56,206 INFO repeater task wh300_lastensq ended after 7.1 seconds 2012-05-15 08:36:56,207 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:36:35 2012-05-15 08:36:56,213 INFO repeater task wh300_quick starting 2.2 seconds late 2012-05-15 08:40:15,535 INFO repeater task wh300_quick ended after 199.3 seconds 2012-05-15 08:40:15,541 INFO repeater task wh300_3dayplots starting 201.5 seconds late 2012-05-15 08:41:23,149 INFO repeater task wh300_3dayplots ended after 67.6 seconds 2012-05-15 08:41:23,153 INFO repeater task os75bb_lastensq starting 98.2 seconds late 2012-05-15 08:41:35,310 INFO repeater task os75bb_lastensq ended after 12.2 seconds 2012-05-15 08:41:35,311 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:39:35 2012-05-15 08:41:35,311 INFO repeater task os75bb_quick starting 281.3 seconds late 2012-05-15 08:43:52,614 INFO repeater task os75bb_quick ended after 137.3 seconds 2012-05-15 08:43:52,619 INFO repeater task os75bb_3dayplots starting 418.6 seconds late 2012-05-15 08:45:01,337 INFO repeater task os75bb_3dayplots ended after 68.7 seconds 2012-05-15 08:45:01,338 INFO repeater task os75nb_lastensq starting 316.3 seconds late 2012-05-15 08:45:08,201 INFO repeater task os75nb_lastensq ended after 6.9 seconds 2012-05-15 08:45:08,201 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:44:35 2012-05-15 08:45:08,205 INFO repeater task os75nb_quick starting 494.2 seconds late 2012-05-15 08:47:48,246 INFO repeater task os75nb_quick ended after 160.0 seconds 2012-05-15 08:47:48,249 INFO repeater task os75nb_3dayplots starting 654.2 seconds late 2012-05-15 08:49:44,052 INFO repeater task os75nb_3dayplots ended after 115.8 seconds 2012-05-15 08:49:44,064 INFO repeater task wh300_beamplots starting 470.1 seconds late 2012-05-15 08:49:58,357 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:49:58,357 INFO repeater task wh300_beamplots ended after 14.3 seconds 2012-05-15 08:49:58,359 INFO repeater task wh300_beamplots starting 484.3 seconds late 2012-05-15 08:50:10,559 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 08:50:10,560 INFO repeater task wh300_beamplots ended after 12.2 seconds 2012-05-15 08:50:10,560 INFO repeater task os75bb_beamplots starting 496.5 seconds late 2012-05-15 08:50:16,873 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:50:16,874 INFO repeater task os75bb_beamplots ended after 6.3 seconds 2012-05-15 08:50:16,874 INFO repeater task os75bb_beamplots starting 502.9 seconds late 2012-05-15 08:50:23,558 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 08:50:23,559 INFO repeater task os75bb_beamplots ended after 6.7 seconds 2012-05-15 08:50:23,565 INFO repeater task os75nb_beamplots starting 509.5 seconds late 2012-05-15 08:50:29,584 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 08:50:29,584 INFO repeater task os75nb_beamplots ended after 6.0 seconds 2012-05-15 08:50:29,590 INFO repeater task os75nb_beamplots starting 515.6 seconds late 2012-05-15 08:50:35,338 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 08:50:35,339 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2012-05-15 08:50:35,340 INFO repeater task phins_hcorrstats starting 821.3 seconds late 2012-05-15 08:50:42,161 INFO repeater task phins_hcorrstats ended after 6.8 seconds 2012-05-15 08:50:42,162 INFO repeater task phins_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 08:50:42,165 INFO repeater task phins2_hcorrstats starting 828.1 seconds late 2012-05-15 08:50:48,478 INFO repeater task phins2_hcorrstats ended after 6.3 seconds 2012-05-15 08:50:48,479 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 08:50:53,481 INFO repeater task wh300_lastensq starting 728.5 seconds late 2012-05-15 08:51:04,132 INFO repeater task wh300_lastensq ended after 10.7 seconds 2012-05-15 08:51:04,133 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 08:50:35 2012-05-15 08:51:04,149 INFO repeater task os75bb_lastensq starting 379.1 seconds late 2012-05-15 08:51:16,157 INFO repeater task os75bb_lastensq ended after 12.0 seconds 2012-05-15 08:51:16,169 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:49:35 2012-05-15 08:51:16,170 INFO repeater task os75nb_lastensq starting 91.2 seconds late 2012-05-15 08:51:22,015 INFO repeater task os75nb_lastensq ended after 5.8 seconds 2012-05-15 08:51:22,016 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:49:35 2012-05-15 08:51:57,195 INFO repeater task wh300_quick starting 3.2 seconds late 2012-05-15 08:56:12,187 INFO repeater task wh300_quick ended after 255.0 seconds 2012-05-15 08:56:12,193 INFO repeater task wh300_3dayplots starting 258.2 seconds late 2012-05-15 08:58:11,594 INFO repeater task wh300_3dayplots ended after 119.4 seconds 2012-05-15 08:58:11,598 INFO repeater task os75bb_lastensq starting 206.6 seconds late 2012-05-15 08:58:33,633 INFO repeater task os75bb_lastensq ended after 22.0 seconds 2012-05-15 08:58:33,633 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 08:54:35 2012-05-15 08:58:33,637 INFO repeater task os75bb_quick starting 399.6 seconds late 2012-05-15 09:01:39,481 INFO repeater task os75bb_quick ended after 185.8 seconds 2012-05-15 09:01:39,482 INFO repeater task os75bb_3dayplots starting 585.5 seconds late 2012-05-15 09:03:11,416 INFO repeater task os75bb_3dayplots ended after 91.9 seconds 2012-05-15 09:03:11,417 INFO repeater task os75nb_lastensq starting 506.4 seconds late 2012-05-15 09:03:21,405 INFO repeater task os75nb_lastensq ended after 10.0 seconds 2012-05-15 09:03:21,406 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 08:59:35 2012-05-15 09:03:21,406 INFO repeater task os75nb_quick starting 687.4 seconds late 2012-05-15 09:06:05,000 INFO repeater task os75nb_quick ended after 163.6 seconds 2012-05-15 09:06:05,005 INFO repeater task os75nb_3dayplots starting 851.0 seconds late 2012-05-15 09:07:33,089 INFO repeater task os75nb_3dayplots ended after 88.1 seconds 2012-05-15 09:07:33,089 INFO repeater task os75nb_3dayplots: added 1 extra increments to time_to_run 2012-05-15 09:07:33,097 INFO repeater task wh300_beamplots starting 939.1 seconds late 2012-05-15 09:07:49,161 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:07:49,162 INFO repeater task wh300_beamplots ended after 16.1 seconds 2012-05-15 09:07:49,162 INFO repeater task wh300_beamplots: added 1 extra increments to time_to_run 2012-05-15 09:07:49,163 INFO repeater task wh300_beamplots starting 955.1 seconds late 2012-05-15 09:08:04,651 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:08:04,652 INFO repeater task wh300_beamplots ended after 15.5 seconds 2012-05-15 09:08:04,652 INFO repeater task wh300_beamplots: added 1 extra increments to time_to_run 2012-05-15 09:08:04,653 INFO repeater task os75bb_beamplots starting 970.6 seconds late 2012-05-15 09:08:13,891 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:08:13,891 INFO repeater task os75bb_beamplots ended after 9.2 seconds 2012-05-15 09:08:13,892 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 09:08:13,893 INFO repeater task os75bb_beamplots starting 979.9 seconds late 2012-05-15 09:08:22,058 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:08:22,058 INFO repeater task os75bb_beamplots ended after 8.2 seconds 2012-05-15 09:08:22,059 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 09:08:22,063 INFO repeater task os75nb_beamplots starting 988.0 seconds late 2012-05-15 09:08:29,719 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:08:29,720 INFO repeater task os75nb_beamplots ended after 7.7 seconds 2012-05-15 09:08:29,720 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 09:08:29,722 INFO repeater task os75nb_beamplots starting 995.7 seconds late 2012-05-15 09:08:37,907 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:08:37,908 INFO repeater task os75nb_beamplots ended after 8.2 seconds 2012-05-15 09:08:37,908 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 09:08:37,913 INFO repeater task phins_hcorrstats starting 1003.9 seconds late 2012-05-15 09:08:47,260 INFO repeater task phins_hcorrstats ended after 9.3 seconds 2012-05-15 09:08:47,261 INFO repeater task phins_hcorrstats: added 3 extra increments to time_to_run 2012-05-15 09:08:47,265 INFO repeater task phins2_hcorrstats starting 1013.2 seconds late 2012-05-15 09:08:55,307 INFO repeater task phins2_hcorrstats ended after 8.0 seconds 2012-05-15 09:08:55,308 INFO repeater task phins2_hcorrstats: added 3 extra increments to time_to_run 2012-05-15 09:09:00,316 INFO repeater task wh300_lastensq starting 975.3 seconds late 2012-05-15 09:09:16,481 INFO repeater task wh300_lastensq ended after 16.2 seconds 2012-05-15 09:09:16,482 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 09:08:36 2012-05-15 09:09:16,490 INFO repeater task wh300_quick starting 142.5 seconds late 2012-05-15 09:12:16,931 INFO Translate translate exiting, gps /home/data/at20/rbin/gps_fixed 2012-05-15 09:12:16,933 INFO Translate translate exiting, gps /home/data/at20/rbin/gpsnav 2012-05-15 09:12:16,936 INFO Translate translate exiting, hdg /home/data/at20/rbin/phins 2012-05-15 09:12:16,939 INFO Translate translate exiting, hdg /home/data/at20/rbin/phins2 2012-05-15 09:12:16,958 INFO Translate translate exiting, hdg /home/data/at20/rbin/gyro 2012-05-15 09:12:55,026 INFO repeater task wh300_quick ended after 218.5 seconds 2012-05-15 09:12:56,398 INFO root Starting DAS_while_logging.py 2012-05-15 09:12:57,373 DEBUG procsetup Starting procsetup 2012-05-15 09:12:57,400 INFO root Cruiseid: at20 2012-05-15 09:13:00,038 INFO repeater task wh300_lastensq starting 134.0 seconds late 2012-05-15 09:13:12,687 INFO repeater task wh300_lastensq ended after 12.6 seconds 2012-05-15 09:13:12,689 INFO repeater task wh300_lastensq: Time from /home/data/at20/proc/wh300/load/wrote_ens_time: 2012/05/15 09:10:36 2012-05-15 09:13:12,690 INFO repeater task wh300_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:13:12,696 INFO repeater task wh300_3dayplots starting 378.7 seconds late 2012-05-15 09:14:25,702 INFO repeater task wh300_3dayplots ended after 73.0 seconds 2012-05-15 09:14:25,704 INFO repeater task os75bb_lastensq starting 880.7 seconds late 2012-05-15 09:14:37,705 INFO repeater task os75bb_lastensq ended after 12.0 seconds 2012-05-15 09:14:37,706 INFO repeater task os75bb_lastensq: Time from /home/data/at20/proc/os75bb/load/wrote_ens_time: 2012/05/15 09:09:35 2012-05-15 09:14:37,713 INFO repeater task os75bb_quick starting 463.7 seconds late 2012-05-15 09:17:09,205 INFO repeater task os75bb_quick ended after 151.5 seconds 2012-05-15 09:17:09,208 INFO repeater task os75bb_3dayplots starting 615.2 seconds late 2012-05-15 09:18:26,984 INFO repeater task os75bb_3dayplots ended after 77.8 seconds 2012-05-15 09:18:26,985 INFO repeater task os75nb_lastensq starting 822.0 seconds late 2012-05-15 09:18:37,046 INFO repeater task os75nb_lastensq ended after 10.1 seconds 2012-05-15 09:18:37,047 INFO repeater task os75nb_lastensq: Time from /home/data/at20/proc/os75nb/load/wrote_ens_time: 2012/05/15 09:09:35 2012-05-15 09:18:37,047 INFO repeater task os75nb_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:18:37,048 INFO repeater task os75nb_quick starting 703.0 seconds late 2012-05-15 09:20:43,228 INFO repeater task os75nb_quick ended after 126.2 seconds 2012-05-15 09:20:43,308 INFO repeater task wh300_beamplots starting 529.3 seconds late 2012-05-15 09:20:54,977 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:20:54,978 INFO repeater task wh300_beamplots ended after 11.7 seconds 2012-05-15 09:20:54,978 INFO repeater task wh300_beamplots starting 541.0 seconds late 2012-05-15 09:21:04,363 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:21:04,364 INFO repeater task wh300_beamplots ended after 9.4 seconds 2012-05-15 09:21:04,364 INFO repeater task os75bb_beamplots starting 550.3 seconds late 2012-05-15 09:21:10,057 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:21:10,058 INFO repeater task os75bb_beamplots ended after 5.7 seconds 2012-05-15 09:21:10,059 INFO repeater task os75bb_beamplots starting 556.0 seconds late 2012-05-15 09:21:15,215 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:21:15,215 INFO repeater task os75bb_beamplots ended after 5.2 seconds 2012-05-15 09:21:15,216 INFO repeater task os75nb_beamplots starting 561.2 seconds late 2012-05-15 09:21:20,298 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:21:20,299 INFO repeater task os75nb_beamplots ended after 5.1 seconds 2012-05-15 09:21:20,300 INFO repeater task os75nb_beamplots starting 566.3 seconds late 2012-05-15 09:21:25,402 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:21:25,403 INFO repeater task os75nb_beamplots ended after 5.1 seconds 2012-05-15 09:21:25,404 INFO repeater task phins_hcorrstats starting 571.4 seconds late 2012-05-15 09:21:32,660 INFO repeater task phins_hcorrstats ended after 7.3 seconds 2012-05-15 09:21:32,661 INFO repeater task phins_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 09:21:32,666 INFO repeater task phins2_hcorrstats starting 578.6 seconds late 2012-05-15 09:21:38,496 INFO repeater task phins2_hcorrstats ended after 5.8 seconds 2012-05-15 09:21:38,496 INFO repeater task phins2_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 09:21:43,501 INFO repeater task wh300_lastensq starting 407.5 seconds late 2012-05-15 09:21:50,138 INFO repeater task wh300_lastensq ended after 6.6 seconds 2012-05-15 09:21:50,138 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -514.413 2012-05-15 09:21:50,155 INFO repeater task wh300_lastensq: added 3 extra increments to time_to_run 2012-05-15 09:21:50,157 INFO repeater task os75bb_lastensq starting 425.2 seconds late 2012-05-15 09:21:53,672 INFO repeater task os75bb_lastensq ended after 3.5 seconds 2012-05-15 09:21:53,673 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -440.097 2012-05-15 09:21:53,673 INFO repeater task os75bb_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:21:53,681 INFO repeater task os75nb_lastensq starting 118.7 seconds late 2012-05-15 09:21:57,168 INFO repeater task os75nb_lastensq ended after 3.5 seconds 2012-05-15 09:21:57,168 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -199.788 2012-05-15 09:21:57,170 INFO repeater task os75nb_quick starting 3.2 seconds late 2012-05-15 09:23:59,560 INFO repeater task os75nb_quick ended after 122.4 seconds 2012-05-15 09:23:59,568 INFO repeater task os75nb_3dayplots starting 125.6 seconds late 2012-05-15 09:25:12,473 INFO repeater task os75nb_3dayplots ended after 72.9 seconds 2012-05-15 09:25:12,477 INFO repeater task wh300_beamplots starting 198.5 seconds late 2012-05-15 09:25:22,040 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:25:22,041 INFO repeater task wh300_beamplots ended after 9.6 seconds 2012-05-15 09:25:22,045 INFO repeater task wh300_beamplots starting 208.0 seconds late 2012-05-15 09:25:31,876 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:25:31,877 INFO repeater task wh300_beamplots ended after 9.8 seconds 2012-05-15 09:25:31,880 INFO repeater task os75bb_beamplots starting 217.9 seconds late 2012-05-15 09:25:38,125 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:25:38,125 INFO repeater task os75bb_beamplots ended after 6.2 seconds 2012-05-15 09:25:38,129 INFO repeater task os75bb_beamplots starting 224.1 seconds late 2012-05-15 09:25:44,548 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:25:44,549 INFO repeater task os75bb_beamplots ended after 6.4 seconds 2012-05-15 09:25:44,552 INFO repeater task os75nb_beamplots starting 230.5 seconds late 2012-05-15 09:25:50,246 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:25:50,246 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2012-05-15 09:25:50,248 INFO repeater task os75nb_beamplots starting 236.2 seconds late 2012-05-15 09:25:55,719 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:25:55,720 INFO repeater task os75nb_beamplots ended after 5.5 seconds 2012-05-15 09:25:55,721 INFO repeater task phins_hcorrstats starting 241.7 seconds late 2012-05-15 09:26:01,844 INFO repeater task phins_hcorrstats ended after 6.1 seconds 2012-05-15 09:26:01,845 INFO repeater task phins2_hcorrstats starting 247.8 seconds late 2012-05-15 09:26:07,709 INFO repeater task phins2_hcorrstats ended after 5.9 seconds 2012-05-15 09:26:12,713 INFO repeater task wh300_lastensq starting 156.7 seconds late 2012-05-15 09:26:18,498 INFO repeater task wh300_lastensq ended after 5.8 seconds 2012-05-15 09:26:18,499 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -783.625 2012-05-15 09:26:18,500 INFO repeater task wh300_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:26:18,502 INFO repeater task wh300_quick starting 264.5 seconds late 2012-05-15 09:29:54,948 INFO repeater task wh300_quick ended after 216.4 seconds 2012-05-15 09:29:54,953 INFO repeater task wh300_3dayplots starting 480.9 seconds late 2012-05-15 09:31:11,924 INFO repeater task wh300_3dayplots ended after 77.0 seconds 2012-05-15 09:31:11,929 INFO repeater task os75bb_lastensq starting 366.9 seconds late 2012-05-15 09:31:18,790 INFO repeater task os75bb_lastensq ended after 6.9 seconds 2012-05-15 09:31:18,790 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -1001.868 2012-05-15 09:31:18,791 INFO repeater task os75bb_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:31:18,793 INFO repeater task os75bb_quick starting 564.8 seconds late 2012-05-15 09:33:43,271 INFO repeater task os75bb_quick ended after 144.5 seconds 2012-05-15 09:33:43,278 INFO repeater task os75bb_3dayplots starting 709.3 seconds late 2012-05-15 09:35:59,646 INFO repeater task os75bb_3dayplots ended after 136.4 seconds 2012-05-15 09:35:59,647 INFO repeater task os75nb_lastensq starting 654.6 seconds late 2012-05-15 09:36:18,878 INFO repeater task os75nb_lastensq ended after 19.2 seconds 2012-05-15 09:36:18,879 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -1045.755 2012-05-15 09:36:18,879 INFO repeater task os75nb_lastensq: added 2 extra increments to time_to_run 2012-05-15 09:36:18,898 INFO repeater task wh300_beamplots starting 264.9 seconds late 2012-05-15 09:36:38,257 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:36:38,257 INFO repeater task wh300_beamplots ended after 19.4 seconds 2012-05-15 09:36:38,261 INFO repeater task wh300_beamplots starting 284.2 seconds late 2012-05-15 09:36:53,127 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:36:53,127 INFO repeater task wh300_beamplots ended after 14.9 seconds 2012-05-15 09:36:53,129 INFO repeater task os75bb_beamplots starting 299.1 seconds late 2012-05-15 09:37:01,259 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:37:01,259 INFO repeater task os75bb_beamplots ended after 8.1 seconds 2012-05-15 09:37:01,260 INFO repeater task os75bb_beamplots starting 307.2 seconds late 2012-05-15 09:37:10,315 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:37:10,315 INFO repeater task os75bb_beamplots ended after 9.1 seconds 2012-05-15 09:37:10,317 INFO repeater task os75nb_beamplots starting 316.3 seconds late 2012-05-15 09:37:16,332 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:37:16,332 INFO repeater task os75nb_beamplots ended after 6.0 seconds 2012-05-15 09:37:16,333 INFO repeater task os75nb_beamplots starting 322.3 seconds late 2012-05-15 09:37:23,649 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:37:23,650 INFO repeater task os75nb_beamplots ended after 7.3 seconds 2012-05-15 09:37:23,651 INFO repeater task phins_hcorrstats starting 629.6 seconds late 2012-05-15 09:37:34,333 INFO repeater task phins_hcorrstats ended after 10.7 seconds 2012-05-15 09:37:34,333 INFO repeater task phins_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 09:37:34,334 INFO repeater task phins2_hcorrstats starting 640.3 seconds late 2012-05-15 09:37:43,915 INFO repeater task phins2_hcorrstats ended after 9.6 seconds 2012-05-15 09:37:43,916 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 09:37:48,926 INFO repeater task wh300_lastensq starting 592.9 seconds late 2012-05-15 09:37:58,793 INFO repeater task wh300_lastensq ended after 9.9 seconds 2012-05-15 09:37:58,794 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -1479.838 2012-05-15 09:37:58,794 INFO repeater task wh300_lastensq: added 4 extra increments to time_to_run 2012-05-15 09:37:58,795 INFO repeater task wh300_quick starting 64.8 seconds late 2012-05-15 09:41:41,374 INFO repeater task wh300_quick ended after 222.6 seconds 2012-05-15 09:41:41,375 INFO repeater task wh300_3dayplots starting 287.4 seconds late 2012-05-15 09:43:05,640 INFO repeater task wh300_3dayplots ended after 84.3 seconds 2012-05-15 09:43:05,642 INFO repeater task os75bb_lastensq starting 460.6 seconds late 2012-05-15 09:43:12,352 INFO repeater task os75bb_lastensq ended after 6.7 seconds 2012-05-15 09:43:12,353 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -1715.581 2012-05-15 09:43:12,356 INFO repeater task os75bb_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:43:12,358 INFO repeater task os75bb_quick starting 378.4 seconds late 2012-05-15 09:45:47,051 INFO repeater task os75bb_quick ended after 154.7 seconds 2012-05-15 09:45:47,057 INFO repeater task os75bb_3dayplots starting 533.0 seconds late 2012-05-15 09:47:06,189 INFO repeater task os75bb_3dayplots ended after 79.1 seconds 2012-05-15 09:47:06,200 INFO repeater task os75nb_lastensq starting 391.2 seconds late 2012-05-15 09:47:12,929 INFO repeater task os75nb_lastensq ended after 6.7 seconds 2012-05-15 09:47:12,929 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -1712.307 2012-05-15 09:47:12,930 INFO repeater task os75nb_lastensq: added 1 extra increments to time_to_run 2012-05-15 09:47:12,931 INFO repeater task os75nb_quick starting 618.9 seconds late 2012-05-15 09:49:32,941 INFO repeater task os75nb_quick ended after 140.0 seconds 2012-05-15 09:49:32,942 INFO repeater task os75nb_3dayplots starting 758.9 seconds late 2012-05-15 09:50:48,127 INFO repeater task os75nb_3dayplots ended after 75.2 seconds 2012-05-15 09:50:48,128 INFO repeater task wh300_beamplots starting 534.1 seconds late 2012-05-15 09:51:00,004 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:51:00,004 INFO repeater task wh300_beamplots ended after 11.9 seconds 2012-05-15 09:51:00,009 INFO repeater task wh300_beamplots starting 546.0 seconds late 2012-05-15 09:51:11,629 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 09:51:11,629 INFO repeater task wh300_beamplots ended after 11.6 seconds 2012-05-15 09:51:11,633 INFO repeater task os75bb_beamplots starting 557.6 seconds late 2012-05-15 09:51:18,715 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:51:18,716 INFO repeater task os75bb_beamplots ended after 7.1 seconds 2012-05-15 09:51:18,718 INFO repeater task os75bb_beamplots starting 564.7 seconds late 2012-05-15 09:51:25,452 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 09:51:25,452 INFO repeater task os75bb_beamplots ended after 6.7 seconds 2012-05-15 09:51:25,456 INFO repeater task os75nb_beamplots starting 571.4 seconds late 2012-05-15 09:51:32,619 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:51:32,619 INFO repeater task os75nb_beamplots ended after 7.2 seconds 2012-05-15 09:51:32,620 INFO repeater task os75nb_beamplots starting 578.6 seconds late 2012-05-15 09:51:38,818 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 09:51:38,818 INFO repeater task os75nb_beamplots ended after 6.2 seconds 2012-05-15 09:51:38,819 INFO repeater task phins_hcorrstats starting 584.8 seconds late 2012-05-15 09:51:46,920 INFO repeater task phins_hcorrstats ended after 8.1 seconds 2012-05-15 09:51:46,921 INFO repeater task phins_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 09:51:46,924 INFO repeater task phins2_hcorrstats starting 592.9 seconds late 2012-05-15 09:51:53,618 INFO repeater task phins2_hcorrstats ended after 6.7 seconds 2012-05-15 09:51:53,619 INFO repeater task phins2_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 09:51:58,621 INFO repeater task wh300_lastensq starting 792.6 seconds late 2012-05-15 09:52:05,759 INFO repeater task wh300_lastensq ended after 7.1 seconds 2012-05-15 09:52:05,760 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -2329.533 2012-05-15 09:52:05,760 INFO repeater task wh300_lastensq: added 6 extra increments to time_to_run 2012-05-15 09:52:05,763 INFO repeater task wh300_quick starting 11.8 seconds late 2012-05-15 09:55:47,349 INFO repeater task wh300_quick ended after 221.6 seconds 2012-05-15 09:55:47,353 INFO repeater task wh300_3dayplots starting 233.3 seconds late 2012-05-15 09:57:04,316 INFO repeater task wh300_3dayplots ended after 77.0 seconds 2012-05-15 09:57:04,321 INFO repeater task os75bb_lastensq starting 679.3 seconds late 2012-05-15 09:57:11,423 INFO repeater task os75bb_lastensq ended after 7.1 seconds 2012-05-15 09:57:11,424 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -2554.260 2012-05-15 09:57:11,424 INFO repeater task os75bb_lastensq: added 2 extra increments to time_to_run 2012-05-15 09:57:11,425 INFO repeater task os75bb_quick starting 317.4 seconds late 2012-05-15 09:59:38,910 INFO repeater task os75bb_quick ended after 147.5 seconds 2012-05-15 09:59:38,911 INFO repeater task os75bb_3dayplots starting 464.9 seconds late 2012-05-15 10:00:56,417 INFO repeater task os75bb_3dayplots ended after 77.5 seconds 2012-05-15 10:00:56,419 INFO repeater task os75nb_lastensq starting 601.4 seconds late 2012-05-15 10:01:03,267 INFO repeater task os75nb_lastensq ended after 6.8 seconds 2012-05-15 10:01:03,267 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -2542.526 2012-05-15 10:01:03,268 INFO repeater task os75nb_lastensq: added 1 extra increments to time_to_run 2012-05-15 10:01:03,273 INFO repeater task os75nb_quick starting 549.3 seconds late 2012-05-15 10:03:20,542 INFO repeater task os75nb_quick ended after 137.3 seconds 2012-05-15 10:03:20,543 INFO repeater task os75nb_3dayplots starting 686.5 seconds late 2012-05-15 10:04:36,034 INFO repeater task os75nb_3dayplots ended after 75.5 seconds 2012-05-15 10:04:36,040 INFO repeater task wh300_beamplots starting 762.0 seconds late 2012-05-15 10:04:47,913 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:04:47,914 INFO repeater task wh300_beamplots ended after 11.9 seconds 2012-05-15 10:04:47,914 INFO repeater task wh300_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:04:47,915 INFO repeater task wh300_beamplots starting 773.9 seconds late 2012-05-15 10:04:59,033 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:04:59,034 INFO repeater task wh300_beamplots ended after 11.1 seconds 2012-05-15 10:04:59,034 INFO repeater task wh300_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:04:59,035 INFO repeater task os75bb_beamplots starting 785.0 seconds late 2012-05-15 10:05:06,618 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:05:06,618 INFO repeater task os75bb_beamplots ended after 7.6 seconds 2012-05-15 10:05:06,619 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:05:06,624 INFO repeater task os75bb_beamplots starting 792.6 seconds late 2012-05-15 10:05:13,046 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:05:13,047 INFO repeater task os75bb_beamplots ended after 6.4 seconds 2012-05-15 10:05:13,047 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:05:13,048 INFO repeater task os75nb_beamplots starting 799.0 seconds late 2012-05-15 10:05:19,360 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:05:19,361 INFO repeater task os75nb_beamplots ended after 6.3 seconds 2012-05-15 10:05:19,361 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:05:19,362 INFO repeater task os75nb_beamplots starting 805.3 seconds late 2012-05-15 10:05:26,043 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:05:26,043 INFO repeater task os75nb_beamplots ended after 6.7 seconds 2012-05-15 10:05:26,044 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:05:26,048 INFO repeater task phins_hcorrstats starting 812.0 seconds late 2012-05-15 10:05:34,027 INFO repeater task phins_hcorrstats ended after 8.0 seconds 2012-05-15 10:05:34,027 INFO repeater task phins_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:05:34,028 INFO repeater task phins2_hcorrstats starting 820.0 seconds late 2012-05-15 10:05:40,270 INFO repeater task phins2_hcorrstats ended after 6.2 seconds 2012-05-15 10:05:40,271 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:05:45,277 INFO repeater task wh300_lastensq starting 709.3 seconds late 2012-05-15 10:05:53,542 INFO repeater task wh300_lastensq ended after 8.3 seconds 2012-05-15 10:05:53,543 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -3156.188 2012-05-15 10:05:53,543 INFO repeater task wh300_lastensq: added 5 extra increments to time_to_run 2012-05-15 10:05:53,553 INFO repeater task os75bb_lastensq starting 278.6 seconds late 2012-05-15 10:05:57,995 INFO repeater task os75bb_lastensq ended after 4.4 seconds 2012-05-15 10:05:57,996 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -3083.493 2012-05-15 10:05:57,998 INFO repeater task os75nb_lastensq starting 283.0 seconds late 2012-05-15 10:06:02,009 INFO repeater task os75nb_lastensq ended after 4.0 seconds 2012-05-15 10:06:02,010 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -2844.106 2012-05-15 10:06:27,147 INFO repeater task os75bb_lastensq starting 2.1 seconds late 2012-05-15 10:06:30,756 INFO repeater task os75bb_lastensq ended after 3.6 seconds 2012-05-15 10:06:30,757 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -3117.086 2012-05-15 10:06:30,759 INFO repeater task os75nb_lastensq starting 5.8 seconds late 2012-05-15 10:06:35,096 INFO repeater task os75nb_lastensq ended after 4.3 seconds 2012-05-15 10:06:35,096 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -2876.866 2012-05-15 10:06:55,238 INFO repeater task wh300_quick starting 1.2 seconds late 2012-05-15 10:10:30,716 INFO repeater task wh300_quick ended after 215.5 seconds 2012-05-15 10:10:30,717 INFO repeater task wh300_3dayplots starting 216.7 seconds late 2012-05-15 10:11:45,637 INFO repeater task wh300_3dayplots ended after 74.9 seconds 2012-05-15 10:11:45,644 INFO repeater task os75bb_lastensq starting 10.6 seconds late 2012-05-15 10:11:52,114 INFO repeater task os75bb_lastensq ended after 6.5 seconds 2012-05-15 10:11:52,114 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -3435.583 2012-05-15 10:11:52,115 INFO repeater task os75bb_quick starting 298.1 seconds late 2012-05-15 10:14:14,821 INFO repeater task os75bb_quick ended after 142.7 seconds 2012-05-15 10:14:14,822 INFO repeater task os75bb_3dayplots starting 440.8 seconds late 2012-05-15 10:15:30,778 INFO repeater task os75bb_3dayplots ended after 76.0 seconds 2012-05-15 10:15:30,779 INFO repeater task os75nb_lastensq starting 235.8 seconds late 2012-05-15 10:15:37,081 INFO repeater task os75nb_lastensq ended after 6.3 seconds 2012-05-15 10:15:37,081 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -3416.887 2012-05-15 10:15:37,085 INFO repeater task os75nb_quick starting 523.1 seconds late 2012-05-15 10:17:38,484 INFO repeater task os75nb_quick ended after 121.4 seconds 2012-05-15 10:17:38,485 INFO repeater task os75nb_3dayplots starting 644.5 seconds late 2012-05-15 10:18:46,891 INFO repeater task os75nb_3dayplots ended after 68.4 seconds 2012-05-15 10:18:46,892 INFO repeater task wh300_beamplots starting 412.9 seconds late 2012-05-15 10:18:57,592 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:18:57,593 INFO repeater task wh300_beamplots ended after 10.7 seconds 2012-05-15 10:18:57,597 INFO repeater task wh300_beamplots starting 423.6 seconds late 2012-05-15 10:19:06,320 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:19:06,321 INFO repeater task wh300_beamplots ended after 8.7 seconds 2012-05-15 10:19:06,322 INFO repeater task os75bb_beamplots starting 432.3 seconds late 2012-05-15 10:19:11,878 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:19:11,878 INFO repeater task os75bb_beamplots ended after 5.6 seconds 2012-05-15 10:19:11,879 INFO repeater task os75bb_beamplots starting 437.9 seconds late 2012-05-15 10:19:16,994 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:19:16,995 INFO repeater task os75bb_beamplots ended after 5.1 seconds 2012-05-15 10:19:16,996 INFO repeater task os75nb_beamplots starting 443.0 seconds late 2012-05-15 10:19:21,960 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:19:21,960 INFO repeater task os75nb_beamplots ended after 5.0 seconds 2012-05-15 10:19:21,961 INFO repeater task os75nb_beamplots starting 447.9 seconds late 2012-05-15 10:19:26,897 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:19:26,898 INFO repeater task os75nb_beamplots ended after 4.9 seconds 2012-05-15 10:19:26,898 INFO repeater task phins_hcorrstats starting 752.9 seconds late 2012-05-15 10:19:34,297 INFO repeater task phins_hcorrstats ended after 7.4 seconds 2012-05-15 10:19:34,298 INFO repeater task phins_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:19:34,298 INFO repeater task phins2_hcorrstats starting 760.3 seconds late 2012-05-15 10:19:39,731 INFO repeater task phins2_hcorrstats ended after 5.4 seconds 2012-05-15 10:19:39,732 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:19:44,737 INFO repeater task wh300_lastensq starting 768.7 seconds late 2012-05-15 10:19:50,728 INFO repeater task wh300_lastensq ended after 6.0 seconds 2012-05-15 10:19:50,728 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -3995.649 2012-05-15 10:19:50,729 INFO repeater task wh300_lastensq: added 5 extra increments to time_to_run 2012-05-15 10:19:50,731 INFO repeater task os75bb_lastensq starting 185.7 seconds late 2012-05-15 10:19:54,309 INFO repeater task os75bb_lastensq ended after 3.6 seconds 2012-05-15 10:19:54,310 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -3920.670 2012-05-15 10:19:54,317 INFO repeater task os75nb_lastensq starting 189.3 seconds late 2012-05-15 10:19:57,629 INFO repeater task os75nb_lastensq ended after 3.3 seconds 2012-05-15 10:19:57,630 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -3680.425 2012-05-15 10:20:02,640 INFO repeater task wh300_lastensq starting 6.6 seconds late 2012-05-15 10:20:06,221 INFO repeater task wh300_lastensq ended after 3.6 seconds 2012-05-15 10:20:06,222 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -4013.552 2012-05-15 10:21:56,501 INFO repeater task wh300_quick starting 2.5 seconds late 2012-05-15 10:25:12,134 INFO repeater task wh300_quick ended after 195.6 seconds 2012-05-15 10:25:12,135 INFO repeater task wh300_3dayplots starting 198.1 seconds late 2012-05-15 10:26:10,333 INFO repeater task wh300_3dayplots ended after 58.2 seconds 2012-05-15 10:26:10,334 INFO repeater task os75bb_lastensq starting 255.3 seconds late 2012-05-15 10:26:13,828 INFO repeater task os75bb_lastensq ended after 3.5 seconds 2012-05-15 10:26:13,828 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -4300.273 2012-05-15 10:26:13,829 INFO repeater task os75bb_quick starting 259.8 seconds late 2012-05-15 10:28:03,251 INFO repeater task os75bb_quick ended after 109.4 seconds 2012-05-15 10:28:03,252 INFO repeater task os75bb_3dayplots starting 369.2 seconds late 2012-05-15 10:28:59,107 INFO repeater task os75bb_3dayplots ended after 55.9 seconds 2012-05-15 10:28:59,108 INFO repeater task os75nb_lastensq starting 424.1 seconds late 2012-05-15 10:29:02,925 INFO repeater task os75nb_lastensq ended after 3.8 seconds 2012-05-15 10:29:02,926 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -4225.215 2012-05-15 10:29:02,926 INFO repeater task os75nb_lastensq: added 1 extra increments to time_to_run 2012-05-15 10:29:02,927 INFO repeater task os75nb_quick starting 428.9 seconds late 2012-05-15 10:30:42,253 INFO repeater task os75nb_quick ended after 99.3 seconds 2012-05-15 10:30:42,257 INFO repeater task os75nb_3dayplots starting 528.2 seconds late 2012-05-15 10:31:36,025 INFO repeater task os75nb_3dayplots ended after 53.8 seconds 2012-05-15 10:31:36,026 INFO repeater task wh300_beamplots starting 582.0 seconds late 2012-05-15 10:31:45,150 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:31:45,150 INFO repeater task wh300_beamplots ended after 9.1 seconds 2012-05-15 10:31:45,151 INFO repeater task wh300_beamplots starting 591.1 seconds late 2012-05-15 10:31:52,438 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:31:52,438 INFO repeater task wh300_beamplots ended after 7.3 seconds 2012-05-15 10:31:52,439 INFO repeater task os75bb_beamplots starting 598.4 seconds late 2012-05-15 10:31:57,320 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:31:57,320 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2012-05-15 10:31:57,320 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:31:57,321 INFO repeater task os75bb_beamplots starting 603.3 seconds late 2012-05-15 10:32:02,074 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:32:02,074 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2012-05-15 10:32:02,074 INFO repeater task os75bb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:32:02,075 INFO repeater task os75nb_beamplots starting 608.1 seconds late 2012-05-15 10:32:06,756 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:32:06,756 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2012-05-15 10:32:06,756 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:32:06,757 INFO repeater task os75nb_beamplots starting 612.7 seconds late 2012-05-15 10:32:11,381 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:32:11,382 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2012-05-15 10:32:11,382 INFO repeater task os75nb_beamplots: added 1 extra increments to time_to_run 2012-05-15 10:32:11,389 INFO repeater task phins_hcorrstats starting 617.4 seconds late 2012-05-15 10:32:17,073 INFO repeater task phins_hcorrstats ended after 5.7 seconds 2012-05-15 10:32:17,073 INFO repeater task phins_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:32:17,074 INFO repeater task phins2_hcorrstats starting 623.1 seconds late 2012-05-15 10:32:22,102 INFO repeater task phins2_hcorrstats ended after 5.0 seconds 2012-05-15 10:32:22,103 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:32:27,105 INFO repeater task wh300_lastensq starting 621.1 seconds late 2012-05-15 10:32:31,069 INFO repeater task wh300_lastensq ended after 4.0 seconds 2012-05-15 10:32:31,070 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -4758.017 2012-05-15 10:32:31,071 INFO repeater task wh300_lastensq: added 4 extra increments to time_to_run 2012-05-15 10:32:31,073 INFO repeater task os75bb_lastensq starting 326.1 seconds late 2012-05-15 10:32:33,520 INFO repeater task os75bb_lastensq ended after 2.4 seconds 2012-05-15 10:32:33,520 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -4681.012 2012-05-15 10:32:33,521 INFO repeater task os75bb_lastensq: added 1 extra increments to time_to_run 2012-05-15 10:32:33,523 INFO repeater task os75nb_lastensq starting 18.5 seconds late 2012-05-15 10:32:36,893 INFO repeater task os75nb_lastensq ended after 3.4 seconds 2012-05-15 10:32:36,894 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -4439.630 2012-05-15 10:32:36,896 INFO repeater task wh300_beamplots starting 42.9 seconds late 2012-05-15 10:32:44,317 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:32:44,318 INFO repeater task wh300_beamplots ended after 7.4 seconds 2012-05-15 10:32:44,321 INFO repeater task wh300_beamplots starting 50.3 seconds late 2012-05-15 10:32:52,637 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:32:52,637 INFO repeater task wh300_beamplots ended after 8.3 seconds 2012-05-15 10:32:57,645 INFO repeater task wh300_lastensq starting 1.6 seconds late 2012-05-15 10:33:00,609 INFO repeater task wh300_lastensq ended after 3.0 seconds 2012-05-15 10:33:00,609 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -4788.557 2012-05-15 10:35:06,521 INFO repeater task wh300_lastensq starting 0.5 seconds late 2012-05-15 10:35:26,962 INFO repeater task wh300_lastensq ended after 20.4 seconds 2012-05-15 10:35:26,962 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -4917.433 2012-05-15 10:36:57,613 INFO repeater task wh300_quick starting 3.6 seconds late 2012-05-15 10:39:43,912 INFO repeater task wh300_quick ended after 166.3 seconds 2012-05-15 10:39:43,913 INFO repeater task wh300_3dayplots starting 169.9 seconds late 2012-05-15 10:40:39,041 INFO repeater task wh300_3dayplots ended after 55.1 seconds 2012-05-15 10:40:39,047 INFO repeater task os75bb_lastensq starting 194.0 seconds late 2012-05-15 10:40:42,345 INFO repeater task os75bb_lastensq ended after 3.3 seconds 2012-05-15 10:40:42,346 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -5168.987 2012-05-15 10:40:42,349 INFO repeater task os75bb_quick starting 228.3 seconds late 2012-05-15 10:42:37,481 INFO repeater task os75bb_quick ended after 115.1 seconds 2012-05-15 10:42:37,482 INFO repeater task os75bb_3dayplots starting 343.5 seconds late 2012-05-15 10:43:33,472 INFO repeater task os75bb_3dayplots ended after 56.0 seconds 2012-05-15 10:43:33,473 INFO repeater task os75nb_lastensq starting 368.5 seconds late 2012-05-15 10:43:36,653 INFO repeater task os75nb_lastensq ended after 3.2 seconds 2012-05-15 10:43:36,654 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -5099.580 2012-05-15 10:43:36,654 INFO repeater task os75nb_lastensq: added 1 extra increments to time_to_run 2012-05-15 10:43:36,657 INFO repeater task os75nb_quick starting 402.6 seconds late 2012-05-15 10:45:11,918 INFO repeater task os75nb_quick ended after 95.3 seconds 2012-05-15 10:45:11,923 INFO repeater task os75nb_3dayplots starting 497.9 seconds late 2012-05-15 10:45:46,038 ERROR root Error in Procs start Traceback (most recent call last): File "/home/currents/programs/uhdas/scripts/DAS_while_logging.py", line 226, in Procs(flagdir = CI.pd.flagD).start() File "/home/currents/programs/pycurrents/system/pmw_process.py", line 142, in __init__ self.check() File "/home/currents/programs/pycurrents/system/pmw_process.py", line 51, in check root.mainloop() File "/usr/lib/python2.6/lib-tk/Tkinter.py", line 1020, in mainloop self.tk.mainloop(n) File "/usr/lib/python2.6/dist-packages/Pmw/Pmw_1_3/lib/PmwBase.py", line 1749, in __call__ raise SystemExit, msg SystemExit 2012-05-15 10:45:46,106 INFO root Exiting DAS_while_logging.py 2012-05-15 10:46:09,886 INFO repeater task os75nb_3dayplots ended after 58.0 seconds 2012-05-15 10:46:09,887 INFO repeater task wh300_beamplots starting 255.9 seconds late 2012-05-15 10:46:17,988 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:46:17,988 INFO repeater task wh300_beamplots ended after 8.1 seconds 2012-05-15 10:46:17,989 INFO repeater task wh300_beamplots starting 264.0 seconds late 2012-05-15 10:46:26,739 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 10:46:26,739 INFO repeater task wh300_beamplots ended after 8.8 seconds 2012-05-15 10:46:26,741 INFO repeater task os75bb_beamplots starting 272.7 seconds late 2012-05-15 10:46:31,768 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:46:31,769 INFO repeater task os75bb_beamplots ended after 5.0 seconds 2012-05-15 10:46:31,773 INFO repeater task os75bb_beamplots starting 277.8 seconds late 2012-05-15 10:46:36,539 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 10:46:36,539 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2012-05-15 10:46:36,540 INFO repeater task os75nb_beamplots starting 282.5 seconds late 2012-05-15 10:46:41,284 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:46:41,285 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2012-05-15 10:46:41,286 INFO repeater task os75nb_beamplots starting 287.3 seconds late 2012-05-15 10:46:45,821 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 10:46:45,821 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2012-05-15 10:46:45,822 INFO repeater task phins_hcorrstats starting 591.8 seconds late 2012-05-15 10:46:51,365 INFO repeater task phins_hcorrstats ended after 5.5 seconds 2012-05-15 10:46:51,365 INFO repeater task phins_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 10:46:51,366 INFO repeater task phins2_hcorrstats starting 597.3 seconds late 2012-05-15 10:46:56,279 INFO repeater task phins2_hcorrstats ended after 4.9 seconds 2012-05-15 10:46:56,279 INFO repeater task phins2_hcorrstats: added 2 extra increments to time_to_run 2012-05-15 10:47:01,289 INFO repeater task wh300_lastensq starting 585.3 seconds late 2012-05-15 10:47:05,204 INFO repeater task wh300_lastensq ended after 3.9 seconds 2012-05-15 10:47:05,204 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -5632.200 2012-05-15 10:47:05,205 INFO repeater task wh300_lastensq: added 4 extra increments to time_to_run 2012-05-15 10:47:05,213 INFO repeater task os75bb_lastensq starting 270.2 seconds late 2012-05-15 10:47:08,332 INFO repeater task os75bb_lastensq ended after 3.1 seconds 2012-05-15 10:47:08,332 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -5555.152 2012-05-15 10:47:08,339 INFO repeater task phins_hcorrstats starting 14.3 seconds late 2012-05-15 10:47:13,344 INFO repeater task phins_hcorrstats ended after 5.0 seconds 2012-05-15 10:47:48,434 INFO repeater task os75bb_lastensq starting 3.4 seconds late 2012-05-15 10:47:51,431 INFO repeater task os75bb_lastensq ended after 3.0 seconds 2012-05-15 10:47:51,432 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -5598.374 2012-05-15 10:47:51,439 INFO repeater task os75nb_lastensq starting 6.4 seconds late 2012-05-15 10:47:53,861 INFO repeater task os75nb_lastensq ended after 2.4 seconds 2012-05-15 10:47:53,861 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -5357.547 2012-05-15 10:48:08,889 INFO repeater task wh300_lastensq starting 2.9 seconds late 2012-05-15 10:48:12,333 INFO repeater task wh300_lastensq ended after 3.4 seconds 2012-05-15 10:48:12,334 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -5699.801 2012-05-15 10:50:17,849 INFO repeater task wh300_lastensq starting 1.8 seconds late 2012-05-15 10:50:20,338 INFO repeater task wh300_lastensq ended after 2.5 seconds 2012-05-15 10:50:20,339 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -5828.761 2012-05-15 10:51:55,649 INFO repeater task wh300_quick starting 1.6 seconds late 2012-05-15 10:54:42,240 INFO repeater task wh300_quick ended after 166.6 seconds 2012-05-15 10:54:42,245 INFO repeater task wh300_3dayplots starting 168.2 seconds late 2012-05-15 10:55:37,601 INFO repeater task wh300_3dayplots ended after 55.4 seconds 2012-05-15 10:55:37,602 INFO repeater task os75bb_lastensq starting 162.6 seconds late 2012-05-15 10:55:43,269 INFO repeater task os75bb_lastensq ended after 5.7 seconds 2012-05-15 10:55:43,270 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -6067.542 2012-05-15 10:55:43,271 INFO repeater task os75bb_quick starting 229.3 seconds late 2012-05-15 10:57:37,707 INFO repeater task os75bb_quick ended after 114.4 seconds 2012-05-15 10:57:37,709 INFO repeater task os75bb_3dayplots starting 343.7 seconds late 2012-05-15 10:58:34,050 INFO repeater task os75bb_3dayplots ended after 56.3 seconds 2012-05-15 10:58:34,052 INFO repeater task os75nb_lastensq starting 339.1 seconds late 2012-05-15 10:58:37,490 INFO repeater task os75nb_lastensq ended after 3.4 seconds 2012-05-15 10:58:37,491 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -6000.160 2012-05-15 10:58:37,491 INFO repeater task os75nb_lastensq: added 1 extra increments to time_to_run 2012-05-15 10:58:37,492 INFO repeater task os75nb_quick starting 403.5 seconds late 2012-05-15 11:00:13,428 INFO repeater task os75nb_quick ended after 95.9 seconds 2012-05-15 11:00:13,433 INFO repeater task os75nb_3dayplots starting 499.4 seconds late 2012-05-15 11:01:06,977 INFO repeater task os75nb_3dayplots ended after 53.5 seconds 2012-05-15 11:01:06,978 INFO repeater task wh300_beamplots starting 553.0 seconds late 2012-05-15 11:01:15,972 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 11:01:15,972 INFO repeater task wh300_beamplots ended after 9.0 seconds 2012-05-15 11:01:15,983 INFO repeater task wh300_beamplots starting 562.0 seconds late 2012-05-15 11:01:23,261 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 11:01:23,262 INFO repeater task wh300_beamplots ended after 7.3 seconds 2012-05-15 11:01:23,263 INFO repeater task os75bb_beamplots starting 569.2 seconds late 2012-05-15 11:01:28,215 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 11:01:28,216 INFO repeater task os75bb_beamplots ended after 5.0 seconds 2012-05-15 11:01:28,216 INFO repeater task os75bb_beamplots starting 574.2 seconds late 2012-05-15 11:01:32,971 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 11:01:32,971 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2012-05-15 11:01:32,972 INFO repeater task os75nb_beamplots starting 579.0 seconds late 2012-05-15 11:01:37,677 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 11:01:37,678 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2012-05-15 11:01:37,679 INFO repeater task os75nb_beamplots starting 583.7 seconds late 2012-05-15 11:01:42,349 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 11:01:42,349 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2012-05-15 11:01:42,350 INFO repeater task phins_hcorrstats starting 588.3 seconds late 2012-05-15 11:01:48,016 INFO repeater task phins_hcorrstats ended after 5.7 seconds 2012-05-15 11:01:48,016 INFO repeater task phins_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 11:01:48,017 INFO repeater task phins2_hcorrstats starting 594.0 seconds late 2012-05-15 11:01:52,924 INFO repeater task phins2_hcorrstats ended after 4.9 seconds 2012-05-15 11:01:52,924 INFO repeater task phins2_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 11:01:57,929 INFO repeater task wh300_lastensq starting 571.9 seconds late 2012-05-15 11:02:02,204 INFO repeater task wh300_lastensq ended after 4.3 seconds 2012-05-15 11:02:02,205 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -6528.841 2012-05-15 11:02:02,206 INFO repeater task wh300_lastensq: added 4 extra increments to time_to_run 2012-05-15 11:02:02,207 INFO repeater task os75bb_lastensq starting 237.2 seconds late 2012-05-15 11:02:05,140 INFO repeater task os75bb_lastensq ended after 2.9 seconds 2012-05-15 11:02:05,141 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -6452.147 2012-05-15 11:02:05,153 INFO repeater task wh300_beamplots starting 11.1 seconds late 2012-05-15 11:02:13,204 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 11:02:13,205 INFO repeater task wh300_beamplots ended after 8.1 seconds 2012-05-15 11:02:13,205 INFO repeater task wh300_beamplots starting 19.2 seconds late 2012-05-15 11:02:21,572 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 11:02:21,572 INFO repeater task wh300_beamplots ended after 8.4 seconds 2012-05-15 11:02:21,573 INFO repeater task os75bb_beamplots starting 27.6 seconds late 2012-05-15 11:02:26,510 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 11:02:26,510 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2012-05-15 11:02:26,513 INFO repeater task os75bb_beamplots starting 32.5 seconds late 2012-05-15 11:02:31,172 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 11:02:31,173 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2012-05-15 11:02:31,175 INFO repeater task os75nb_beamplots starting 37.2 seconds late 2012-05-15 11:02:35,625 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 11:02:35,625 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2012-05-15 11:02:35,629 INFO repeater task os75nb_beamplots starting 41.6 seconds late 2012-05-15 11:02:39,836 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 11:02:39,836 INFO repeater task os75nb_beamplots ended after 4.2 seconds 2012-05-15 11:02:39,839 INFO repeater task phins_hcorrstats starting 45.8 seconds late 2012-05-15 11:02:44,798 INFO repeater task phins_hcorrstats ended after 5.0 seconds 2012-05-15 11:02:44,799 INFO repeater task phins2_hcorrstats starting 50.8 seconds late 2012-05-15 11:02:49,708 INFO repeater task phins2_hcorrstats ended after 4.9 seconds 2012-05-15 11:03:19,770 INFO repeater task wh300_lastensq starting 3.8 seconds late 2012-05-15 11:03:23,225 INFO repeater task wh300_lastensq ended after 3.5 seconds 2012-05-15 11:03:23,226 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -6610.682 2012-05-15 11:03:23,228 INFO repeater task os75bb_lastensq starting 8.2 seconds late 2012-05-15 11:03:25,382 INFO repeater task os75bb_lastensq ended after 2.2 seconds 2012-05-15 11:03:25,383 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -6533.168 2012-05-15 11:03:25,385 INFO repeater task os75nb_lastensq starting 10.4 seconds late 2012-05-15 11:03:28,656 INFO repeater task os75nb_lastensq ended after 3.3 seconds 2012-05-15 11:03:28,657 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -6291.492 2012-05-15 11:05:29,105 INFO repeater task wh300_lastensq starting 3.1 seconds late 2012-05-15 11:05:31,630 INFO repeater task wh300_lastensq ended after 2.5 seconds 2012-05-15 11:05:31,631 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -6740.017 2012-05-15 11:06:56,837 INFO repeater task wh300_quick starting 2.8 seconds late 2012-05-15 11:09:34,457 INFO repeater task wh300_quick ended after 157.6 seconds 2012-05-15 11:09:34,464 INFO repeater task wh300_3dayplots starting 160.5 seconds late 2012-05-15 11:10:31,366 INFO repeater task wh300_3dayplots ended after 56.9 seconds 2012-05-15 11:10:31,368 INFO repeater task os75bb_lastensq starting 126.4 seconds late 2012-05-15 11:10:35,197 INFO repeater task os75bb_lastensq ended after 3.8 seconds 2012-05-15 11:10:35,198 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -6961.307 2012-05-15 11:10:35,199 INFO repeater task os75bb_quick starting 221.2 seconds late 2012-05-15 11:12:31,321 INFO repeater task os75bb_quick ended after 116.1 seconds 2012-05-15 11:12:31,325 INFO repeater task os75bb_3dayplots starting 337.3 seconds late 2012-05-15 11:13:27,654 INFO repeater task os75bb_3dayplots ended after 56.3 seconds 2012-05-15 11:13:27,655 INFO repeater task os75nb_lastensq starting 302.7 seconds late 2012-05-15 11:13:30,709 INFO repeater task os75nb_lastensq ended after 3.1 seconds 2012-05-15 11:13:30,710 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -6893.763 2012-05-15 11:13:30,712 INFO repeater task os75nb_quick starting 396.7 seconds late 2012-05-15 11:15:11,502 INFO repeater task os75nb_quick ended after 100.8 seconds 2012-05-15 11:15:11,504 INFO repeater task os75nb_3dayplots starting 497.5 seconds late 2012-05-15 11:16:06,376 INFO repeater task os75nb_3dayplots ended after 54.9 seconds 2012-05-15 11:16:06,377 INFO repeater task wh300_beamplots starting 252.4 seconds late 2012-05-15 11:16:15,559 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 11:16:15,560 INFO repeater task wh300_beamplots ended after 9.2 seconds 2012-05-15 11:16:15,565 INFO repeater task wh300_beamplots starting 261.5 seconds late 2012-05-15 11:16:23,130 INFO repeater task wh300_beamplots output: Starting procsetup 2012-05-15 11:16:23,131 INFO repeater task wh300_beamplots ended after 7.6 seconds 2012-05-15 11:16:23,132 INFO repeater task os75bb_beamplots starting 269.1 seconds late 2012-05-15 11:16:28,049 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 11:16:28,050 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2012-05-15 11:16:28,051 INFO repeater task os75bb_beamplots starting 274.0 seconds late 2012-05-15 11:16:32,909 INFO repeater task os75bb_beamplots output: Starting procsetup 2012-05-15 11:16:32,910 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2012-05-15 11:16:32,910 INFO repeater task os75nb_beamplots starting 278.9 seconds late 2012-05-15 11:16:37,511 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 11:16:37,511 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2012-05-15 11:16:37,512 INFO repeater task os75nb_beamplots starting 283.5 seconds late 2012-05-15 11:16:42,235 INFO repeater task os75nb_beamplots output: Starting procsetup 2012-05-15 11:16:42,235 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2012-05-15 11:16:42,236 INFO repeater task phins_hcorrstats starting 588.2 seconds late 2012-05-15 11:16:48,090 INFO repeater task phins_hcorrstats ended after 5.9 seconds 2012-05-15 11:16:48,091 INFO repeater task phins_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 11:16:48,092 INFO repeater task phins2_hcorrstats starting 594.1 seconds late 2012-05-15 11:16:52,903 INFO repeater task phins2_hcorrstats ended after 4.8 seconds 2012-05-15 11:16:52,903 INFO repeater task phins2_hcorrstats: added 1 extra increments to time_to_run 2012-05-15 11:16:57,909 INFO repeater task wh300_lastensq starting 561.9 seconds late 2012-05-15 11:17:02,199 INFO repeater task wh300_lastensq ended after 4.3 seconds 2012-05-15 11:17:02,200 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -7428.821 2012-05-15 11:17:02,200 INFO repeater task wh300_lastensq: added 4 extra increments to time_to_run 2012-05-15 11:17:02,202 INFO repeater task os75bb_lastensq starting 207.2 seconds late 2012-05-15 11:17:05,058 INFO repeater task os75bb_lastensq ended after 2.9 seconds 2012-05-15 11:17:05,059 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -7352.142 2012-05-15 11:17:05,065 INFO repeater task os75nb_lastensq starting 210.1 seconds late 2012-05-15 11:17:07,990 INFO repeater task os75nb_lastensq ended after 2.9 seconds 2012-05-15 11:17:07,990 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -7111.173 2012-05-15 11:17:07,995 INFO repeater task phins_hcorrstats starting 14.0 seconds late 2012-05-15 11:17:12,991 INFO repeater task phins_hcorrstats ended after 5.0 seconds 2012-05-15 11:17:12,992 INFO repeater task phins2_hcorrstats starting 19.0 seconds late 2012-05-15 11:17:17,908 INFO repeater task phins2_hcorrstats ended after 4.9 seconds 2012-05-15 11:18:28,077 INFO repeater task wh300_lastensq starting 2.1 seconds late 2012-05-15 11:18:30,420 INFO repeater task wh300_lastensq ended after 2.3 seconds 2012-05-15 11:18:30,420 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -7518.989 2012-05-15 11:18:45,458 INFO repeater task os75bb_lastensq starting 0.5 seconds late 2012-05-15 11:18:48,701 INFO repeater task os75bb_lastensq ended after 3.2 seconds 2012-05-15 11:18:48,701 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -7455.398 2012-05-15 11:18:48,706 INFO repeater task os75nb_lastensq starting 3.7 seconds late 2012-05-15 11:18:50,999 INFO repeater task os75nb_lastensq ended after 2.3 seconds 2012-05-15 11:18:51,000 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -7214.814 2012-05-15 11:20:36,541 INFO repeater task wh300_lastensq starting 0.5 seconds late 2012-05-15 11:20:39,997 INFO repeater task wh300_lastensq ended after 3.5 seconds 2012-05-15 11:20:39,998 WARNING repeater task wh300_lastensq: Expected update of /home/data/at20/proc/wh300/load/wrote_ens_time did not occur, dt = -7647.453 2012-05-15 11:21:55,217 INFO repeater task wh300_quick starting 1.2 seconds late 2012-05-15 11:24:40,033 INFO repeater task wh300_quick ended after 164.8 seconds 2012-05-15 11:24:40,034 INFO repeater task wh300_3dayplots starting 166.0 seconds late 2012-05-15 11:25:35,454 INFO repeater task wh300_3dayplots ended after 55.4 seconds 2012-05-15 11:25:35,455 INFO repeater task os75bb_lastensq starting 100.5 seconds late 2012-05-15 11:25:39,657 INFO repeater task os75bb_lastensq ended after 4.2 seconds 2012-05-15 11:25:39,658 WARNING repeater task os75bb_lastensq: Expected update of /home/data/at20/proc/os75bb/load/wrote_ens_time did not occur, dt = -7865.394 2012-05-15 11:25:39,659 INFO repeater task os75bb_quick starting 225.7 seconds late 2012-05-15 11:27:39,900 INFO repeater task os75bb_quick ended after 120.2 seconds 2012-05-15 11:27:39,905 INFO repeater task os75bb_3dayplots starting 345.9 seconds late 2012-05-15 11:28:35,964 INFO repeater task os75bb_3dayplots ended after 56.1 seconds 2012-05-15 11:28:35,970 INFO repeater task os75nb_lastensq starting 281.0 seconds late 2012-05-15 11:28:39,555 INFO repeater task os75nb_lastensq ended after 3.6 seconds 2012-05-15 11:28:39,556 WARNING repeater task os75nb_lastensq: Expected update of /home/data/at20/proc/os75nb/load/wrote_ens_time did not occur, dt = -7802.077 2012-05-15 11:28:39,557 INFO repeater task os75nb_quick starting 405.5 seconds late