2010-02-25 20:03:45,943 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:45,973 INFO repeater task os75bb_beamplots starting 526.6 seconds late 2010-02-25 20:03:50,537 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 20:03:50,537 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:50,537 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:50,538 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:50,538 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:50,540 INFO repeater task os75nb_beamplots starting 531.2 seconds late 2010-02-25 20:03:54,877 INFO repeater task os75nb_beamplots ended after 4.3 seconds 2010-02-25 20:03:54,877 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:54,878 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:54,878 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:54,878 INFO repeater Extra increment added to time_to_run 2010-02-25 20:03:54,879 INFO repeater task wh300_beamplots starting 175.5 seconds late 2010-02-25 20:04:16,458 INFO repeater task wh300_beamplots ended after 21.6 seconds 2010-02-25 20:04:16,470 INFO repeater task os75bb_beamplots starting 197.1 seconds late 2010-02-25 20:04:38,434 INFO repeater task os75bb_beamplots ended after 22.0 seconds 2010-02-25 20:04:38,435 INFO repeater task os75nb_beamplots starting 219.1 seconds late 2010-02-25 20:05:03,086 INFO repeater task os75nb_beamplots ended after 24.7 seconds 2010-02-25 20:05:08,090 INFO repeater task wh300_lastensq starting 66.1 seconds late 2010-02-25 20:05:35,864 INFO repeater task wh300_lastensq ended after 27.8 seconds 2010-02-25 20:05:35,865 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:03:52 2010-02-25 20:05:35,869 INFO repeater task wh300_beamplots starting 36.5 seconds late 2010-02-25 20:05:40,465 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 20:05:40,466 INFO repeater task os75bb_beamplots starting 41.1 seconds late 2010-02-25 20:05:45,301 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 20:05:45,319 INFO repeater task os75nb_beamplots starting 45.9 seconds late 2010-02-25 20:05:51,857 INFO repeater task os75nb_beamplots ended after 6.5 seconds 2010-02-25 20:06:01,927 INFO repeater task posmv_hcorrstats starting 2.5 seconds late 2010-02-25 20:06:24,742 INFO repeater task posmv_hcorrstats ended after 22.8 seconds 2010-02-25 20:06:24,743 INFO repeater task phins_hcorrstats starting 25.4 seconds late 2010-02-25 20:06:42,696 INFO repeater task phins_hcorrstats ended after 18.0 seconds 2010-02-25 20:06:47,695 INFO repeater task wh300_lastensq starting 45.7 seconds late 2010-02-25 20:07:12,634 INFO repeater task wh300_lastensq ended after 24.9 seconds 2010-02-25 20:07:12,635 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:05:52 2010-02-25 20:07:12,637 INFO repeater task os75nb_lastensq starting 24.6 seconds late 2010-02-25 20:07:36,708 INFO repeater task os75nb_lastensq ended after 24.1 seconds 2010-02-25 20:07:36,709 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:06:40 2010-02-25 20:07:36,711 INFO repeater task wh300_beamplots starting 37.3 seconds late 2010-02-25 20:07:41,171 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 20:07:41,172 INFO repeater task os75bb_beamplots starting 41.8 seconds late 2010-02-25 20:07:45,874 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 20:07:45,876 INFO repeater task os75nb_beamplots starting 46.5 seconds late 2010-02-25 20:07:51,920 INFO repeater task os75nb_beamplots ended after 6.0 seconds 2010-02-25 20:07:57,032 INFO repeater task os75bb_lastensq starting 34.0 seconds late 2010-02-25 20:08:17,881 INFO repeater task os75bb_lastensq ended after 20.9 seconds 2010-02-25 20:08:17,882 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:07:14 2010-02-25 20:08:22,887 INFO repeater task wh300_lastensq starting 20.9 seconds late 2010-02-25 20:08:47,323 INFO repeater task wh300_lastensq ended after 24.4 seconds 2010-02-25 20:08:47,323 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:07:52 2010-02-25 20:09:02,529 INFO repeater task wh300_beamplots starting 3.2 seconds late 2010-02-25 20:09:06,509 INFO repeater task wh300_beamplots ended after 4.0 seconds 2010-02-25 20:09:06,521 INFO repeater task os75bb_beamplots starting 7.2 seconds late 2010-02-25 20:09:11,142 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 20:09:11,143 INFO repeater task os75nb_beamplots starting 11.8 seconds late 2010-02-25 20:09:15,673 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 20:10:06,339 INFO repeater task wh300_lastensq starting 4.3 seconds late 2010-02-25 20:10:31,502 INFO repeater task wh300_lastensq ended after 25.2 seconds 2010-02-25 20:10:31,503 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:09:52 2010-02-25 20:11:01,764 INFO repeater task wh300_beamplots starting 2.4 seconds late 2010-02-25 20:11:06,177 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 20:11:06,178 INFO repeater task os75bb_beamplots starting 6.8 seconds late 2010-02-25 20:11:11,004 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 20:11:11,005 INFO repeater task os75nb_beamplots starting 11.6 seconds late 2010-02-25 20:11:15,702 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 20:11:15,703 INFO repeater task wh300_beamplots starting 16.3 seconds late 2010-02-25 20:11:39,353 INFO repeater task wh300_beamplots ended after 23.7 seconds 2010-02-25 20:11:39,356 INFO repeater task os75bb_beamplots starting 40.0 seconds late 2010-02-25 20:12:02,718 INFO repeater task os75bb_beamplots ended after 23.4 seconds 2010-02-25 20:12:02,719 INFO repeater task os75nb_beamplots starting 63.3 seconds late 2010-02-25 20:12:25,697 INFO repeater task os75nb_beamplots ended after 23.0 seconds 2010-02-25 20:12:25,735 INFO repeater task posmv_hcorrstats starting 86.4 seconds late 2010-02-25 20:12:46,329 INFO repeater task posmv_hcorrstats ended after 20.6 seconds 2010-02-25 20:12:46,339 INFO repeater task phins_hcorrstats starting 107.0 seconds late 2010-02-25 20:13:05,981 INFO repeater task phins_hcorrstats ended after 19.6 seconds 2010-02-25 20:13:10,986 INFO repeater task wh300_lastensq starting 69.0 seconds late 2010-02-25 20:13:38,438 INFO repeater task wh300_lastensq ended after 27.5 seconds 2010-02-25 20:13:38,439 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:11:52 2010-02-25 20:13:38,441 INFO repeater task os75bb_lastensq starting 74.4 seconds late 2010-02-25 20:14:00,851 INFO repeater task os75bb_lastensq ended after 22.4 seconds 2010-02-25 20:14:00,851 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:12:16 2010-02-25 20:14:00,853 INFO repeater task os75nb_lastensq starting 130.9 seconds late 2010-02-25 20:14:21,078 INFO repeater task os75nb_lastensq ended after 20.2 seconds 2010-02-25 20:14:21,079 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:11:41 2010-02-25 20:14:21,081 INFO repeater task wh300_beamplots starting 81.7 seconds late 2010-02-25 20:14:25,754 INFO repeater task wh300_beamplots ended after 4.7 seconds 2010-02-25 20:14:25,759 INFO repeater task os75bb_beamplots starting 86.4 seconds late 2010-02-25 20:14:31,323 INFO repeater task os75bb_beamplots ended after 5.6 seconds 2010-02-25 20:14:31,324 INFO repeater task os75nb_beamplots starting 92.0 seconds late 2010-02-25 20:14:36,020 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 20:14:41,023 INFO repeater task wh300_lastensq starting 39.0 seconds late 2010-02-25 20:15:07,082 INFO repeater task wh300_lastensq ended after 26.1 seconds 2010-02-25 20:15:07,083 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:13:52 2010-02-25 20:15:07,086 INFO repeater task wh300_beamplots starting 7.7 seconds late 2010-02-25 20:15:11,772 INFO repeater task wh300_beamplots ended after 4.7 seconds 2010-02-25 20:15:11,773 INFO repeater task os75bb_beamplots starting 12.4 seconds late 2010-02-25 20:15:16,319 INFO repeater task os75bb_beamplots ended after 4.5 seconds 2010-02-25 20:15:16,329 INFO repeater task os75nb_beamplots starting 17.0 seconds late 2010-02-25 20:15:20,773 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2010-02-25 20:16:01,502 INFO repeater task posmv_hcorrstats starting 2.1 seconds late 2010-02-25 20:16:22,368 INFO repeater task posmv_hcorrstats ended after 20.9 seconds 2010-02-25 20:16:22,378 INFO repeater task phins_hcorrstats starting 23.0 seconds late 2010-02-25 20:16:41,191 INFO repeater task phins_hcorrstats ended after 18.8 seconds 2010-02-25 20:16:46,199 INFO repeater task wh300_lastensq starting 44.2 seconds late 2010-02-25 20:17:11,588 INFO repeater task wh300_lastensq ended after 25.4 seconds 2010-02-25 20:17:11,589 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:15:52 2010-02-25 20:17:11,645 INFO repeater task os75nb_lastensq starting 20.6 seconds late 2010-02-25 20:17:35,390 INFO repeater task os75nb_lastensq ended after 23.7 seconds 2010-02-25 20:17:35,391 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:16:43 2010-02-25 20:17:35,392 INFO repeater task wh300_beamplots starting 36.0 seconds late 2010-02-25 20:17:39,916 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 20:17:39,917 INFO repeater task os75bb_beamplots starting 40.5 seconds late 2010-02-25 20:17:44,851 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2010-02-25 20:17:44,852 INFO repeater task os75nb_beamplots starting 45.5 seconds late 2010-02-25 20:17:49,418 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:17:54,462 INFO repeater task os75bb_lastensq starting 28.5 seconds late 2010-02-25 20:18:16,320 INFO repeater task os75bb_lastensq ended after 21.9 seconds 2010-02-25 20:18:16,321 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:17:18 2010-02-25 20:18:21,327 INFO repeater task wh300_lastensq starting 19.3 seconds late 2010-02-25 20:18:46,735 INFO repeater task wh300_lastensq ended after 25.4 seconds 2010-02-25 20:18:46,735 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:17:53 2010-02-25 20:19:01,762 INFO repeater task wh300_beamplots starting 2.4 seconds late 2010-02-25 20:19:06,547 INFO repeater task wh300_beamplots ended after 4.8 seconds 2010-02-25 20:19:06,548 INFO repeater task os75bb_beamplots starting 7.2 seconds late 2010-02-25 20:19:11,375 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 20:19:11,376 INFO repeater task os75nb_beamplots starting 12.0 seconds late 2010-02-25 20:19:15,980 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:20:06,585 INFO repeater task wh300_lastensq starting 3.6 seconds late 2010-02-25 20:20:29,793 INFO repeater task wh300_lastensq ended after 23.2 seconds 2010-02-25 20:20:29,794 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:19:54 2010-02-25 20:21:00,255 INFO repeater task wh300_quick starting 0.9 seconds late 2010-02-25 20:21:40,737 INFO repeater task wh300_quick ended after 40.5 seconds 2010-02-25 20:21:40,738 INFO repeater task wh300_3dayplots starting 41.4 seconds late 2010-02-25 20:21:57,215 INFO repeater task wh300_3dayplots ended after 16.5 seconds 2010-02-25 20:21:57,216 INFO repeater task os75bb_quick starting 57.9 seconds late 2010-02-25 20:22:31,733 INFO repeater task os75bb_quick ended after 34.5 seconds 2010-02-25 20:22:31,741 INFO repeater task os75bb_3dayplots starting 92.4 seconds late 2010-02-25 20:22:50,448 INFO repeater task os75bb_3dayplots output: zgrid: Warning: solution did not converge. 2010-02-25 20:22:50,448 INFO repeater task os75bb_3dayplots ended after 18.7 seconds 2010-02-25 20:22:50,457 INFO repeater task os75nb_lastensq starting 57.5 seconds late 2010-02-25 20:23:12,090 INFO repeater task os75nb_lastensq ended after 21.6 seconds 2010-02-25 20:23:12,091 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:21:45 2010-02-25 20:23:12,092 INFO repeater task os75nb_quick starting 132.7 seconds late 2010-02-25 20:23:48,007 INFO repeater task os75nb_quick ended after 35.9 seconds 2010-02-25 20:23:48,019 INFO repeater task os75nb_3dayplots starting 168.7 seconds late 2010-02-25 20:24:04,472 INFO repeater task os75nb_3dayplots ended after 16.5 seconds 2010-02-25 20:24:04,483 INFO repeater task wh300_beamplots starting 185.1 seconds late 2010-02-25 20:24:09,804 INFO repeater task wh300_beamplots ended after 5.3 seconds 2010-02-25 20:24:09,804 INFO repeater Extra increment added to time_to_run 2010-02-25 20:24:09,814 INFO repeater task os75bb_beamplots starting 190.4 seconds late 2010-02-25 20:24:14,453 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 20:24:14,454 INFO repeater Extra increment added to time_to_run 2010-02-25 20:24:14,454 INFO repeater task os75nb_beamplots starting 195.1 seconds late 2010-02-25 20:24:19,216 INFO repeater task os75nb_beamplots ended after 4.8 seconds 2010-02-25 20:24:19,217 INFO repeater Extra increment added to time_to_run 2010-02-25 20:24:19,217 INFO repeater task wh300_beamplots starting 199.8 seconds late 2010-02-25 20:24:40,283 INFO repeater task wh300_beamplots ended after 21.1 seconds 2010-02-25 20:24:40,284 INFO repeater task os75bb_beamplots starting 220.9 seconds late 2010-02-25 20:25:03,783 INFO repeater task os75bb_beamplots ended after 23.5 seconds 2010-02-25 20:25:03,798 INFO repeater task os75nb_beamplots starting 244.4 seconds late 2010-02-25 20:25:26,708 INFO repeater task os75nb_beamplots ended after 22.9 seconds 2010-02-25 20:25:26,746 INFO repeater task posmv_hcorrstats starting 267.4 seconds late 2010-02-25 20:25:51,680 INFO repeater task posmv_hcorrstats ended after 24.9 seconds 2010-02-25 20:25:51,681 INFO repeater task phins_hcorrstats starting 292.3 seconds late 2010-02-25 20:26:11,229 INFO repeater task phins_hcorrstats ended after 19.5 seconds 2010-02-25 20:26:11,230 INFO repeater Extra increment added to time_to_run 2010-02-25 20:26:16,231 INFO repeater task wh300_lastensq starting 252.2 seconds late 2010-02-25 20:26:46,866 INFO repeater task wh300_lastensq ended after 30.6 seconds 2010-02-25 20:26:46,867 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:25:54 2010-02-25 20:26:46,896 INFO repeater task os75bb_lastensq starting 258.9 seconds late 2010-02-25 20:27:08,089 INFO repeater task os75bb_lastensq ended after 21.2 seconds 2010-02-25 20:27:08,089 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:22:19 2010-02-25 20:27:08,091 INFO repeater task os75nb_lastensq starting 13.1 seconds late 2010-02-25 20:27:29,088 INFO repeater task os75nb_lastensq ended after 21.0 seconds 2010-02-25 20:27:29,088 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:26:47 2010-02-25 20:27:29,118 INFO repeater task wh300_beamplots starting 149.7 seconds late 2010-02-25 20:27:33,914 INFO repeater task wh300_beamplots ended after 4.8 seconds 2010-02-25 20:27:33,915 INFO repeater Extra increment added to time_to_run 2010-02-25 20:27:33,921 INFO repeater task os75bb_beamplots starting 154.6 seconds late 2010-02-25 20:27:42,301 INFO repeater task os75bb_beamplots ended after 8.4 seconds 2010-02-25 20:27:42,301 INFO repeater Extra increment added to time_to_run 2010-02-25 20:27:42,302 INFO repeater task os75nb_beamplots starting 162.9 seconds late 2010-02-25 20:27:46,606 INFO repeater task os75nb_beamplots ended after 4.3 seconds 2010-02-25 20:27:46,607 INFO repeater Extra increment added to time_to_run 2010-02-25 20:27:46,668 INFO repeater task posmv_hcorrstats starting 107.3 seconds late 2010-02-25 20:28:09,449 INFO repeater task posmv_hcorrstats ended after 22.8 seconds 2010-02-25 20:28:14,455 INFO repeater task wh300_lastensq starting 10.5 seconds late 2010-02-25 20:28:39,956 INFO repeater task wh300_lastensq ended after 25.5 seconds 2010-02-25 20:28:39,957 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:27:54 2010-02-25 20:28:39,998 INFO repeater task os75bb_lastensq starting 71.0 seconds late 2010-02-25 20:29:01,372 INFO repeater task os75bb_lastensq ended after 21.4 seconds 2010-02-25 20:29:01,373 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:27:21 2010-02-25 20:29:01,430 INFO repeater task wh300_beamplots starting 2.1 seconds late 2010-02-25 20:29:05,978 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 20:29:05,979 INFO repeater task os75bb_beamplots starting 6.6 seconds late 2010-02-25 20:29:12,841 INFO repeater task os75bb_beamplots ended after 6.9 seconds 2010-02-25 20:29:12,842 INFO repeater task os75nb_beamplots starting 13.5 seconds late 2010-02-25 20:29:17,431 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:30:07,549 INFO repeater task wh300_lastensq starting 3.5 seconds late 2010-02-25 20:30:32,668 INFO repeater task wh300_lastensq ended after 25.1 seconds 2010-02-25 20:30:32,669 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:29:54 2010-02-25 20:31:02,714 INFO repeater task wh300_beamplots starting 3.3 seconds late 2010-02-25 20:31:07,335 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 20:31:07,336 INFO repeater task os75bb_beamplots starting 8.0 seconds late 2010-02-25 20:31:13,579 INFO repeater task os75bb_beamplots ended after 6.2 seconds 2010-02-25 20:31:13,587 INFO repeater task os75nb_beamplots starting 14.2 seconds late 2010-02-25 20:31:18,152 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:31:18,159 INFO repeater task wh300_beamplots starting 18.8 seconds late 2010-02-25 20:31:39,973 INFO repeater task wh300_beamplots ended after 21.8 seconds 2010-02-25 20:31:39,985 INFO repeater task os75bb_beamplots starting 40.6 seconds late 2010-02-25 20:32:03,802 INFO repeater task os75bb_beamplots ended after 23.8 seconds 2010-02-25 20:32:03,812 INFO repeater task os75nb_beamplots starting 64.4 seconds late 2010-02-25 20:32:27,792 INFO repeater task os75nb_beamplots ended after 24.0 seconds 2010-02-25 20:32:27,807 INFO repeater task posmv_hcorrstats starting 88.4 seconds late 2010-02-25 20:32:49,992 INFO repeater task posmv_hcorrstats ended after 22.2 seconds 2010-02-25 20:32:49,999 INFO repeater task phins_hcorrstats starting 110.6 seconds late 2010-02-25 20:33:07,557 INFO repeater task phins_hcorrstats ended after 17.6 seconds 2010-02-25 20:33:12,562 INFO repeater task wh300_lastensq starting 68.6 seconds late 2010-02-25 20:33:38,803 INFO repeater task wh300_lastensq ended after 26.2 seconds 2010-02-25 20:33:38,804 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:31:54 2010-02-25 20:33:38,829 INFO repeater task os75bb_lastensq starting 67.8 seconds late 2010-02-25 20:34:03,320 INFO repeater task os75bb_lastensq ended after 24.5 seconds 2010-02-25 20:34:03,321 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:32:23 2010-02-25 20:34:03,323 INFO repeater task os75nb_lastensq starting 126.3 seconds late 2010-02-25 20:34:25,833 INFO repeater task os75nb_lastensq ended after 22.5 seconds 2010-02-25 20:34:25,834 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:31:48 2010-02-25 20:34:25,855 INFO repeater task wh300_beamplots starting 86.5 seconds late 2010-02-25 20:34:30,151 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 20:34:30,152 INFO repeater task os75bb_beamplots starting 90.8 seconds late 2010-02-25 20:34:34,944 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 20:34:34,945 INFO repeater task os75nb_beamplots starting 95.6 seconds late 2010-02-25 20:34:39,541 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:34:44,543 INFO repeater task wh300_lastensq starting 40.5 seconds late 2010-02-25 20:35:09,751 INFO repeater task wh300_lastensq ended after 25.2 seconds 2010-02-25 20:35:09,752 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:33:54 2010-02-25 20:35:09,756 INFO repeater task wh300_beamplots starting 10.4 seconds late 2010-02-25 20:35:15,718 INFO repeater task wh300_beamplots ended after 6.0 seconds 2010-02-25 20:35:15,729 INFO repeater task os75bb_beamplots starting 16.4 seconds late 2010-02-25 20:35:21,178 INFO repeater task os75bb_beamplots ended after 5.4 seconds 2010-02-25 20:35:21,179 INFO repeater task os75nb_beamplots starting 21.8 seconds late 2010-02-25 20:35:25,743 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:36:01,525 INFO repeater task posmv_hcorrstats starting 2.1 seconds late 2010-02-25 20:36:24,763 INFO repeater task posmv_hcorrstats ended after 23.2 seconds 2010-02-25 20:36:24,771 INFO repeater task phins_hcorrstats starting 25.4 seconds late 2010-02-25 20:36:42,535 INFO repeater task phins_hcorrstats ended after 17.8 seconds 2010-02-25 20:36:47,551 INFO repeater task wh300_lastensq starting 43.6 seconds late 2010-02-25 20:37:13,138 INFO repeater task wh300_lastensq ended after 25.6 seconds 2010-02-25 20:37:13,139 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:35:54 2010-02-25 20:37:13,142 INFO repeater task os75nb_lastensq starting 15.1 seconds late 2010-02-25 20:37:35,291 INFO repeater task os75nb_lastensq ended after 22.1 seconds 2010-02-25 20:37:35,292 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:36:50 2010-02-25 20:37:35,300 INFO repeater task wh300_beamplots starting 35.9 seconds late 2010-02-25 20:37:40,648 INFO repeater task wh300_beamplots ended after 5.3 seconds 2010-02-25 20:37:40,657 INFO repeater task os75bb_beamplots starting 41.3 seconds late 2010-02-25 20:37:47,179 INFO repeater task os75bb_beamplots ended after 6.5 seconds 2010-02-25 20:37:47,180 INFO repeater task os75nb_beamplots starting 47.8 seconds late 2010-02-25 20:37:52,917 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2010-02-25 20:37:57,920 INFO repeater task os75bb_lastensq starting 24.9 seconds late 2010-02-25 20:38:18,996 INFO repeater task os75bb_lastensq ended after 21.1 seconds 2010-02-25 20:38:18,997 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:37:25 2010-02-25 20:38:24,006 INFO repeater task wh300_lastensq starting 20.0 seconds late 2010-02-25 20:38:48,781 INFO repeater task wh300_lastensq ended after 24.8 seconds 2010-02-25 20:38:48,782 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:37:54 2010-02-25 20:39:04,079 INFO repeater task wh300_beamplots starting 4.7 seconds late 2010-02-25 20:39:08,475 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 20:39:08,476 INFO repeater task os75bb_beamplots starting 9.1 seconds late 2010-02-25 20:39:13,379 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2010-02-25 20:39:13,380 INFO repeater task os75nb_beamplots starting 14.0 seconds late 2010-02-25 20:39:17,858 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 20:40:08,620 INFO repeater task wh300_lastensq starting 4.6 seconds late 2010-02-25 20:40:41,612 INFO repeater task wh300_lastensq ended after 33.0 seconds 2010-02-25 20:40:41,613 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:39:54 2010-02-25 20:41:01,946 INFO repeater task wh300_beamplots starting 2.6 seconds late 2010-02-25 20:41:07,526 INFO repeater task wh300_beamplots ended after 5.6 seconds 2010-02-25 20:41:07,527 INFO repeater task os75bb_beamplots starting 8.2 seconds late 2010-02-25 20:41:12,370 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 20:41:12,371 INFO repeater task os75nb_beamplots starting 13.0 seconds late 2010-02-25 20:41:17,070 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 20:41:17,071 INFO repeater task wh300_beamplots starting 17.7 seconds late 2010-02-25 20:41:39,066 INFO repeater task wh300_beamplots ended after 22.0 seconds 2010-02-25 20:41:39,074 INFO repeater task os75bb_beamplots starting 39.7 seconds late 2010-02-25 20:42:04,953 INFO repeater task os75bb_beamplots ended after 25.9 seconds 2010-02-25 20:42:04,954 INFO repeater task os75nb_beamplots starting 65.6 seconds late 2010-02-25 20:42:27,564 INFO repeater task os75nb_beamplots ended after 22.6 seconds 2010-02-25 20:42:27,635 INFO repeater task posmv_hcorrstats starting 88.3 seconds late 2010-02-25 20:42:49,959 INFO repeater task posmv_hcorrstats ended after 22.3 seconds 2010-02-25 20:42:49,960 INFO repeater task phins_hcorrstats starting 110.6 seconds late 2010-02-25 20:43:09,143 INFO repeater task phins_hcorrstats ended after 19.2 seconds 2010-02-25 20:43:14,152 INFO repeater task wh300_lastensq starting 70.2 seconds late 2010-02-25 20:43:40,229 INFO repeater task wh300_lastensq ended after 26.1 seconds 2010-02-25 20:43:40,230 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:41:54 2010-02-25 20:43:40,232 INFO repeater task os75bb_lastensq starting 65.2 seconds late 2010-02-25 20:44:05,521 INFO repeater task os75bb_lastensq ended after 25.3 seconds 2010-02-25 20:44:05,522 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:42:26 2010-02-25 20:44:05,531 INFO repeater task os75nb_lastensq starting 125.5 seconds late 2010-02-25 20:44:26,646 INFO repeater task os75nb_lastensq ended after 21.1 seconds 2010-02-25 20:44:26,647 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:41:52 2010-02-25 20:44:26,649 INFO repeater task wh300_beamplots starting 87.3 seconds late 2010-02-25 20:44:32,546 INFO repeater task wh300_beamplots ended after 5.9 seconds 2010-02-25 20:44:32,550 INFO repeater task os75bb_beamplots starting 93.2 seconds late 2010-02-25 20:44:37,265 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 20:44:37,270 INFO repeater task os75nb_beamplots starting 97.9 seconds late 2010-02-25 20:44:41,801 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 20:44:46,806 INFO repeater task wh300_lastensq starting 42.8 seconds late 2010-02-25 20:45:12,712 INFO repeater task wh300_lastensq ended after 25.9 seconds 2010-02-25 20:45:12,712 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:43:55 2010-02-25 20:45:12,737 INFO repeater task wh300_beamplots starting 13.4 seconds late 2010-02-25 20:45:17,284 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 20:45:17,292 INFO repeater task os75bb_beamplots starting 17.9 seconds late 2010-02-25 20:45:22,043 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 20:45:22,044 INFO repeater task os75nb_beamplots starting 22.7 seconds late 2010-02-25 20:45:26,687 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:46:02,455 INFO repeater task posmv_hcorrstats starting 3.1 seconds late 2010-02-25 20:46:24,035 INFO repeater task posmv_hcorrstats ended after 21.6 seconds 2010-02-25 20:46:24,036 INFO repeater task phins_hcorrstats starting 24.7 seconds late 2010-02-25 20:46:43,041 INFO repeater task phins_hcorrstats ended after 19.0 seconds 2010-02-25 20:46:48,051 INFO repeater task wh300_lastensq starting 43.1 seconds late 2010-02-25 20:47:13,704 INFO repeater task wh300_lastensq ended after 25.7 seconds 2010-02-25 20:47:13,705 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:45:55 2010-02-25 20:47:13,707 INFO repeater task os75nb_lastensq starting 11.7 seconds late 2010-02-25 20:47:36,653 INFO repeater task os75nb_lastensq ended after 22.9 seconds 2010-02-25 20:47:36,654 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:46:54 2010-02-25 20:47:36,681 INFO repeater task wh300_beamplots starting 37.3 seconds late 2010-02-25 20:47:41,039 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 20:47:41,039 INFO repeater task os75bb_beamplots starting 41.7 seconds late 2010-02-25 20:47:46,844 INFO repeater task os75bb_beamplots ended after 5.8 seconds 2010-02-25 20:47:46,858 INFO repeater task os75nb_beamplots starting 47.5 seconds late 2010-02-25 20:47:52,932 INFO repeater task os75nb_beamplots ended after 6.1 seconds 2010-02-25 20:47:58,004 INFO repeater task os75bb_lastensq starting 22.0 seconds late 2010-02-25 20:48:20,699 INFO repeater task os75bb_lastensq ended after 22.7 seconds 2010-02-25 20:48:20,700 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 20:47:28 2010-02-25 20:48:25,710 INFO repeater task wh300_lastensq starting 20.7 seconds late 2010-02-25 20:48:52,665 INFO repeater task wh300_lastensq ended after 27.0 seconds 2010-02-25 20:48:52,666 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:47:55 2010-02-25 20:49:02,721 INFO repeater task wh300_beamplots starting 3.4 seconds late 2010-02-25 20:49:07,270 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 20:49:07,271 INFO repeater task os75bb_beamplots starting 7.9 seconds late 2010-02-25 20:49:13,065 INFO repeater task os75bb_beamplots ended after 5.8 seconds 2010-02-25 20:49:13,066 INFO repeater task os75nb_beamplots starting 13.7 seconds late 2010-02-25 20:49:17,705 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:50:08,023 INFO repeater task wh300_lastensq starting 3.0 seconds late 2010-02-25 20:50:34,787 INFO repeater task wh300_lastensq ended after 26.8 seconds 2010-02-25 20:50:34,788 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 20:49:55 2010-02-25 20:50:59,831 INFO repeater task wh300_quick starting 0.5 seconds late 2010-02-25 20:51:37,887 INFO repeater task wh300_quick ended after 38.1 seconds 2010-02-25 20:51:37,919 INFO repeater task wh300_3dayplots starting 38.6 seconds late 2010-02-25 20:52:00,418 INFO repeater task wh300_3dayplots ended after 22.5 seconds 2010-02-25 20:52:00,435 INFO repeater task os75bb_quick starting 61.1 seconds late 2010-02-25 20:52:34,042 INFO repeater task os75bb_quick ended after 33.6 seconds 2010-02-25 20:52:34,046 INFO repeater task os75bb_3dayplots starting 94.7 seconds late 2010-02-25 20:52:53,780 INFO repeater task os75bb_3dayplots output: zgrid: Warning: solution did not converge. zgrid: Warning: solution did not converge. 2010-02-25 20:52:53,780 INFO repeater task os75bb_3dayplots ended after 19.7 seconds 2010-02-25 20:52:53,790 INFO repeater task os75nb_lastensq starting 49.8 seconds late 2010-02-25 20:53:16,138 INFO repeater task os75nb_lastensq ended after 22.3 seconds 2010-02-25 20:53:16,139 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 20:51:55 2010-02-25 20:53:16,147 INFO repeater task os75nb_quick starting 136.8 seconds late 2010-02-25 20:53:54,706 INFO repeater task os75nb_quick ended after 38.6 seconds 2010-02-25 20:53:54,707 INFO repeater task os75nb_3dayplots starting 175.3 seconds late 2010-02-25 20:54:10,673 INFO repeater task os75nb_3dayplots ended after 16.0 seconds 2010-02-25 20:54:10,683 INFO repeater task wh300_beamplots starting 191.3 seconds late 2010-02-25 20:54:15,448 INFO repeater task wh300_beamplots ended after 4.8 seconds 2010-02-25 20:54:15,448 INFO repeater Extra increment added to time_to_run 2010-02-25 20:54:15,451 INFO repeater task os75bb_beamplots starting 196.1 seconds late 2010-02-25 20:54:20,887 INFO repeater task os75bb_beamplots ended after 5.4 seconds 2010-02-25 20:54:20,888 INFO repeater Extra increment added to time_to_run 2010-02-25 20:54:20,889 INFO repeater task os75nb_beamplots starting 201.5 seconds late 2010-02-25 20:54:25,514 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 20:54:25,514 INFO repeater Extra increment added to time_to_run 2010-02-25 20:54:25,516 INFO repeater task wh300_beamplots starting 206.1 seconds late 2010-02-25 20:54:47,333 INFO repeater task wh300_beamplots ended after 21.8 seconds 2010-02-25 20:54:47,343 INFO repeater task os75bb_beamplots starting 228.0 seconds late 2010-02-25 20:55:10,273 INFO repeater task os75bb_beamplots ended after 22.9 seconds 2010-02-25 20:55:10,274 INFO repeater task os75nb_beamplots starting 250.9 seconds late 2010-02-25 20:55:32,445 INFO repeater task os75nb_beamplots ended after 22.2 seconds 2010-02-25 20:55:32,455 INFO repeater task wh300_beamhistory starting 273.1 seconds late 2010-02-25 20:58:20,981 INFO repeater task wh300_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to wh300_bin_history.png 2010-02-25 20:58:21,000 INFO repeater task wh300_beamhistory ended after 168.5 seconds 2010-02-25 20:58:21,001 INFO repeater task os75bb_beamhistory starting 441.6 seconds late 2010-02-25 21:00:00,004 INFO repeater task os75bb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75bb_bin_history.png 2010-02-25 21:00:00,005 INFO repeater task os75bb_beamhistory ended after 99.0 seconds 2010-02-25 21:00:00,009 INFO repeater task os75nb_beamhistory starting 540.6 seconds late 2010-02-25 21:01:05,721 INFO repeater task os75nb_beamhistory output: saving file to os75nb_bin_history.png 2010-02-25 21:01:05,721 INFO repeater task os75nb_beamhistory ended after 65.7 seconds 2010-02-25 21:01:05,732 INFO repeater task posmv_hcorrstats starting 606.4 seconds late 2010-02-25 21:01:36,102 INFO repeater task posmv_hcorrstats ended after 30.4 seconds 2010-02-25 21:01:36,103 INFO repeater Extra increment added to time_to_run 2010-02-25 21:01:36,103 INFO repeater Extra increment added to time_to_run 2010-02-25 21:01:36,104 INFO repeater task phins_hcorrstats starting 636.7 seconds late 2010-02-25 21:01:57,937 INFO repeater task phins_hcorrstats ended after 21.8 seconds 2010-02-25 21:01:57,938 INFO repeater Extra increment added to time_to_run 2010-02-25 21:01:57,938 INFO repeater Extra increment added to time_to_run 2010-02-25 21:02:02,939 INFO repeater task wh300_lastensq starting 597.9 seconds late 2010-02-25 21:02:41,127 INFO repeater task wh300_lastensq ended after 38.2 seconds 2010-02-25 21:02:41,127 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:01:57 2010-02-25 21:02:41,129 INFO repeater task os75bb_lastensq starting 603.1 seconds late 2010-02-25 21:03:11,048 INFO repeater task os75bb_lastensq ended after 29.9 seconds 2010-02-25 21:03:11,049 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:02:33 2010-02-25 21:03:11,060 INFO repeater task os75nb_lastensq starting 366.1 seconds late 2010-02-25 21:03:35,872 INFO repeater task os75nb_lastensq ended after 24.8 seconds 2010-02-25 21:03:35,873 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:01:59 2010-02-25 21:03:35,897 INFO repeater task wh300_beamplots starting 516.5 seconds late 2010-02-25 21:03:40,379 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 21:03:40,380 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:40,380 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:40,380 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:40,381 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:40,390 INFO repeater task os75bb_beamplots starting 521.0 seconds late 2010-02-25 21:03:45,063 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:03:45,064 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:45,064 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:45,064 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:45,064 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:45,065 INFO repeater task os75nb_beamplots starting 525.7 seconds late 2010-02-25 21:03:49,364 INFO repeater task os75nb_beamplots ended after 4.3 seconds 2010-02-25 21:03:49,364 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:49,364 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:49,365 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:49,365 INFO repeater Extra increment added to time_to_run 2010-02-25 21:03:49,399 INFO repeater task wh300_beamplots starting 170.0 seconds late 2010-02-25 21:04:15,005 INFO repeater task wh300_beamplots ended after 25.6 seconds 2010-02-25 21:04:15,006 INFO repeater task os75bb_beamplots starting 195.6 seconds late 2010-02-25 21:04:37,441 INFO repeater task os75bb_beamplots ended after 22.4 seconds 2010-02-25 21:04:37,451 INFO repeater task os75nb_beamplots starting 218.1 seconds late 2010-02-25 21:04:59,832 INFO repeater task os75nb_beamplots ended after 22.4 seconds 2010-02-25 21:05:04,879 INFO repeater task wh300_lastensq starting 57.9 seconds late 2010-02-25 21:05:31,392 INFO repeater task wh300_lastensq ended after 26.5 seconds 2010-02-25 21:05:31,393 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:03:57 2010-02-25 21:05:31,397 INFO repeater task wh300_beamplots starting 32.0 seconds late 2010-02-25 21:05:36,285 INFO repeater task wh300_beamplots ended after 4.9 seconds 2010-02-25 21:05:36,286 INFO repeater task os75bb_beamplots starting 36.9 seconds late 2010-02-25 21:05:41,775 INFO repeater task os75bb_beamplots ended after 5.5 seconds 2010-02-25 21:05:41,776 INFO repeater task os75nb_beamplots starting 42.4 seconds late 2010-02-25 21:05:46,556 INFO repeater task os75nb_beamplots ended after 4.8 seconds 2010-02-25 21:06:01,578 INFO repeater task posmv_hcorrstats starting 2.2 seconds late 2010-02-25 21:06:25,433 INFO repeater task posmv_hcorrstats ended after 23.9 seconds 2010-02-25 21:06:25,447 INFO repeater task phins_hcorrstats starting 26.1 seconds late 2010-02-25 21:06:44,490 INFO repeater task phins_hcorrstats ended after 19.0 seconds 2010-02-25 21:06:49,499 INFO repeater task wh300_lastensq starting 42.5 seconds late 2010-02-25 21:07:15,675 INFO repeater task wh300_lastensq ended after 26.2 seconds 2010-02-25 21:07:15,676 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:05:57 2010-02-25 21:07:15,693 INFO repeater task os75nb_lastensq starting 6.7 seconds late 2010-02-25 21:07:36,650 INFO repeater task os75nb_lastensq ended after 21.0 seconds 2010-02-25 21:07:36,651 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:07:00 2010-02-25 21:07:36,675 INFO repeater task wh300_beamplots starting 37.3 seconds late 2010-02-25 21:07:42,466 INFO repeater task wh300_beamplots ended after 5.8 seconds 2010-02-25 21:07:42,467 INFO repeater task os75bb_beamplots starting 43.1 seconds late 2010-02-25 21:07:47,182 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:07:47,195 INFO repeater task os75nb_beamplots starting 47.8 seconds late 2010-02-25 21:07:51,568 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2010-02-25 21:07:56,599 INFO repeater task os75bb_lastensq starting 13.6 seconds late 2010-02-25 21:08:21,661 INFO repeater task os75bb_lastensq ended after 25.1 seconds 2010-02-25 21:08:21,662 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:07:35 2010-02-25 21:08:26,667 INFO repeater task wh300_lastensq starting 19.7 seconds late 2010-02-25 21:08:52,740 INFO repeater task wh300_lastensq ended after 26.1 seconds 2010-02-25 21:08:52,741 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:07:58 2010-02-25 21:09:02,898 INFO repeater task wh300_beamplots starting 3.5 seconds late 2010-02-25 21:09:07,556 INFO repeater task wh300_beamplots ended after 4.7 seconds 2010-02-25 21:09:07,567 INFO repeater task os75bb_beamplots starting 8.2 seconds late 2010-02-25 21:09:13,206 INFO repeater task os75bb_beamplots ended after 5.6 seconds 2010-02-25 21:09:13,207 INFO repeater task os75nb_beamplots starting 13.8 seconds late 2010-02-25 21:09:17,869 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 21:10:08,119 INFO repeater task wh300_lastensq starting 0.1 seconds late 2010-02-25 21:10:34,080 INFO repeater task wh300_lastensq ended after 26.0 seconds 2010-02-25 21:10:34,081 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:09:58 2010-02-25 21:11:04,372 INFO repeater task wh300_beamplots starting 5.0 seconds late 2010-02-25 21:11:10,999 INFO repeater task wh300_beamplots ended after 6.6 seconds 2010-02-25 21:11:11,009 INFO repeater task os75bb_beamplots starting 11.6 seconds late 2010-02-25 21:11:15,696 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:11:15,697 INFO repeater task os75nb_beamplots starting 16.3 seconds late 2010-02-25 21:11:20,356 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 21:11:20,357 INFO repeater task wh300_beamplots starting 21.0 seconds late 2010-02-25 21:11:42,348 INFO repeater task wh300_beamplots ended after 22.0 seconds 2010-02-25 21:11:42,357 INFO repeater task os75bb_beamplots starting 43.0 seconds late 2010-02-25 21:12:05,457 INFO repeater task os75bb_beamplots ended after 23.1 seconds 2010-02-25 21:12:05,471 INFO repeater task os75nb_beamplots starting 66.1 seconds late 2010-02-25 21:12:30,656 INFO repeater task os75nb_beamplots ended after 25.2 seconds 2010-02-25 21:12:30,658 INFO repeater task posmv_hcorrstats starting 91.3 seconds late 2010-02-25 21:12:53,106 INFO repeater task posmv_hcorrstats ended after 22.4 seconds 2010-02-25 21:12:53,118 INFO repeater task phins_hcorrstats starting 113.7 seconds late 2010-02-25 21:13:12,754 INFO repeater task phins_hcorrstats ended after 19.6 seconds 2010-02-25 21:13:17,755 INFO repeater task wh300_lastensq starting 69.8 seconds late 2010-02-25 21:13:43,913 INFO repeater task wh300_lastensq ended after 26.2 seconds 2010-02-25 21:13:43,913 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:11:58 2010-02-25 21:13:43,928 INFO repeater task os75bb_lastensq starting 58.9 seconds late 2010-02-25 21:14:07,778 INFO repeater task os75bb_lastensq ended after 23.9 seconds 2010-02-25 21:14:07,779 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:12:37 2010-02-25 21:14:07,817 INFO repeater task os75nb_lastensq starting 117.8 seconds late 2010-02-25 21:14:31,041 INFO repeater task os75nb_lastensq ended after 23.2 seconds 2010-02-25 21:14:31,042 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:12:02 2010-02-25 21:14:31,043 INFO repeater task wh300_beamplots starting 91.7 seconds late 2010-02-25 21:14:35,404 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 21:14:35,405 INFO repeater task os75bb_beamplots starting 96.0 seconds late 2010-02-25 21:14:39,922 INFO repeater task os75bb_beamplots ended after 4.5 seconds 2010-02-25 21:14:39,930 INFO repeater task os75nb_beamplots starting 100.6 seconds late 2010-02-25 21:14:45,613 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2010-02-25 21:14:50,694 INFO repeater task wh300_lastensq starting 42.7 seconds late 2010-02-25 21:15:17,415 INFO repeater task wh300_lastensq ended after 26.7 seconds 2010-02-25 21:15:17,416 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:13:59 2010-02-25 21:15:17,463 INFO repeater task wh300_beamplots starting 18.1 seconds late 2010-02-25 21:15:21,951 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 21:15:21,952 INFO repeater task os75bb_beamplots starting 22.6 seconds late 2010-02-25 21:15:26,583 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 21:15:26,591 INFO repeater task os75nb_beamplots starting 27.2 seconds late 2010-02-25 21:15:31,176 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 21:16:01,363 INFO repeater task posmv_hcorrstats starting 2.0 seconds late 2010-02-25 21:16:26,481 INFO repeater task posmv_hcorrstats ended after 25.1 seconds 2010-02-25 21:16:26,491 INFO repeater task phins_hcorrstats starting 27.1 seconds late 2010-02-25 21:16:45,820 INFO repeater task phins_hcorrstats ended after 19.3 seconds 2010-02-25 21:16:50,819 INFO repeater task wh300_lastensq starting 41.8 seconds late 2010-02-25 21:17:17,593 INFO repeater task wh300_lastensq ended after 26.8 seconds 2010-02-25 21:17:17,594 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:15:59 2010-02-25 21:17:17,646 INFO repeater task os75nb_lastensq starting 5.6 seconds late 2010-02-25 21:17:38,720 INFO repeater task os75nb_lastensq ended after 21.1 seconds 2010-02-25 21:17:38,721 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:17:04 2010-02-25 21:17:38,723 INFO repeater task wh300_beamplots starting 39.4 seconds late 2010-02-25 21:17:43,011 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 21:17:43,012 INFO repeater task os75bb_beamplots starting 43.6 seconds late 2010-02-25 21:17:48,990 INFO repeater task os75bb_beamplots ended after 6.0 seconds 2010-02-25 21:17:48,991 INFO repeater task os75nb_beamplots starting 49.6 seconds late 2010-02-25 21:17:53,441 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:17:58,518 INFO repeater task os75bb_lastensq starting 11.5 seconds late 2010-02-25 21:18:25,131 INFO repeater task os75bb_lastensq ended after 26.6 seconds 2010-02-25 21:18:25,131 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:17:38 2010-02-25 21:18:30,158 INFO repeater task wh300_lastensq starting 21.2 seconds late 2010-02-25 21:18:56,681 INFO repeater task wh300_lastensq ended after 26.5 seconds 2010-02-25 21:18:56,681 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:17:59 2010-02-25 21:19:01,840 INFO repeater task wh300_beamplots starting 2.5 seconds late 2010-02-25 21:19:06,260 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 21:19:06,261 INFO repeater task os75bb_beamplots starting 6.9 seconds late 2010-02-25 21:19:10,891 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 21:19:10,892 INFO repeater task os75nb_beamplots starting 11.5 seconds late 2010-02-25 21:19:15,461 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 21:20:11,227 INFO repeater task wh300_lastensq starting 2.2 seconds late 2010-02-25 21:20:38,687 INFO repeater task wh300_lastensq ended after 27.5 seconds 2010-02-25 21:20:38,688 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:20:00 2010-02-25 21:21:03,863 INFO repeater task wh300_quick starting 4.5 seconds late 2010-02-25 21:21:43,765 INFO repeater task wh300_quick ended after 39.9 seconds 2010-02-25 21:21:43,766 INFO repeater task wh300_3dayplots starting 44.4 seconds late 2010-02-25 21:22:01,943 INFO repeater task wh300_3dayplots ended after 18.2 seconds 2010-02-25 21:22:01,944 INFO repeater task os75bb_quick starting 62.6 seconds late 2010-02-25 21:22:39,416 INFO repeater task os75bb_quick ended after 37.5 seconds 2010-02-25 21:22:39,427 INFO repeater task os75bb_3dayplots starting 100.1 seconds late 2010-02-25 21:22:58,455 INFO repeater task os75bb_3dayplots output: zgrid: Warning: solution did not converge. 2010-02-25 21:22:58,456 INFO repeater task os75bb_3dayplots ended after 19.0 seconds 2010-02-25 21:22:58,457 INFO repeater task os75nb_lastensq starting 44.5 seconds late 2010-02-25 21:23:19,654 INFO repeater task os75nb_lastensq ended after 21.2 seconds 2010-02-25 21:23:19,654 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:22:05 2010-02-25 21:23:19,655 INFO repeater task os75nb_quick starting 140.3 seconds late 2010-02-25 21:23:53,562 INFO repeater task os75nb_quick ended after 33.9 seconds 2010-02-25 21:23:53,571 INFO repeater task os75nb_3dayplots starting 174.2 seconds late 2010-02-25 21:24:12,694 INFO repeater task os75nb_3dayplots ended after 19.1 seconds 2010-02-25 21:24:12,697 INFO repeater task wh300_beamplots starting 193.3 seconds late 2010-02-25 21:24:17,134 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 21:24:17,135 INFO repeater Extra increment added to time_to_run 2010-02-25 21:24:17,136 INFO repeater task os75bb_beamplots starting 197.8 seconds late 2010-02-25 21:24:22,497 INFO repeater task os75bb_beamplots ended after 5.4 seconds 2010-02-25 21:24:22,498 INFO repeater Extra increment added to time_to_run 2010-02-25 21:24:22,499 INFO repeater task os75nb_beamplots starting 203.1 seconds late 2010-02-25 21:24:28,837 INFO repeater task os75nb_beamplots ended after 6.3 seconds 2010-02-25 21:24:28,837 INFO repeater Extra increment added to time_to_run 2010-02-25 21:24:28,838 INFO repeater task wh300_beamplots starting 209.5 seconds late 2010-02-25 21:24:49,002 INFO repeater task wh300_beamplots ended after 20.2 seconds 2010-02-25 21:24:49,015 INFO repeater task os75bb_beamplots starting 229.6 seconds late 2010-02-25 21:25:12,175 INFO repeater task os75bb_beamplots ended after 23.2 seconds 2010-02-25 21:25:12,176 INFO repeater task os75nb_beamplots starting 252.8 seconds late 2010-02-25 21:25:35,231 INFO repeater task os75nb_beamplots ended after 23.1 seconds 2010-02-25 21:25:35,279 INFO repeater task posmv_hcorrstats starting 275.9 seconds late 2010-02-25 21:25:58,086 INFO repeater task posmv_hcorrstats ended after 22.8 seconds 2010-02-25 21:25:58,099 INFO repeater task phins_hcorrstats starting 298.7 seconds late 2010-02-25 21:26:18,884 INFO repeater task phins_hcorrstats ended after 20.8 seconds 2010-02-25 21:26:18,885 INFO repeater Extra increment added to time_to_run 2010-02-25 21:26:23,883 INFO repeater task wh300_lastensq starting 253.9 seconds late 2010-02-25 21:26:56,073 INFO repeater task wh300_lastensq ended after 32.2 seconds 2010-02-25 21:26:56,074 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:26:01 2010-02-25 21:26:56,076 INFO repeater task os75bb_lastensq starting 248.1 seconds late 2010-02-25 21:27:18,088 INFO repeater task os75bb_lastensq ended after 22.0 seconds 2010-02-25 21:27:18,089 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:22:40 2010-02-25 21:27:18,091 INFO repeater task os75nb_lastensq starting 3.1 seconds late 2010-02-25 21:27:41,109 INFO repeater task os75nb_lastensq ended after 23.0 seconds 2010-02-25 21:27:41,110 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:27:07 2010-02-25 21:27:41,112 INFO repeater task wh300_beamplots starting 161.7 seconds late 2010-02-25 21:27:45,737 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:27:45,738 INFO repeater Extra increment added to time_to_run 2010-02-25 21:27:45,739 INFO repeater task os75bb_beamplots starting 166.4 seconds late 2010-02-25 21:27:50,457 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:27:50,458 INFO repeater Extra increment added to time_to_run 2010-02-25 21:27:50,459 INFO repeater task os75nb_beamplots starting 171.1 seconds late 2010-02-25 21:27:55,133 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 21:27:55,134 INFO repeater Extra increment added to time_to_run 2010-02-25 21:27:55,147 INFO repeater task posmv_hcorrstats starting 115.8 seconds late 2010-02-25 21:28:19,864 INFO repeater task posmv_hcorrstats ended after 24.7 seconds 2010-02-25 21:28:24,871 INFO repeater task wh300_lastensq starting 13.9 seconds late 2010-02-25 21:28:51,793 INFO repeater task wh300_lastensq ended after 26.9 seconds 2010-02-25 21:28:51,794 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:28:01 2010-02-25 21:28:51,833 INFO repeater task os75bb_lastensq starting 61.8 seconds late 2010-02-25 21:29:14,808 INFO repeater task os75bb_lastensq ended after 23.0 seconds 2010-02-25 21:29:14,809 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:27:42 2010-02-25 21:29:14,812 INFO repeater task wh300_beamplots starting 15.4 seconds late 2010-02-25 21:29:19,371 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:29:19,372 INFO repeater task os75bb_beamplots starting 20.0 seconds late 2010-02-25 21:29:24,026 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:29:24,034 INFO repeater task os75nb_beamplots starting 24.7 seconds late 2010-02-25 21:29:28,579 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:30:13,775 INFO repeater task wh300_lastensq starting 2.8 seconds late 2010-02-25 21:30:41,866 INFO repeater task wh300_lastensq ended after 28.1 seconds 2010-02-25 21:30:41,867 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:30:02 2010-02-25 21:31:02,259 INFO repeater task wh300_beamplots starting 2.9 seconds late 2010-02-25 21:31:06,655 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 21:31:06,662 INFO repeater task os75bb_beamplots starting 7.3 seconds late 2010-02-25 21:31:12,356 INFO repeater task os75bb_beamplots ended after 5.7 seconds 2010-02-25 21:31:12,357 INFO repeater task os75nb_beamplots starting 13.0 seconds late 2010-02-25 21:31:16,871 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:31:16,872 INFO repeater task wh300_beamplots starting 17.5 seconds late 2010-02-25 21:31:38,214 INFO repeater task wh300_beamplots ended after 21.3 seconds 2010-02-25 21:31:38,222 INFO repeater task os75bb_beamplots starting 38.9 seconds late 2010-02-25 21:32:00,944 INFO repeater task os75bb_beamplots ended after 22.7 seconds 2010-02-25 21:32:00,945 INFO repeater task os75nb_beamplots starting 61.6 seconds late 2010-02-25 21:32:26,507 INFO repeater task os75nb_beamplots ended after 25.6 seconds 2010-02-25 21:32:26,509 INFO repeater task posmv_hcorrstats starting 87.1 seconds late 2010-02-25 21:32:50,428 INFO repeater task posmv_hcorrstats ended after 23.9 seconds 2010-02-25 21:32:50,429 INFO repeater task phins_hcorrstats starting 111.1 seconds late 2010-02-25 21:33:08,635 INFO repeater task phins_hcorrstats ended after 18.2 seconds 2010-02-25 21:33:13,643 INFO repeater task wh300_lastensq starting 61.6 seconds late 2010-02-25 21:33:40,143 INFO repeater task wh300_lastensq ended after 26.5 seconds 2010-02-25 21:33:40,143 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:32:02 2010-02-25 21:33:40,189 INFO repeater task os75bb_lastensq starting 48.2 seconds late 2010-02-25 21:34:02,862 INFO repeater task os75bb_lastensq ended after 22.7 seconds 2010-02-25 21:34:02,863 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:32:43 2010-02-25 21:34:02,887 INFO repeater task os75nb_lastensq starting 105.9 seconds late 2010-02-25 21:34:27,402 INFO repeater task os75nb_lastensq ended after 24.5 seconds 2010-02-25 21:34:27,403 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:32:09 2010-02-25 21:34:27,412 INFO repeater task wh300_beamplots starting 88.0 seconds late 2010-02-25 21:34:31,675 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 21:34:31,676 INFO repeater task os75bb_beamplots starting 92.3 seconds late 2010-02-25 21:34:36,485 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 21:34:36,485 INFO repeater task os75nb_beamplots starting 97.1 seconds late 2010-02-25 21:34:42,049 INFO repeater task os75nb_beamplots ended after 5.6 seconds 2010-02-25 21:34:47,058 INFO repeater task wh300_lastensq starting 35.1 seconds late 2010-02-25 21:35:13,682 INFO repeater task wh300_lastensq ended after 26.6 seconds 2010-02-25 21:35:13,683 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:34:02 2010-02-25 21:35:13,764 INFO repeater task wh300_beamplots starting 14.4 seconds late 2010-02-25 21:35:17,936 INFO repeater task wh300_beamplots ended after 4.2 seconds 2010-02-25 21:35:17,937 INFO repeater task os75bb_beamplots starting 18.6 seconds late 2010-02-25 21:35:22,574 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 21:35:22,575 INFO repeater task os75nb_beamplots starting 23.2 seconds late 2010-02-25 21:35:27,024 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2010-02-25 21:36:02,737 INFO repeater task posmv_hcorrstats starting 3.4 seconds late 2010-02-25 21:36:28,743 INFO repeater task posmv_hcorrstats ended after 26.0 seconds 2010-02-25 21:36:28,744 INFO repeater task phins_hcorrstats starting 29.4 seconds late 2010-02-25 21:36:48,964 INFO repeater task phins_hcorrstats ended after 20.2 seconds 2010-02-25 21:36:53,963 INFO repeater task wh300_lastensq starting 42.0 seconds late 2010-02-25 21:37:20,604 INFO repeater task wh300_lastensq ended after 26.6 seconds 2010-02-25 21:37:20,605 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:36:03 2010-02-25 21:37:20,608 INFO repeater task os75nb_lastensq starting 1.6 seconds late 2010-02-25 21:37:42,153 INFO repeater task os75nb_lastensq ended after 21.5 seconds 2010-02-25 21:37:42,153 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:37:10 2010-02-25 21:37:42,179 INFO repeater task wh300_beamplots starting 42.8 seconds late 2010-02-25 21:37:48,709 INFO repeater task wh300_beamplots ended after 6.5 seconds 2010-02-25 21:37:48,710 INFO repeater task os75bb_beamplots starting 49.3 seconds late 2010-02-25 21:37:53,423 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:37:53,433 INFO repeater task os75nb_beamplots starting 54.1 seconds late 2010-02-25 21:37:57,692 INFO repeater task os75nb_beamplots ended after 4.3 seconds 2010-02-25 21:38:02,760 INFO repeater task os75bb_lastensq starting 9.8 seconds late 2010-02-25 21:38:27,774 INFO repeater task os75bb_lastensq ended after 25.0 seconds 2010-02-25 21:38:27,776 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:37:45 2010-02-25 21:38:32,791 INFO repeater task wh300_lastensq starting 19.8 seconds late 2010-02-25 21:39:00,513 INFO repeater task wh300_lastensq ended after 27.7 seconds 2010-02-25 21:39:00,514 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:38:03 2010-02-25 21:39:00,518 INFO repeater task wh300_beamplots starting 1.1 seconds late 2010-02-25 21:39:05,091 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:39:05,092 INFO repeater task os75bb_beamplots starting 5.7 seconds late 2010-02-25 21:39:09,727 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 21:39:09,736 INFO repeater task os75nb_beamplots starting 10.4 seconds late 2010-02-25 21:39:14,074 INFO repeater task os75nb_beamplots ended after 4.3 seconds 2010-02-25 21:40:15,234 INFO repeater task wh300_lastensq starting 2.2 seconds late 2010-02-25 21:40:43,316 INFO repeater task wh300_lastensq ended after 28.1 seconds 2010-02-25 21:40:43,317 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:40:03 2010-02-25 21:41:03,338 INFO repeater task wh300_beamplots starting 4.0 seconds late 2010-02-25 21:41:07,981 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:41:07,982 INFO repeater task os75bb_beamplots starting 8.6 seconds late 2010-02-25 21:41:12,722 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:41:12,723 INFO repeater task os75nb_beamplots starting 13.4 seconds late 2010-02-25 21:41:17,130 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2010-02-25 21:41:17,141 INFO repeater task wh300_beamplots starting 17.8 seconds late 2010-02-25 21:41:39,004 INFO repeater task wh300_beamplots ended after 21.9 seconds 2010-02-25 21:41:39,005 INFO repeater task os75bb_beamplots starting 39.6 seconds late 2010-02-25 21:42:02,077 INFO repeater task os75bb_beamplots ended after 23.1 seconds 2010-02-25 21:42:02,087 INFO repeater task os75nb_beamplots starting 62.7 seconds late 2010-02-25 21:42:27,852 INFO repeater task os75nb_beamplots ended after 25.8 seconds 2010-02-25 21:42:27,854 INFO repeater task posmv_hcorrstats starting 88.5 seconds late 2010-02-25 21:42:49,824 INFO repeater task posmv_hcorrstats ended after 22.0 seconds 2010-02-25 21:42:49,825 INFO repeater task phins_hcorrstats starting 110.4 seconds late 2010-02-25 21:43:09,475 INFO repeater task phins_hcorrstats ended after 19.7 seconds 2010-02-25 21:43:14,486 INFO repeater task wh300_lastensq starting 61.5 seconds late 2010-02-25 21:43:41,427 INFO repeater task wh300_lastensq ended after 26.9 seconds 2010-02-25 21:43:41,427 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:42:04 2010-02-25 21:43:41,447 INFO repeater task os75bb_lastensq starting 46.4 seconds late 2010-02-25 21:44:04,904 INFO repeater task os75bb_lastensq ended after 23.5 seconds 2010-02-25 21:44:04,904 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:42:47 2010-02-25 21:44:04,906 INFO repeater task os75nb_lastensq starting 104.9 seconds late 2010-02-25 21:44:30,323 INFO repeater task os75nb_lastensq ended after 25.4 seconds 2010-02-25 21:44:30,324 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:42:12 2010-02-25 21:44:30,350 INFO repeater task wh300_beamplots starting 91.0 seconds late 2010-02-25 21:44:34,946 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:44:34,958 INFO repeater task os75bb_beamplots starting 95.6 seconds late 2010-02-25 21:44:39,634 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:44:39,635 INFO repeater task os75nb_beamplots starting 100.3 seconds late 2010-02-25 21:44:44,100 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:44:49,103 INFO repeater task wh300_lastensq starting 35.1 seconds late 2010-02-25 21:45:16,507 INFO repeater task wh300_lastensq ended after 27.4 seconds 2010-02-25 21:45:16,508 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:44:04 2010-02-25 21:45:16,512 INFO repeater task wh300_beamplots starting 17.1 seconds late 2010-02-25 21:45:21,119 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:45:21,120 INFO repeater task os75bb_beamplots starting 21.8 seconds late 2010-02-25 21:45:26,887 INFO repeater task os75bb_beamplots ended after 5.8 seconds 2010-02-25 21:45:26,888 INFO repeater task os75nb_beamplots starting 27.5 seconds late 2010-02-25 21:45:31,419 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:46:01,882 INFO repeater task posmv_hcorrstats starting 2.5 seconds late 2010-02-25 21:46:24,387 INFO repeater task posmv_hcorrstats ended after 22.5 seconds 2010-02-25 21:46:24,399 INFO repeater task phins_hcorrstats starting 25.0 seconds late 2010-02-25 21:46:45,924 INFO repeater task phins_hcorrstats ended after 21.5 seconds 2010-02-25 21:46:50,926 INFO repeater task wh300_lastensq starting 36.9 seconds late 2010-02-25 21:47:18,938 INFO repeater task wh300_lastensq ended after 28.0 seconds 2010-02-25 21:47:18,939 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:46:04 2010-02-25 21:47:18,969 INFO repeater task wh300_beamplots starting 19.6 seconds late 2010-02-25 21:47:23,522 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:47:23,523 INFO repeater task os75bb_beamplots starting 24.2 seconds late 2010-02-25 21:47:29,274 INFO repeater task os75bb_beamplots ended after 5.8 seconds 2010-02-25 21:47:29,275 INFO repeater task os75nb_beamplots starting 29.9 seconds late 2010-02-25 21:47:33,808 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:47:38,909 INFO repeater task os75nb_lastensq starting 16.9 seconds late 2010-02-25 21:48:02,424 INFO repeater task os75nb_lastensq ended after 23.5 seconds 2010-02-25 21:48:02,425 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:47:14 2010-02-25 21:48:07,438 INFO repeater task os75bb_lastensq starting 10.4 seconds late 2010-02-25 21:48:32,742 INFO repeater task os75bb_lastensq ended after 25.3 seconds 2010-02-25 21:48:32,742 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 21:47:48 2010-02-25 21:48:37,943 INFO repeater task wh300_lastensq starting 23.9 seconds late 2010-02-25 21:49:04,914 INFO repeater task wh300_lastensq ended after 27.0 seconds 2010-02-25 21:49:04,915 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:48:05 2010-02-25 21:49:04,992 INFO repeater task wh300_beamplots starting 5.6 seconds late 2010-02-25 21:49:09,616 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-25 21:49:09,617 INFO repeater task os75bb_beamplots starting 10.2 seconds late 2010-02-25 21:49:14,423 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 21:49:14,424 INFO repeater task os75nb_beamplots starting 15.1 seconds late 2010-02-25 21:49:18,875 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 21:50:19,270 INFO repeater task wh300_lastensq starting 4.3 seconds late 2010-02-25 21:50:52,281 INFO repeater task wh300_lastensq ended after 33.0 seconds 2010-02-25 21:50:52,282 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 21:50:05 2010-02-25 21:51:02,576 INFO repeater task wh300_quick starting 3.2 seconds late 2010-02-25 21:51:41,044 INFO repeater task wh300_quick ended after 38.5 seconds 2010-02-25 21:51:41,059 INFO repeater task wh300_3dayplots starting 41.7 seconds late 2010-02-25 21:51:58,115 INFO repeater task wh300_3dayplots ended after 17.1 seconds 2010-02-25 21:51:58,125 INFO repeater task os75bb_quick starting 58.8 seconds late 2010-02-25 21:52:34,721 INFO repeater task os75bb_quick ended after 36.6 seconds 2010-02-25 21:52:34,722 INFO repeater task os75bb_3dayplots starting 95.4 seconds late 2010-02-25 21:52:54,498 INFO repeater task os75bb_3dayplots ended after 19.8 seconds 2010-02-25 21:52:54,499 INFO repeater task os75nb_lastensq starting 30.5 seconds late 2010-02-25 21:53:18,466 INFO repeater task os75nb_lastensq ended after 24.0 seconds 2010-02-25 21:53:18,467 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 21:52:16 2010-02-25 21:53:18,475 INFO repeater task os75nb_quick starting 139.1 seconds late 2010-02-25 21:53:52,364 INFO repeater task os75nb_quick ended after 33.9 seconds 2010-02-25 21:53:52,375 INFO repeater task os75nb_3dayplots starting 173.0 seconds late 2010-02-25 21:54:11,362 INFO repeater task os75nb_3dayplots output: zgrid: Warning: solution did not converge. 2010-02-25 21:54:11,363 INFO repeater task os75nb_3dayplots ended after 19.0 seconds 2010-02-25 21:54:11,364 INFO repeater task wh300_beamplots starting 192.0 seconds late 2010-02-25 21:54:15,866 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-25 21:54:15,866 INFO repeater Extra increment added to time_to_run 2010-02-25 21:54:15,867 INFO repeater task os75bb_beamplots starting 196.5 seconds late 2010-02-25 21:54:20,574 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 21:54:20,575 INFO repeater Extra increment added to time_to_run 2010-02-25 21:54:20,585 INFO repeater task os75nb_beamplots starting 201.2 seconds late 2010-02-25 21:54:24,863 INFO repeater task os75nb_beamplots ended after 4.3 seconds 2010-02-25 21:54:24,863 INFO repeater Extra increment added to time_to_run 2010-02-25 21:54:24,872 INFO repeater task wh300_beamplots starting 205.5 seconds late 2010-02-25 21:54:48,534 INFO repeater task wh300_beamplots ended after 23.7 seconds 2010-02-25 21:54:48,535 INFO repeater task os75bb_beamplots starting 229.2 seconds late 2010-02-25 21:55:13,232 INFO repeater task os75bb_beamplots ended after 24.7 seconds 2010-02-25 21:55:13,233 INFO repeater task os75nb_beamplots starting 253.9 seconds late 2010-02-25 21:55:35,593 INFO repeater task os75nb_beamplots ended after 22.4 seconds 2010-02-25 21:55:35,606 INFO repeater task wh300_beamhistory starting 276.2 seconds late 2010-02-25 21:58:27,668 INFO repeater task wh300_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to wh300_bin_history.png 2010-02-25 21:58:27,675 INFO repeater task wh300_beamhistory ended after 172.1 seconds 2010-02-25 21:58:27,676 INFO repeater task os75bb_beamhistory starting 448.3 seconds late 2010-02-25 22:00:07,710 INFO repeater task os75bb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75bb_bin_history.png 2010-02-25 22:00:07,711 INFO repeater task os75bb_beamhistory ended after 100.0 seconds 2010-02-25 22:00:07,712 INFO repeater task os75nb_beamhistory starting 548.3 seconds late 2010-02-25 22:01:17,633 INFO repeater task os75nb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75nb_bin_history.png 2010-02-25 22:01:17,633 INFO repeater task os75nb_beamhistory ended after 69.9 seconds 2010-02-25 22:01:17,645 INFO repeater task posmv_hcorrstats starting 618.3 seconds late 2010-02-25 22:01:39,185 INFO repeater task posmv_hcorrstats ended after 21.5 seconds 2010-02-25 22:01:39,186 INFO repeater Extra increment added to time_to_run 2010-02-25 22:01:39,186 INFO repeater Extra increment added to time_to_run 2010-02-25 22:01:39,199 INFO repeater task phins_hcorrstats starting 639.8 seconds late 2010-02-25 22:01:58,528 INFO repeater task phins_hcorrstats ended after 19.3 seconds 2010-02-25 22:01:58,528 INFO repeater Extra increment added to time_to_run 2010-02-25 22:01:58,528 INFO repeater Extra increment added to time_to_run 2010-02-25 22:02:03,527 INFO repeater task wh300_lastensq starting 588.5 seconds late 2010-02-25 22:02:47,031 INFO repeater task wh300_lastensq ended after 43.5 seconds 2010-02-25 22:02:47,032 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:02:07 2010-02-25 22:02:47,058 INFO repeater task os75bb_lastensq starting 589.1 seconds late 2010-02-25 22:03:18,571 INFO repeater task os75bb_lastensq ended after 31.5 seconds 2010-02-25 22:03:18,572 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 22:02:49 2010-02-25 22:03:18,573 INFO repeater task os75nb_lastensq starting 352.6 seconds late 2010-02-25 22:03:41,888 INFO repeater task os75nb_lastensq ended after 23.3 seconds 2010-02-25 22:03:41,889 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 22:02:19 2010-02-25 22:03:41,890 INFO repeater task wh300_beamplots starting 522.5 seconds late 2010-02-25 22:03:47,422 INFO repeater task wh300_beamplots ended after 5.5 seconds 2010-02-25 22:03:47,423 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:47,423 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:47,424 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:47,424 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:47,431 INFO repeater task os75bb_beamplots starting 528.1 seconds late 2010-02-25 22:03:52,156 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 22:03:52,157 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:52,157 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:52,157 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:52,157 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:52,158 INFO repeater task os75nb_beamplots starting 532.8 seconds late 2010-02-25 22:03:56,901 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 22:03:56,902 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:56,902 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:56,902 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:56,902 INFO repeater Extra increment added to time_to_run 2010-02-25 22:03:56,903 INFO repeater task wh300_beamplots starting 177.5 seconds late 2010-02-25 22:04:19,261 INFO repeater task wh300_beamplots ended after 22.4 seconds 2010-02-25 22:04:19,262 INFO repeater task os75bb_beamplots starting 199.9 seconds late 2010-02-25 22:04:42,062 INFO repeater task os75bb_beamplots ended after 22.8 seconds 2010-02-25 22:04:42,066 INFO repeater task os75nb_beamplots starting 222.7 seconds late 2010-02-25 22:05:06,799 INFO repeater task os75nb_beamplots ended after 24.7 seconds 2010-02-25 22:05:11,805 INFO repeater task wh300_lastensq starting 54.8 seconds late 2010-02-25 22:05:37,105 INFO repeater task wh300_lastensq ended after 25.3 seconds 2010-02-25 22:05:37,106 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:04:08 2010-02-25 22:05:37,179 INFO repeater task wh300_beamplots starting 37.8 seconds late 2010-02-25 22:05:41,353 INFO repeater task wh300_beamplots ended after 4.2 seconds 2010-02-25 22:05:41,360 INFO repeater task os75bb_beamplots starting 42.0 seconds late 2010-02-25 22:05:45,945 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 22:05:45,953 INFO repeater task os75nb_beamplots starting 46.6 seconds late 2010-02-25 22:05:51,658 INFO repeater task os75nb_beamplots ended after 5.7 seconds 2010-02-25 22:06:01,960 INFO repeater task posmv_hcorrstats starting 2.6 seconds late 2010-02-25 22:06:24,414 INFO repeater task posmv_hcorrstats ended after 22.5 seconds 2010-02-25 22:06:24,416 INFO repeater task phins_hcorrstats starting 25.0 seconds late 2010-02-25 22:06:43,221 INFO repeater task phins_hcorrstats ended after 18.8 seconds 2010-02-25 22:06:48,237 INFO repeater task wh300_lastensq starting 30.2 seconds late 2010-02-25 22:07:13,088 INFO repeater task wh300_lastensq ended after 24.9 seconds 2010-02-25 22:07:13,089 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:06:08 2010-02-25 22:07:13,158 INFO repeater task wh300_beamplots starting 13.8 seconds late 2010-02-25 22:07:17,446 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 22:07:17,454 INFO repeater task os75bb_beamplots starting 18.1 seconds late 2010-02-25 22:07:23,322 INFO repeater task os75bb_beamplots ended after 5.9 seconds 2010-02-25 22:07:23,323 INFO repeater task os75nb_beamplots starting 24.0 seconds late 2010-02-25 22:07:27,848 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 22:07:32,973 INFO repeater task os75nb_lastensq starting 4.0 seconds late 2010-02-25 22:07:54,224 INFO repeater task os75nb_lastensq ended after 21.3 seconds 2010-02-25 22:07:54,224 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 22:07:21 2010-02-25 22:07:59,243 INFO repeater task os75bb_lastensq starting 0.2 seconds late 2010-02-25 22:08:20,924 INFO repeater task os75bb_lastensq ended after 21.7 seconds 2010-02-25 22:08:20,925 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 22:07:55 2010-02-25 22:08:26,154 INFO repeater task wh300_lastensq starting 8.2 seconds late 2010-02-25 22:08:53,980 INFO repeater task wh300_lastensq ended after 27.8 seconds 2010-02-25 22:08:53,981 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:08:09 2010-02-25 22:09:03,998 INFO repeater task wh300_beamplots starting 4.6 seconds late 2010-02-25 22:09:08,344 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 22:09:08,345 INFO repeater task os75bb_beamplots starting 9.0 seconds late 2010-02-25 22:09:13,085 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 22:09:13,086 INFO repeater task os75nb_beamplots starting 13.7 seconds late 2010-02-25 22:09:17,575 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 22:10:23,086 INFO repeater task wh300_lastensq starting 4.1 seconds late 2010-02-25 22:10:48,455 INFO repeater task wh300_lastensq ended after 25.4 seconds 2010-02-25 22:10:48,456 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:10:09 2010-02-25 22:11:03,825 INFO repeater task wh300_beamplots starting 4.5 seconds late 2010-02-25 22:11:08,222 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-02-25 22:11:08,223 INFO repeater task os75bb_beamplots starting 8.9 seconds late 2010-02-25 22:11:12,963 INFO repeater task os75bb_beamplots ended after 4.7 seconds 2010-02-25 22:11:12,976 INFO repeater task os75nb_beamplots starting 13.6 seconds late 2010-02-25 22:11:17,597 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-25 22:11:17,598 INFO repeater task wh300_beamplots starting 18.2 seconds late 2010-02-25 22:11:40,481 INFO repeater task wh300_beamplots ended after 22.9 seconds 2010-02-25 22:11:40,482 INFO repeater task os75bb_beamplots starting 41.1 seconds late 2010-02-25 22:12:03,822 INFO repeater task os75bb_beamplots ended after 23.3 seconds 2010-02-25 22:12:03,823 INFO repeater task os75nb_beamplots starting 64.5 seconds late 2010-02-25 22:12:26,874 INFO repeater task os75nb_beamplots ended after 23.1 seconds 2010-02-25 22:12:26,937 INFO repeater task posmv_hcorrstats starting 87.6 seconds late 2010-02-25 22:12:50,888 INFO repeater task posmv_hcorrstats ended after 24.0 seconds 2010-02-25 22:12:50,898 INFO repeater task phins_hcorrstats starting 111.5 seconds late 2010-02-25 22:13:10,367 INFO repeater task phins_hcorrstats ended after 19.5 seconds 2010-02-25 22:13:15,367 INFO repeater task wh300_lastensq starting 56.4 seconds late 2010-02-25 22:13:41,216 INFO repeater task wh300_lastensq ended after 25.8 seconds 2010-02-25 22:13:41,217 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:12:09 2010-02-25 22:13:41,218 INFO repeater task os75bb_lastensq starting 36.2 seconds late 2010-02-25 22:14:03,545 INFO repeater task os75bb_lastensq ended after 22.3 seconds 2010-02-25 22:14:03,546 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 22:12:57 2010-02-25 22:14:03,548 INFO repeater task os75nb_lastensq starting 92.5 seconds late 2010-02-25 22:14:23,654 INFO repeater task os75nb_lastensq ended after 20.1 seconds 2010-02-25 22:14:23,654 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 22:12:22 2010-02-25 22:14:23,685 INFO repeater task wh300_beamplots starting 84.3 seconds late 2010-02-25 22:14:30,350 INFO repeater task wh300_beamplots ended after 6.7 seconds 2010-02-25 22:14:30,359 INFO repeater task os75bb_beamplots starting 91.0 seconds late 2010-02-25 22:14:34,975 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 22:14:34,989 INFO repeater task os75nb_beamplots starting 95.6 seconds late 2010-02-25 22:14:39,505 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-25 22:14:44,507 INFO repeater task wh300_lastensq starting 25.5 seconds late 2010-02-25 22:15:12,823 INFO repeater task wh300_lastensq ended after 28.3 seconds 2010-02-25 22:15:12,824 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:14:10 2010-02-25 22:15:12,844 INFO repeater task wh300_beamplots starting 13.5 seconds late 2010-02-25 22:15:17,130 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 22:15:17,131 INFO repeater task os75bb_beamplots starting 17.8 seconds late 2010-02-25 22:15:21,898 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 22:15:21,908 INFO repeater task os75nb_beamplots starting 22.5 seconds late 2010-02-25 22:15:26,335 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2010-02-25 22:16:01,947 INFO repeater task posmv_hcorrstats starting 2.6 seconds late 2010-02-25 22:16:23,140 INFO repeater task posmv_hcorrstats ended after 21.2 seconds 2010-02-25 22:16:23,141 INFO repeater task phins_hcorrstats starting 23.8 seconds late 2010-02-25 22:16:42,582 INFO repeater task phins_hcorrstats ended after 19.4 seconds 2010-02-25 22:16:47,583 INFO repeater task wh300_lastensq starting 27.6 seconds late 2010-02-25 22:17:15,269 INFO repeater task wh300_lastensq ended after 27.7 seconds 2010-02-25 22:17:15,270 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:16:10 2010-02-25 22:17:15,273 INFO repeater task wh300_beamplots starting 15.9 seconds late 2010-02-25 22:17:19,564 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-02-25 22:17:19,565 INFO repeater task os75bb_beamplots starting 20.2 seconds late 2010-02-25 22:17:24,394 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-25 22:17:24,395 INFO repeater task os75nb_beamplots starting 25.0 seconds late 2010-02-25 22:17:29,063 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-02-25 22:17:34,085 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2010-02-25 22:17:54,927 INFO repeater task os75nb_lastensq ended after 20.8 seconds 2010-02-25 22:17:54,928 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 22:17:24 2010-02-25 22:18:09,948 INFO repeater task os75bb_lastensq starting 2.9 seconds late 2010-02-25 22:18:32,090 INFO repeater task os75bb_lastensq ended after 22.1 seconds 2010-02-25 22:18:32,091 INFO repeater Time from /home/data/kn197_4/proc/os75bb/load/wrote_ens_time: 2010/02/25 22:17:58 2010-02-25 22:18:37,171 INFO repeater task wh300_lastensq starting 17.2 seconds late 2010-02-25 22:19:05,613 INFO repeater task wh300_lastensq ended after 28.4 seconds 2010-02-25 22:19:05,614 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:18:11 2010-02-25 22:19:05,694 INFO repeater task wh300_beamplots starting 6.3 seconds late 2010-02-25 22:19:09,909 INFO repeater task wh300_beamplots ended after 4.2 seconds 2010-02-25 22:19:09,918 INFO repeater task os75bb_beamplots starting 10.5 seconds late 2010-02-25 22:19:14,458 INFO repeater task os75bb_beamplots ended after 4.5 seconds 2010-02-25 22:19:14,466 INFO repeater task os75nb_beamplots starting 15.1 seconds late 2010-02-25 22:19:18,882 INFO repeater task os75nb_beamplots ended after 4.4 seconds 2010-02-25 22:20:24,759 INFO repeater task wh300_lastensq starting 3.8 seconds late 2010-02-25 22:20:51,790 INFO repeater task wh300_lastensq ended after 27.0 seconds 2010-02-25 22:20:51,791 INFO repeater Time from /home/data/kn197_4/proc/wh300/load/wrote_ens_time: 2010/02/25 22:20:11 2010-02-25 22:21:02,051 INFO repeater task wh300_quick starting 2.7 seconds late 2010-02-25 22:21:42,725 INFO repeater task wh300_quick ended after 40.7 seconds 2010-02-25 22:21:42,733 INFO repeater task wh300_3dayplots starting 43.4 seconds late 2010-02-25 22:22:25,463 INFO repeater task wh300_3dayplots ended after 42.7 seconds 2010-02-25 22:22:25,465 INFO repeater task os75bb_quick starting 86.1 seconds late 2010-02-25 22:23:03,679 INFO repeater task os75bb_quick ended after 38.2 seconds 2010-02-25 22:23:03,687 INFO repeater task os75bb_3dayplots starting 124.3 seconds late 2010-02-25 22:23:22,348 INFO repeater task os75bb_3dayplots output: zgrid: Warning: solution did not converge. zgrid: Warning: solution did not converge. 2010-02-25 22:23:22,349 INFO repeater task os75bb_3dayplots ended after 18.7 seconds 2010-02-25 22:23:22,350 INFO repeater task os75nb_lastensq starting 48.4 seconds late 2010-02-25 22:23:44,672 INFO repeater task os75nb_lastensq ended after 22.3 seconds 2010-02-25 22:23:44,673 INFO repeater Time from /home/data/kn197_4/proc/os75nb/load/wrote_ens_time: 2010/02/25 22:22:25 2010-02-25 22:23:44,683 INFO repeater task os75nb_quick starting 165.3 seconds late 2010-02-25 22:24:18,364 INFO repeater task os75nb_quick ended after 33.7 seconds 2010-02-25 22:24:18,365 INFO repeater task os75nb_3dayplots starting 199.0 seconds late 2010-02-25 22:24:34,106 INFO repeater task os75nb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-02-25 22:24:34,106 INFO repeater task os75nb_3dayplots ended after 15.7 seconds 2010-02-25 22:24:34,127 INFO repeater task wh300_beamplots starting 214.8 seconds late 2010-02-25 22:24:40,564 INFO repeater task wh300_beamplots ended after 6.4 seconds 2010-02-25 22:24:40,565 INFO repeater Extra increment added to time_to_run 2010-02-25 22:24:40,566 INFO repeater task os75bb_beamplots starting 221.2 seconds late 2010-02-25 22:24:45,172 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-25 22:24:45,173 INFO repeater Extra increment added to time_to_run 2010-02-25 22:24:45,179 INFO repeater task os75nb_beamplots starting 225.8 seconds late 2010-02-25 22:24:50,151 INFO repeater task os75nb_beamplots ended after 5.0 seconds 2010-02-25 22:24:50,151 INFO repeater Extra increment added to time_to_run 2010-02-25 22:24:50,165 INFO repeater task wh300_beamplots starting 230.8 seconds late 2010-02-25 22:25:14,388 INFO repeater task wh300_beamplots ended after 24.2 seconds 2010-02-25 22:25:14,390 INFO repeater task os75bb_beamplots starting 255.0 seconds late 2010-02-25 22:25:35,874 INFO repeater task os75bb_beamplots ended after 21.5 seconds 2010-02-25 22:25:35,879 INFO repeater task os75nb_beamplots starting 276.5 seconds late 2010-02-25 22:25:59,428 INFO repeater task os75nb_beamplots ended after 23.5 seconds 2010-02-25 22:25:59,431 INFO repeater task posmv_hcorrstats starting 300.1 seconds late 2010-02-25 22:26:21,698 INFO repeater task posmv_hcorrstats ended after 22.3 seconds 2010-02-25 22:26:21,699 INFO repeater Extra increment added to time_to_run 2010-02-25 22:26:21,707 INFO repeater task phins_hcorrstats starting 322.3 seconds late 2010-02-25 22:26:40,467 INFO repeater task phins_hcorrstats ended after 18.8 seconds 2010-02-25 22:26:40,468 INFO repeater Extra increment added to time_to_run 2010-02-25 22:26:45,474 INFO repeater task wh300_lastensq starting 264.5 seconds late