2010-02-12 17:19:40,593 INFO repeater Extra increment added to time_to_run 2010-02-12 17:19:40,601 INFO repeater Extra increment added to time_to_run 2010-02-12 17:19:40,602 INFO repeater task wh300_beamplots starting 17.1 seconds late 2010-02-12 17:19:56,949 INFO repeater task wh300_beamplots ended after 16.3 seconds 2010-02-12 17:19:56,950 INFO repeater task os75bb_beamplots starting 33.4 seconds late 2010-02-12 17:20:16,087 INFO repeater task os75bb_beamplots ended after 19.1 seconds 2010-02-12 17:20:16,089 INFO repeater task os75nb_beamplots starting 52.6 seconds late 2010-02-12 17:20:33,317 INFO repeater task os75nb_beamplots ended after 17.2 seconds 2010-02-12 17:20:33,319 INFO repeater task posmv_hcorrstats starting 69.8 seconds late 2010-02-12 17:21:03,480 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:21:03,480 INFO repeater task posmv_hcorrstats ended after 30.2 seconds 2010-02-12 17:21:03,481 INFO repeater task phins_hcorrstats starting 99.9 seconds late 2010-02-12 17:21:24,218 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:21:24,219 INFO repeater task phins_hcorrstats ended after 20.7 seconds 2010-02-12 17:21:29,227 INFO repeater task wh300_lastensq starting 135.2 seconds late 2010-02-12 17:21:52,399 INFO repeater task wh300_lastensq ended after 23.2 seconds 2010-02-12 17:21:52,400 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:21:05 2010-02-12 17:21:52,445 INFO repeater task os75nb_lastensq starting 84.4 seconds late 2010-02-12 17:22:12,632 INFO repeater task os75nb_lastensq ended after 20.2 seconds 2010-02-12 17:22:12,633 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:20:20 2010-02-12 17:22:12,635 INFO repeater task wh300_beamplots starting 49.1 seconds late 2010-02-12 17:22:16,438 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 17:22:16,449 INFO repeater task os75bb_beamplots starting 52.9 seconds late 2010-02-12 17:22:19,928 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 17:22:19,929 INFO repeater task os75nb_beamplots starting 56.4 seconds late 2010-02-12 17:22:23,458 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 17:23:19,087 INFO repeater task wh300_lastensq starting 4.1 seconds late 2010-02-12 17:23:40,287 INFO repeater task wh300_lastensq ended after 21.2 seconds 2010-02-12 17:23:40,288 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:23:05 2010-02-12 17:23:40,292 INFO repeater task wh300_beamplots starting 16.8 seconds late 2010-02-12 17:23:43,543 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 17:23:43,544 INFO repeater task os75bb_beamplots starting 20.0 seconds late 2010-02-12 17:23:47,219 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 17:23:47,221 INFO repeater task os75nb_beamplots starting 23.7 seconds late 2010-02-12 17:23:50,752 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 17:23:55,756 INFO repeater task os75bb_lastensq starting 5.8 seconds late 2010-02-12 17:24:16,114 INFO repeater task os75bb_lastensq ended after 20.4 seconds 2010-02-12 17:24:16,114 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:23:42 2010-02-12 17:24:26,189 INFO repeater task posmv_hcorrstats starting 2.6 seconds late 2010-02-12 17:24:50,910 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:24:50,910 INFO repeater task posmv_hcorrstats ended after 24.7 seconds 2010-02-12 17:24:50,917 INFO repeater task phins_hcorrstats starting 27.4 seconds late 2010-02-12 17:25:12,893 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:25:12,894 INFO repeater task phins_hcorrstats ended after 22.0 seconds 2010-02-12 17:25:17,900 INFO repeater task wh300_lastensq starting 2.9 seconds late 2010-02-12 17:25:39,247 INFO repeater task wh300_lastensq ended after 21.3 seconds 2010-02-12 17:25:39,248 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:25:05 2010-02-12 17:25:39,303 INFO repeater task os75nb_lastensq starting 9.3 seconds late 2010-02-12 17:25:56,556 INFO repeater task os75nb_lastensq ended after 17.3 seconds 2010-02-12 17:25:56,557 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:25:21 2010-02-12 17:25:56,559 INFO repeater task wh300_beamplots starting 33.0 seconds late 2010-02-12 17:26:01,527 INFO repeater task wh300_beamplots ended after 5.0 seconds 2010-02-12 17:26:01,534 INFO repeater task os75bb_beamplots starting 38.0 seconds late 2010-02-12 17:26:07,193 INFO repeater task os75bb_beamplots ended after 5.7 seconds 2010-02-12 17:26:07,194 INFO repeater task os75nb_beamplots starting 43.7 seconds late 2010-02-12 17:26:11,081 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-02-12 17:27:16,455 INFO repeater task wh300_lastensq starting 1.5 seconds late 2010-02-12 17:27:37,488 INFO repeater task wh300_lastensq ended after 21.0 seconds 2010-02-12 17:27:37,489 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:27:05 2010-02-12 17:27:37,502 INFO repeater task wh300_beamplots starting 14.0 seconds late 2010-02-12 17:27:40,926 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 17:27:40,928 INFO repeater task os75bb_beamplots starting 17.4 seconds late 2010-02-12 17:27:44,472 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 17:27:44,475 INFO repeater task os75nb_beamplots starting 20.9 seconds late 2010-02-12 17:27:48,090 INFO repeater task os75nb_beamplots ended after 3.6 seconds 2010-02-12 17:28:53,176 INFO repeater task os75bb_lastensq starting 1.2 seconds late 2010-02-12 17:29:11,968 INFO repeater task os75bb_lastensq ended after 18.8 seconds 2010-02-12 17:29:11,969 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:28:44 2010-02-12 17:29:16,975 INFO repeater task wh300_lastensq starting 2.0 seconds late 2010-02-12 17:29:37,275 INFO repeater task wh300_lastensq ended after 20.3 seconds 2010-02-12 17:29:37,276 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:29:05 2010-02-12 17:29:37,280 INFO repeater task wh300_beamplots starting 13.8 seconds late 2010-02-12 17:29:41,059 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 17:29:41,060 INFO repeater task os75bb_beamplots starting 17.5 seconds late 2010-02-12 17:29:45,070 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-02-12 17:29:45,071 INFO repeater task os75nb_beamplots starting 21.5 seconds late 2010-02-12 17:29:48,398 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-02-12 17:29:48,406 INFO repeater task wh300_beamplots starting 24.9 seconds late 2010-02-12 17:30:06,354 INFO repeater task wh300_beamplots ended after 17.9 seconds 2010-02-12 17:30:06,365 INFO repeater task os75bb_beamplots starting 42.8 seconds late 2010-02-12 17:30:24,162 INFO repeater task os75bb_beamplots ended after 17.8 seconds 2010-02-12 17:30:24,164 INFO repeater task os75nb_beamplots starting 60.6 seconds late 2010-02-12 17:30:41,515 INFO repeater task os75nb_beamplots ended after 17.4 seconds 2010-02-12 17:30:41,517 INFO repeater task posmv_hcorrstats starting 78.0 seconds late 2010-02-12 17:31:05,805 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:31:05,805 INFO repeater task posmv_hcorrstats ended after 24.3 seconds 2010-02-12 17:31:05,813 INFO repeater task phins_hcorrstats starting 102.2 seconds late 2010-02-12 17:31:27,620 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:31:27,620 INFO repeater task phins_hcorrstats ended after 21.8 seconds 2010-02-12 17:31:32,619 INFO repeater task wh300_lastensq starting 17.6 seconds late 2010-02-12 17:31:54,148 INFO repeater task wh300_lastensq ended after 21.5 seconds 2010-02-12 17:31:54,149 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:31:05 2010-02-12 17:31:54,205 INFO repeater task os75nb_lastensq starting 83.2 seconds late 2010-02-12 17:32:15,381 INFO repeater task os75nb_lastensq ended after 21.2 seconds 2010-02-12 17:32:15,382 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:30:23 2010-02-12 17:32:15,392 INFO repeater task wh300_beamplots starting 51.9 seconds late 2010-02-12 17:32:18,799 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 17:32:18,800 INFO repeater task os75bb_beamplots starting 55.3 seconds late 2010-02-12 17:32:22,708 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-02-12 17:32:22,720 INFO repeater task os75nb_beamplots starting 59.2 seconds late 2010-02-12 17:32:26,137 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-02-12 17:33:16,218 INFO repeater task wh300_lastensq starting 1.2 seconds late 2010-02-12 17:33:36,798 INFO repeater task wh300_lastensq ended after 20.6 seconds 2010-02-12 17:33:36,799 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:33:05 2010-02-12 17:33:36,803 INFO repeater task wh300_beamplots starting 13.3 seconds late 2010-02-12 17:33:40,087 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 17:33:40,088 INFO repeater task os75bb_beamplots starting 16.6 seconds late 2010-02-12 17:33:44,488 INFO repeater task os75bb_beamplots ended after 4.4 seconds 2010-02-12 17:33:44,489 INFO repeater task os75nb_beamplots starting 21.0 seconds late 2010-02-12 17:33:47,877 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-02-12 17:33:57,888 INFO repeater task os75bb_lastensq starting 3.9 seconds late 2010-02-12 17:34:18,488 INFO repeater task os75bb_lastensq ended after 20.6 seconds 2010-02-12 17:34:18,489 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:33:46 2010-02-12 17:34:28,526 INFO repeater task posmv_hcorrstats starting 5.0 seconds late 2010-02-12 17:34:53,349 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:34:53,350 INFO repeater task posmv_hcorrstats ended after 24.8 seconds 2010-02-12 17:34:53,350 INFO repeater task phins_hcorrstats starting 29.8 seconds late 2010-02-12 17:35:15,183 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:35:15,184 INFO repeater task phins_hcorrstats ended after 21.8 seconds 2010-02-12 17:35:20,194 INFO repeater task wh300_lastensq starting 5.2 seconds late 2010-02-12 17:35:41,738 INFO repeater task wh300_lastensq ended after 21.5 seconds 2010-02-12 17:35:41,739 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:35:05 2010-02-12 17:35:41,802 INFO repeater task os75nb_lastensq starting 8.8 seconds late 2010-02-12 17:35:58,814 INFO repeater task os75nb_lastensq ended after 17.0 seconds 2010-02-12 17:35:58,815 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:35:25 2010-02-12 17:35:58,839 INFO repeater task wh300_beamplots starting 35.3 seconds late 2010-02-12 17:36:02,592 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 17:36:02,593 INFO repeater task os75bb_beamplots starting 39.1 seconds late 2010-02-12 17:36:07,679 INFO repeater task os75bb_beamplots ended after 5.1 seconds 2010-02-12 17:36:07,689 INFO repeater task os75nb_beamplots starting 44.2 seconds late 2010-02-12 17:36:13,115 INFO repeater task os75nb_beamplots ended after 5.4 seconds 2010-02-12 17:37:18,515 INFO repeater task wh300_lastensq starting 3.5 seconds late 2010-02-12 17:37:39,074 INFO repeater task wh300_lastensq ended after 20.6 seconds 2010-02-12 17:37:39,075 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:37:05 2010-02-12 17:37:39,079 INFO repeater task wh300_beamplots starting 15.6 seconds late 2010-02-12 17:37:42,387 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 17:37:42,388 INFO repeater task os75bb_beamplots starting 18.9 seconds late 2010-02-12 17:37:46,736 INFO repeater task os75bb_beamplots ended after 4.3 seconds 2010-02-12 17:37:46,737 INFO repeater task os75nb_beamplots starting 23.2 seconds late 2010-02-12 17:37:50,014 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-02-12 17:39:00,084 INFO repeater task os75bb_lastensq starting 4.1 seconds late 2010-02-12 17:39:18,678 INFO repeater task os75bb_lastensq ended after 18.6 seconds 2010-02-12 17:39:18,679 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:38:48 2010-02-12 17:39:23,687 INFO repeater task wh300_lastensq starting 8.7 seconds late 2010-02-12 17:39:44,138 INFO repeater task wh300_lastensq ended after 20.5 seconds 2010-02-12 17:39:44,139 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:39:05 2010-02-12 17:39:44,140 INFO repeater task wh300_quick starting 20.6 seconds late 2010-02-12 17:40:57,719 INFO repeater task wh300_quick ended after 73.6 seconds 2010-02-12 17:40:57,720 INFO repeater task wh300_3dayplots starting 94.2 seconds late 2010-02-12 17:41:11,417 INFO repeater task wh300_3dayplots ended after 13.7 seconds 2010-02-12 17:41:11,418 INFO repeater task os75bb_quick starting 107.9 seconds late 2010-02-12 17:42:07,882 INFO repeater task os75bb_quick ended after 56.5 seconds 2010-02-12 17:42:07,895 INFO repeater task os75bb_3dayplots starting 164.4 seconds late 2010-02-12 17:42:24,841 INFO repeater task os75bb_3dayplots ended after 16.9 seconds 2010-02-12 17:42:24,842 INFO repeater task os75nb_lastensq starting 109.8 seconds late 2010-02-12 17:42:41,957 INFO repeater task os75nb_lastensq ended after 17.1 seconds 2010-02-12 17:42:41,958 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:40:27 2010-02-12 17:42:41,972 INFO repeater task os75nb_quick starting 198.4 seconds late 2010-02-12 17:43:41,895 INFO repeater task os75nb_quick ended after 59.9 seconds 2010-02-12 17:43:41,896 INFO repeater task os75nb_3dayplots starting 258.4 seconds late 2010-02-12 17:44:03,018 INFO repeater task os75nb_3dayplots ended after 21.1 seconds 2010-02-12 17:44:03,019 INFO repeater task wh300_beamplots starting 279.5 seconds late 2010-02-12 17:44:06,942 INFO repeater task wh300_beamplots ended after 3.9 seconds 2010-02-12 17:44:06,942 INFO repeater Extra increment added to time_to_run 2010-02-12 17:44:06,943 INFO repeater Extra increment added to time_to_run 2010-02-12 17:44:06,943 INFO repeater task os75bb_beamplots starting 283.4 seconds late 2010-02-12 17:44:11,245 INFO repeater task os75bb_beamplots ended after 4.3 seconds 2010-02-12 17:44:11,245 INFO repeater Extra increment added to time_to_run 2010-02-12 17:44:11,246 INFO repeater Extra increment added to time_to_run 2010-02-12 17:44:11,262 INFO repeater task os75nb_beamplots starting 287.7 seconds late 2010-02-12 17:44:16,425 INFO repeater task os75nb_beamplots ended after 5.2 seconds 2010-02-12 17:44:16,426 INFO repeater Extra increment added to time_to_run 2010-02-12 17:44:16,426 INFO repeater Extra increment added to time_to_run 2010-02-12 17:44:16,427 INFO repeater task wh300_beamplots starting 292.9 seconds late 2010-02-12 17:44:33,245 INFO repeater task wh300_beamplots ended after 16.8 seconds 2010-02-12 17:44:33,246 INFO repeater task os75bb_beamplots starting 309.7 seconds late 2010-02-12 17:44:50,585 INFO repeater task os75bb_beamplots ended after 17.3 seconds 2010-02-12 17:44:50,586 INFO repeater task os75nb_beamplots starting 327.0 seconds late 2010-02-12 17:45:07,846 INFO repeater task os75nb_beamplots ended after 17.3 seconds 2010-02-12 17:45:07,880 INFO repeater task posmv_hcorrstats starting 344.3 seconds late 2010-02-12 17:45:34,531 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:45:34,532 INFO repeater task posmv_hcorrstats ended after 26.7 seconds 2010-02-12 17:45:34,532 INFO repeater Extra increment added to time_to_run 2010-02-12 17:45:34,533 INFO repeater task phins_hcorrstats starting 371.0 seconds late 2010-02-12 17:46:01,206 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:46:01,206 INFO repeater task phins_hcorrstats ended after 26.7 seconds 2010-02-12 17:46:01,207 INFO repeater Extra increment added to time_to_run 2010-02-12 17:46:06,210 INFO repeater task wh300_lastensq starting 291.2 seconds late 2010-02-12 17:46:35,616 INFO repeater task wh300_lastensq ended after 29.4 seconds 2010-02-12 17:46:35,818 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:45:05 2010-02-12 17:46:35,820 INFO repeater task os75bb_lastensq starting 157.8 seconds late 2010-02-12 17:46:53,867 INFO repeater task os75bb_lastensq ended after 18.0 seconds 2010-02-12 17:46:53,868 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:43:49 2010-02-12 17:46:53,869 INFO repeater task os75nb_lastensq starting 76.9 seconds late 2010-02-12 17:47:12,011 INFO repeater task os75nb_lastensq ended after 18.1 seconds 2010-02-12 17:47:12,012 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:45:29 2010-02-12 17:47:12,013 INFO repeater task wh300_beamplots starting 108.5 seconds late 2010-02-12 17:47:15,826 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 17:47:15,833 INFO repeater task os75bb_beamplots starting 112.3 seconds late 2010-02-12 17:47:19,431 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-02-12 17:47:19,432 INFO repeater task os75nb_beamplots starting 115.9 seconds late 2010-02-12 17:47:22,671 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-02-12 17:47:27,675 INFO repeater task wh300_lastensq starting 12.7 seconds late 2010-02-12 17:47:49,338 INFO repeater task wh300_lastensq ended after 21.7 seconds 2010-02-12 17:47:49,339 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:47:05 2010-02-12 17:47:49,344 INFO repeater task wh300_beamplots starting 25.8 seconds late 2010-02-12 17:47:52,647 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 17:47:52,648 INFO repeater task os75bb_beamplots starting 29.1 seconds late 2010-02-12 17:47:56,343 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 17:47:56,354 INFO repeater task os75nb_beamplots starting 32.8 seconds late 2010-02-12 17:48:00,421 INFO repeater task os75nb_beamplots ended after 4.1 seconds 2010-02-12 17:49:00,777 INFO repeater task os75bb_lastensq starting 1.8 seconds late 2010-02-12 17:49:18,259 INFO repeater task os75bb_lastensq ended after 17.5 seconds 2010-02-12 17:49:18,260 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:48:51 2010-02-12 17:49:23,267 INFO repeater task wh300_lastensq starting 8.3 seconds late 2010-02-12 17:49:44,449 INFO repeater task wh300_lastensq ended after 21.2 seconds 2010-02-12 17:49:44,450 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:49:05 2010-02-12 17:49:44,453 INFO repeater task wh300_beamplots starting 20.9 seconds late 2010-02-12 17:49:47,934 INFO repeater task wh300_beamplots ended after 3.5 seconds 2010-02-12 17:49:47,935 INFO repeater task os75bb_beamplots starting 24.4 seconds late 2010-02-12 17:49:51,632 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 17:49:51,634 INFO repeater task os75nb_beamplots starting 28.1 seconds late 2010-02-12 17:49:55,014 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-02-12 17:49:55,015 INFO repeater task wh300_beamplots starting 31.5 seconds late 2010-02-12 17:50:11,800 INFO repeater task wh300_beamplots ended after 16.8 seconds 2010-02-12 17:50:11,809 INFO repeater task os75bb_beamplots starting 48.3 seconds late 2010-02-12 17:50:32,408 INFO repeater task os75bb_beamplots ended after 20.6 seconds 2010-02-12 17:50:32,409 INFO repeater task os75nb_beamplots starting 68.9 seconds late 2010-02-12 17:50:48,936 INFO repeater task os75nb_beamplots ended after 16.5 seconds 2010-02-12 17:50:48,938 INFO repeater task posmv_hcorrstats starting 85.4 seconds late 2010-02-12 17:51:13,988 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:51:13,988 INFO repeater task posmv_hcorrstats ended after 25.1 seconds 2010-02-12 17:51:13,989 INFO repeater task phins_hcorrstats starting 110.4 seconds late 2010-02-12 17:51:36,279 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:51:36,279 INFO repeater task phins_hcorrstats ended after 22.3 seconds 2010-02-12 17:51:41,279 INFO repeater task wh300_lastensq starting 26.3 seconds late 2010-02-12 17:52:02,808 INFO repeater task wh300_lastensq ended after 21.5 seconds 2010-02-12 17:52:02,809 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:51:05 2010-02-12 17:52:02,812 INFO repeater task os75nb_lastensq starting 83.8 seconds late 2010-02-12 17:52:23,273 INFO repeater task os75nb_lastensq ended after 20.5 seconds 2010-02-12 17:52:23,274 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:50:31 2010-02-12 17:52:23,275 INFO repeater task wh300_beamplots starting 59.7 seconds late 2010-02-12 17:52:26,762 INFO repeater task wh300_beamplots ended after 3.5 seconds 2010-02-12 17:52:26,763 INFO repeater task os75bb_beamplots starting 63.2 seconds late 2010-02-12 17:52:30,954 INFO repeater task os75bb_beamplots ended after 4.2 seconds 2010-02-12 17:52:30,958 INFO repeater task os75nb_beamplots starting 67.4 seconds late 2010-02-12 17:52:35,136 INFO repeater task os75nb_beamplots ended after 4.2 seconds 2010-02-12 17:53:15,394 INFO repeater task wh300_lastensq starting 0.4 seconds late 2010-02-12 17:53:37,359 INFO repeater task wh300_lastensq ended after 22.0 seconds 2010-02-12 17:53:37,360 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:53:05 2010-02-12 17:53:37,364 INFO repeater task wh300_beamplots starting 13.8 seconds late 2010-02-12 17:53:41,075 INFO repeater task wh300_beamplots ended after 3.7 seconds 2010-02-12 17:53:41,076 INFO repeater task os75bb_beamplots starting 17.5 seconds late 2010-02-12 17:53:44,493 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 17:53:44,494 INFO repeater task os75nb_beamplots starting 21.0 seconds late 2010-02-12 17:53:47,975 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 17:54:03,025 INFO repeater task os75bb_lastensq starting 2.0 seconds late 2010-02-12 17:54:23,723 INFO repeater task os75bb_lastensq ended after 20.7 seconds 2010-02-12 17:54:23,724 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:53:54 2010-02-12 17:54:23,730 INFO repeater task posmv_hcorrstats starting 0.2 seconds late 2010-02-12 17:54:49,504 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:54:49,504 INFO repeater task posmv_hcorrstats ended after 25.8 seconds 2010-02-12 17:54:49,505 INFO repeater task phins_hcorrstats starting 25.9 seconds late 2010-02-12 17:55:11,216 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 17:55:11,216 INFO repeater task phins_hcorrstats ended after 21.7 seconds 2010-02-12 17:55:16,215 INFO repeater task wh300_lastensq starting 1.2 seconds late 2010-02-12 17:55:38,987 INFO repeater task wh300_lastensq ended after 22.8 seconds 2010-02-12 17:55:38,988 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:55:06 2010-02-12 17:55:38,992 INFO repeater task wh300_beamplots starting 15.5 seconds late 2010-02-12 17:55:42,250 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 17:55:42,250 INFO repeater task os75bb_beamplots starting 18.7 seconds late 2010-02-12 17:55:45,932 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 17:55:45,941 INFO repeater task os75nb_beamplots starting 22.4 seconds late 2010-02-12 17:55:49,443 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 17:55:54,449 INFO repeater task os75nb_lastensq starting 13.4 seconds late 2010-02-12 17:56:12,297 INFO repeater task os75nb_lastensq ended after 17.8 seconds 2010-02-12 17:56:12,298 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 17:55:33 2010-02-12 17:57:17,602 INFO repeater task wh300_lastensq starting 1.6 seconds late 2010-02-12 17:57:40,007 INFO repeater task wh300_lastensq ended after 22.4 seconds 2010-02-12 17:57:40,008 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 17:57:06 2010-02-12 17:57:40,079 INFO repeater task wh300_beamplots starting 16.6 seconds late 2010-02-12 17:57:43,787 INFO repeater task wh300_beamplots ended after 3.7 seconds 2010-02-12 17:57:43,788 INFO repeater task os75bb_beamplots starting 20.3 seconds late 2010-02-12 17:57:47,295 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 17:57:47,296 INFO repeater task os75nb_beamplots starting 23.8 seconds late 2010-02-12 17:57:50,753 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 17:59:06,707 INFO repeater task os75bb_lastensq starting 2.7 seconds late 2010-02-12 17:59:31,733 INFO repeater task os75bb_lastensq ended after 25.0 seconds 2010-02-12 17:59:31,915 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 17:58:56 2010-02-12 17:59:31,919 INFO repeater task wh300_beamplots starting 8.4 seconds late 2010-02-12 17:59:38,600 INFO repeater task wh300_beamplots ended after 6.7 seconds 2010-02-12 17:59:38,610 INFO repeater task os75bb_beamplots starting 15.1 seconds late 2010-02-12 17:59:46,539 INFO repeater task os75bb_beamplots ended after 7.9 seconds 2010-02-12 17:59:46,540 INFO repeater task os75nb_beamplots starting 23.0 seconds late 2010-02-12 17:59:51,627 INFO repeater task os75nb_beamplots ended after 5.1 seconds 2010-02-12 17:59:51,632 INFO repeater task wh300_beamplots starting 28.1 seconds late 2010-02-12 18:00:12,275 INFO repeater task wh300_beamplots ended after 20.6 seconds 2010-02-12 18:00:12,276 INFO repeater task os75bb_beamplots starting 48.7 seconds late 2010-02-12 18:00:31,449 INFO repeater task os75bb_beamplots ended after 19.2 seconds 2010-02-12 18:00:31,450 INFO repeater task os75nb_beamplots starting 67.9 seconds late 2010-02-12 18:00:49,426 INFO repeater task os75nb_beamplots ended after 18.0 seconds 2010-02-12 18:00:49,432 INFO repeater task posmv_hcorrstats starting 85.9 seconds late 2010-02-12 18:01:23,378 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:01:23,393 INFO repeater task posmv_hcorrstats ended after 34.0 seconds 2010-02-12 18:01:23,394 INFO repeater task phins_hcorrstats starting 119.8 seconds late 2010-02-12 18:01:49,949 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:01:49,950 INFO repeater task phins_hcorrstats ended after 26.6 seconds 2010-02-12 18:01:54,947 INFO repeater task wh300_lastensq starting 158.9 seconds late 2010-02-12 18:02:22,602 INFO repeater task wh300_lastensq ended after 27.7 seconds 2010-02-12 18:02:22,602 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:01:06 2010-02-12 18:02:22,605 INFO repeater task os75nb_lastensq starting 99.6 seconds late 2010-02-12 18:02:44,170 INFO repeater task os75nb_lastensq ended after 21.6 seconds 2010-02-12 18:02:44,171 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:00:35 2010-02-12 18:02:44,173 INFO repeater task wh300_beamplots starting 80.6 seconds late 2010-02-12 18:02:47,941 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 18:02:47,942 INFO repeater task os75bb_beamplots starting 84.4 seconds late 2010-02-12 18:02:51,328 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 18:02:51,329 INFO repeater task os75nb_beamplots starting 87.8 seconds late 2010-02-12 18:02:54,858 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 18:03:20,160 INFO repeater task wh300_lastensq starting 4.2 seconds late 2010-02-12 18:03:40,433 INFO repeater task wh300_lastensq ended after 20.3 seconds 2010-02-12 18:03:40,434 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:03:06 2010-02-12 18:03:40,437 INFO repeater task wh300_beamplots starting 16.9 seconds late 2010-02-12 18:03:44,677 INFO repeater task wh300_beamplots ended after 4.2 seconds 2010-02-12 18:03:44,679 INFO repeater task os75bb_beamplots starting 21.1 seconds late 2010-02-12 18:03:48,556 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-02-12 18:03:48,557 INFO repeater task os75nb_beamplots starting 25.0 seconds late 2010-02-12 18:03:52,020 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 18:04:07,036 INFO repeater task os75bb_lastensq starting 1.0 seconds late 2010-02-12 18:04:27,188 INFO repeater task os75bb_lastensq ended after 20.2 seconds 2010-02-12 18:04:27,189 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:03:59 2010-02-12 18:04:27,339 INFO repeater task posmv_hcorrstats starting 3.8 seconds late 2010-02-12 18:04:52,227 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:04:52,228 INFO repeater task posmv_hcorrstats ended after 24.9 seconds 2010-02-12 18:04:52,229 INFO repeater task phins_hcorrstats starting 28.7 seconds late 2010-02-12 18:05:13,803 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:05:13,803 INFO repeater task phins_hcorrstats ended after 21.6 seconds 2010-02-12 18:05:18,803 INFO repeater task wh300_lastensq starting 2.8 seconds late 2010-02-12 18:05:38,372 INFO repeater task wh300_lastensq ended after 19.6 seconds 2010-02-12 18:05:38,373 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:05:06 2010-02-12 18:05:38,393 INFO repeater task wh300_beamplots starting 14.9 seconds late 2010-02-12 18:05:42,512 INFO repeater task wh300_beamplots ended after 4.1 seconds 2010-02-12 18:05:42,514 INFO repeater task os75bb_beamplots starting 19.0 seconds late 2010-02-12 18:05:46,895 INFO repeater task os75bb_beamplots ended after 4.4 seconds 2010-02-12 18:05:46,896 INFO repeater task os75nb_beamplots starting 23.4 seconds late 2010-02-12 18:05:50,388 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 18:05:55,474 INFO repeater task os75nb_lastensq starting 10.5 seconds late 2010-02-12 18:06:12,600 INFO repeater task os75nb_lastensq ended after 17.1 seconds 2010-02-12 18:06:12,601 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:05:38 2010-02-12 18:07:17,807 INFO repeater task wh300_lastensq starting 1.8 seconds late 2010-02-12 18:07:36,814 INFO repeater task wh300_lastensq ended after 19.0 seconds 2010-02-12 18:07:36,815 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:07:06 2010-02-12 18:07:36,820 INFO repeater task wh300_beamplots starting 13.3 seconds late 2010-02-12 18:07:40,406 INFO repeater task wh300_beamplots ended after 3.6 seconds 2010-02-12 18:07:40,407 INFO repeater task os75bb_beamplots starting 16.9 seconds late 2010-02-12 18:07:45,048 INFO repeater task os75bb_beamplots ended after 4.6 seconds 2010-02-12 18:07:45,049 INFO repeater task os75nb_beamplots starting 21.5 seconds late 2010-02-12 18:07:49,286 INFO repeater task os75nb_beamplots ended after 4.2 seconds 2010-02-12 18:09:09,736 INFO repeater task os75bb_lastensq starting 0.7 seconds late 2010-02-12 18:09:27,054 INFO repeater task os75bb_lastensq ended after 17.3 seconds 2010-02-12 18:09:27,054 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:09:01 2010-02-12 18:09:27,055 INFO repeater task os75bb_quick starting 3.5 seconds late 2010-02-12 18:10:28,767 INFO repeater task os75bb_quick ended after 61.7 seconds 2010-02-12 18:10:28,774 INFO repeater task os75bb_3dayplots starting 65.2 seconds late 2010-02-12 18:10:44,692 INFO repeater task os75bb_3dayplots ended after 15.9 seconds 2010-02-12 18:10:44,693 INFO repeater task os75nb_quick starting 81.2 seconds late 2010-02-12 18:11:41,946 INFO repeater task os75nb_quick ended after 57.3 seconds 2010-02-12 18:11:41,947 INFO repeater task os75nb_3dayplots starting 138.4 seconds late 2010-02-12 18:12:04,464 INFO repeater task os75nb_3dayplots ended after 22.5 seconds 2010-02-12 18:12:04,465 INFO repeater task wh300_beamplots starting 160.9 seconds late 2010-02-12 18:12:07,744 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 18:12:07,745 INFO repeater Extra increment added to time_to_run 2010-02-12 18:12:07,746 INFO repeater task os75bb_beamplots starting 164.2 seconds late 2010-02-12 18:12:11,705 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-02-12 18:12:11,706 INFO repeater Extra increment added to time_to_run 2010-02-12 18:12:11,707 INFO repeater task os75nb_beamplots starting 168.2 seconds late 2010-02-12 18:12:14,991 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-02-12 18:12:14,992 INFO repeater Extra increment added to time_to_run 2010-02-12 18:12:15,001 INFO repeater task wh300_beamplots starting 171.5 seconds late 2010-02-12 18:12:33,847 INFO repeater task wh300_beamplots ended after 18.8 seconds 2010-02-12 18:12:33,861 INFO repeater task os75bb_beamplots starting 190.3 seconds late 2010-02-12 18:12:52,768 INFO repeater task os75bb_beamplots ended after 18.9 seconds 2010-02-12 18:12:52,769 INFO repeater task os75nb_beamplots starting 209.2 seconds late 2010-02-12 18:13:09,356 INFO repeater task os75nb_beamplots ended after 16.6 seconds 2010-02-12 18:13:09,357 INFO repeater task wh300_beamhistory starting 225.8 seconds late 2010-02-12 18:13:12,419 ERROR repeater task wh300_beamhistory returncode 1 2010-02-12 18:13:12,439 INFO repeater task wh300_beamhistory output: Traceback (most recent call last): File "/home/currents/programs/uhdas/scripts/beam_history.py", line 151, in yearbase=yearbase) File "/home/currents/programs/pycurrents/py_utils/adcp_diagnostics.py", line 734, in beam_stats_byfile data=m.read(ends=1) File "/home/currents/programs/pycurrents/adcp/rdiraw.py", line 1647, in read binlist.append(a[ilist]) IndexError: index (6407) out of range (0<=index<6406) in dimension 0 2010-02-12 18:13:12,439 INFO repeater task wh300_beamhistory ended after 3.1 seconds 2010-02-12 18:13:12,448 INFO repeater task os75bb_beamhistory starting 228.9 seconds late 2010-02-12 18:14:25,432 INFO repeater task os75bb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75bb_bin_history.png 2010-02-12 18:14:25,445 INFO repeater task os75bb_beamhistory ended after 73.0 seconds 2010-02-12 18:14:25,446 INFO repeater task os75nb_beamhistory starting 301.9 seconds late 2010-02-12 18:15:17,123 INFO repeater task os75nb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75nb_bin_history.png 2010-02-12 18:15:17,124 INFO repeater task os75nb_beamhistory ended after 51.7 seconds 2010-02-12 18:15:17,125 INFO repeater task posmv_hcorrstats starting 353.6 seconds late 2010-02-12 18:15:46,556 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:15:46,572 INFO repeater task posmv_hcorrstats ended after 29.4 seconds 2010-02-12 18:15:46,573 INFO repeater Extra increment added to time_to_run 2010-02-12 18:15:46,581 INFO repeater task phins_hcorrstats starting 383.0 seconds late 2010-02-12 18:16:15,168 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:16:15,169 INFO repeater task phins_hcorrstats ended after 28.6 seconds 2010-02-12 18:16:15,169 INFO repeater Extra increment added to time_to_run 2010-02-12 18:16:20,167 INFO repeater task wh300_lastensq starting 424.2 seconds late 2010-02-12 18:16:49,895 INFO repeater task wh300_lastensq ended after 29.7 seconds 2010-02-12 18:16:49,896 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:15:07 2010-02-12 18:16:49,904 INFO repeater task wh300_quick starting 446.4 seconds late 2010-02-12 18:18:03,041 INFO repeater task wh300_quick ended after 73.1 seconds 2010-02-12 18:18:03,043 INFO repeater task wh300_3dayplots starting 519.5 seconds late 2010-02-12 18:18:16,240 INFO repeater task wh300_3dayplots ended after 13.2 seconds 2010-02-12 18:18:16,240 INFO repeater task os75bb_lastensq starting 245.2 seconds late 2010-02-12 18:18:38,756 INFO repeater task os75bb_lastensq ended after 22.5 seconds 2010-02-12 18:18:38,757 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:18:18 2010-02-12 18:18:38,759 INFO repeater task os75nb_lastensq starting 470.8 seconds late 2010-02-12 18:18:58,505 INFO repeater task os75nb_lastensq ended after 19.7 seconds 2010-02-12 18:18:58,506 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:15:42 2010-02-12 18:18:58,508 INFO repeater task wh300_beamplots starting 335.0 seconds late 2010-02-12 18:19:02,269 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 18:19:02,270 INFO repeater Extra increment added to time_to_run 2010-02-12 18:19:02,270 INFO repeater Extra increment added to time_to_run 2010-02-12 18:19:02,280 INFO repeater task os75bb_beamplots starting 338.7 seconds late 2010-02-12 18:19:06,047 INFO repeater task os75bb_beamplots ended after 3.8 seconds 2010-02-12 18:19:06,048 INFO repeater Extra increment added to time_to_run 2010-02-12 18:19:06,048 INFO repeater Extra increment added to time_to_run 2010-02-12 18:19:06,055 INFO repeater task os75nb_beamplots starting 342.5 seconds late 2010-02-12 18:19:09,538 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 18:19:09,540 INFO repeater Extra increment added to time_to_run 2010-02-12 18:19:09,540 INFO repeater Extra increment added to time_to_run 2010-02-12 18:19:14,543 INFO repeater task wh300_lastensq starting 117.5 seconds late 2010-02-12 18:19:37,627 INFO repeater task wh300_lastensq ended after 23.1 seconds 2010-02-12 18:19:37,628 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:19:07 2010-02-12 18:19:37,632 INFO repeater task wh300_beamplots starting 14.1 seconds late 2010-02-12 18:19:41,182 INFO repeater task wh300_beamplots ended after 3.6 seconds 2010-02-12 18:19:41,184 INFO repeater task os75bb_beamplots starting 17.7 seconds late 2010-02-12 18:19:44,897 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 18:19:44,898 INFO repeater task os75nb_beamplots starting 21.4 seconds late 2010-02-12 18:19:48,112 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-02-12 18:19:48,113 INFO repeater task wh300_beamplots starting 24.6 seconds late 2010-02-12 18:20:05,331 INFO repeater task wh300_beamplots ended after 17.2 seconds 2010-02-12 18:20:05,332 INFO repeater task os75bb_beamplots starting 41.8 seconds late 2010-02-12 18:20:22,511 INFO repeater task os75bb_beamplots ended after 17.2 seconds 2010-02-12 18:20:22,519 INFO repeater task os75nb_beamplots starting 59.0 seconds late 2010-02-12 18:20:42,333 INFO repeater task os75nb_beamplots ended after 19.8 seconds 2010-02-12 18:20:42,335 INFO repeater task posmv_hcorrstats starting 78.8 seconds late 2010-02-12 18:21:13,433 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:21:13,433 INFO repeater task posmv_hcorrstats ended after 31.1 seconds 2010-02-12 18:21:13,434 INFO repeater task phins_hcorrstats starting 109.9 seconds late 2010-02-12 18:21:38,655 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:21:38,655 INFO repeater task phins_hcorrstats ended after 25.2 seconds 2010-02-12 18:21:43,661 INFO repeater task wh300_lastensq starting 26.7 seconds late 2010-02-12 18:22:05,340 INFO repeater task wh300_lastensq ended after 21.7 seconds 2010-02-12 18:22:05,341 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:21:07 2010-02-12 18:22:05,344 INFO repeater task os75nb_lastensq starting 73.3 seconds late 2010-02-12 18:22:21,893 INFO repeater task os75nb_lastensq ended after 16.5 seconds 2010-02-12 18:22:21,894 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:20:45 2010-02-12 18:22:21,923 INFO repeater task wh300_beamplots starting 58.4 seconds late 2010-02-12 18:22:25,518 INFO repeater task wh300_beamplots ended after 3.6 seconds 2010-02-12 18:22:25,519 INFO repeater task os75bb_beamplots starting 62.0 seconds late 2010-02-12 18:22:30,040 INFO repeater task os75bb_beamplots ended after 4.5 seconds 2010-02-12 18:22:30,041 INFO repeater task os75nb_beamplots starting 66.5 seconds late 2010-02-12 18:22:34,648 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-12 18:23:20,175 INFO repeater task wh300_lastensq starting 3.2 seconds late 2010-02-12 18:23:41,580 INFO repeater task wh300_lastensq ended after 21.4 seconds 2010-02-12 18:23:41,581 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:23:08 2010-02-12 18:23:41,610 INFO repeater task os75bb_lastensq starting 13.6 seconds late 2010-02-12 18:23:58,606 INFO repeater task os75bb_lastensq ended after 17.0 seconds 2010-02-12 18:23:58,607 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:23:24 2010-02-12 18:23:58,676 INFO repeater task wh300_beamplots starting 35.1 seconds late 2010-02-12 18:24:03,303 INFO repeater task wh300_beamplots ended after 4.6 seconds 2010-02-12 18:24:03,304 INFO repeater task os75bb_beamplots starting 39.8 seconds late 2010-02-12 18:24:07,704 INFO repeater task os75bb_beamplots ended after 4.4 seconds 2010-02-12 18:24:07,705 INFO repeater task os75nb_beamplots starting 44.2 seconds late 2010-02-12 18:24:11,134 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-02-12 18:24:26,388 INFO repeater task posmv_hcorrstats starting 2.8 seconds late 2010-02-12 18:24:53,301 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:24:53,302 INFO repeater task posmv_hcorrstats ended after 26.9 seconds 2010-02-12 18:24:53,317 INFO repeater task phins_hcorrstats starting 29.8 seconds late 2010-02-12 18:25:15,015 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:25:15,015 INFO repeater task phins_hcorrstats ended after 21.7 seconds 2010-02-12 18:25:20,015 INFO repeater task wh300_lastensq starting 2.0 seconds late 2010-02-12 18:25:41,540 INFO repeater task wh300_lastensq ended after 21.5 seconds 2010-02-12 18:25:41,541 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:25:08 2010-02-12 18:25:41,546 INFO repeater task wh300_beamplots starting 18.0 seconds late 2010-02-12 18:25:44,983 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 18:25:44,984 INFO repeater task os75bb_beamplots starting 21.5 seconds late 2010-02-12 18:25:48,560 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-02-12 18:25:48,561 INFO repeater task os75nb_beamplots starting 25.0 seconds late 2010-02-12 18:25:52,177 INFO repeater task os75nb_beamplots ended after 3.6 seconds 2010-02-12 18:25:57,181 INFO repeater task os75nb_lastensq starting 2.2 seconds late 2010-02-12 18:26:15,023 INFO repeater task os75nb_lastensq ended after 17.8 seconds 2010-02-12 18:26:15,024 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:25:47 2010-02-12 18:27:20,774 INFO repeater task wh300_lastensq starting 2.8 seconds late 2010-02-12 18:27:41,705 INFO repeater task wh300_lastensq ended after 20.9 seconds 2010-02-12 18:27:41,706 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:27:08 2010-02-12 18:27:41,710 INFO repeater task wh300_beamplots starting 18.2 seconds late 2010-02-12 18:27:45,096 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 18:27:45,097 INFO repeater task os75bb_beamplots starting 21.6 seconds late 2010-02-12 18:27:48,780 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 18:27:48,781 INFO repeater task os75nb_beamplots starting 25.2 seconds late 2010-02-12 18:27:52,217 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-02-12 18:28:37,292 INFO repeater task os75bb_lastensq starting 3.3 seconds late 2010-02-12 18:28:55,047 INFO repeater task os75bb_lastensq ended after 17.8 seconds 2010-02-12 18:28:55,048 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:28:27 2010-02-12 18:29:20,407 INFO repeater task wh300_lastensq starting 2.4 seconds late 2010-02-12 18:29:40,163 INFO repeater task wh300_lastensq ended after 19.8 seconds 2010-02-12 18:29:40,163 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:29:08 2010-02-12 18:29:40,237 INFO repeater task wh300_beamplots starting 16.7 seconds late 2010-02-12 18:29:44,251 INFO repeater task wh300_beamplots ended after 4.0 seconds 2010-02-12 18:29:44,252 INFO repeater task os75bb_beamplots starting 20.7 seconds late 2010-02-12 18:29:47,951 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 18:29:47,952 INFO repeater task os75nb_beamplots starting 24.4 seconds late 2010-02-12 18:29:51,459 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 18:29:51,460 INFO repeater task wh300_beamplots starting 27.9 seconds late 2010-02-12 18:30:08,532 INFO repeater task wh300_beamplots ended after 17.1 seconds 2010-02-12 18:30:08,533 INFO repeater task os75bb_beamplots starting 45.0 seconds late 2010-02-12 18:30:26,285 INFO repeater task os75bb_beamplots ended after 17.8 seconds 2010-02-12 18:30:26,293 INFO repeater task os75nb_beamplots starting 62.8 seconds late 2010-02-12 18:30:46,021 INFO repeater task os75nb_beamplots ended after 19.7 seconds 2010-02-12 18:30:46,023 INFO repeater task posmv_hcorrstats starting 82.5 seconds late 2010-02-12 18:31:10,860 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:31:10,860 INFO repeater task posmv_hcorrstats ended after 24.8 seconds 2010-02-12 18:31:10,861 INFO repeater task phins_hcorrstats starting 107.3 seconds late 2010-02-12 18:31:31,967 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:31:31,967 INFO repeater task phins_hcorrstats ended after 21.1 seconds 2010-02-12 18:31:36,967 INFO repeater task wh300_lastensq starting 19.0 seconds late 2010-02-12 18:31:57,815 INFO repeater task wh300_lastensq ended after 20.8 seconds 2010-02-12 18:31:57,816 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:31:08 2010-02-12 18:31:57,819 INFO repeater task os75nb_lastensq starting 60.8 seconds late 2010-02-12 18:32:16,464 INFO repeater task os75nb_lastensq ended after 18.6 seconds 2010-02-12 18:32:16,465 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:30:49 2010-02-12 18:32:16,466 INFO repeater task wh300_beamplots starting 52.9 seconds late 2010-02-12 18:32:20,034 INFO repeater task wh300_beamplots ended after 3.6 seconds 2010-02-12 18:32:20,035 INFO repeater task os75bb_beamplots starting 56.5 seconds late 2010-02-12 18:32:23,437 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 18:32:23,438 INFO repeater task os75nb_beamplots starting 59.9 seconds late 2010-02-12 18:32:27,257 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-02-12 18:33:22,806 INFO repeater task wh300_lastensq starting 4.8 seconds late 2010-02-12 18:33:43,024 INFO repeater task wh300_lastensq ended after 20.2 seconds 2010-02-12 18:33:43,024 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:33:09 2010-02-12 18:33:43,051 INFO repeater task os75bb_lastensq starting 6.1 seconds late 2010-02-12 18:34:00,369 INFO repeater task os75bb_lastensq ended after 17.3 seconds 2010-02-12 18:34:00,370 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:33:29 2010-02-12 18:34:00,373 INFO repeater task wh300_beamplots starting 36.8 seconds late 2010-02-12 18:34:04,193 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 18:34:04,193 INFO repeater task os75bb_beamplots starting 40.7 seconds late 2010-02-12 18:34:09,046 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2010-02-12 18:34:09,047 INFO repeater task os75nb_beamplots starting 45.5 seconds late 2010-02-12 18:34:12,302 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-02-12 18:34:27,318 INFO repeater task posmv_hcorrstats starting 3.8 seconds late 2010-02-12 18:34:53,647 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:34:53,648 INFO repeater task posmv_hcorrstats ended after 26.3 seconds 2010-02-12 18:34:53,649 INFO repeater task phins_hcorrstats starting 30.1 seconds late 2010-02-12 18:35:15,695 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:35:15,696 INFO repeater task phins_hcorrstats ended after 22.0 seconds 2010-02-12 18:35:20,695 INFO repeater task wh300_lastensq starting 1.7 seconds late 2010-02-12 18:35:41,088 INFO repeater task wh300_lastensq ended after 20.4 seconds 2010-02-12 18:35:41,089 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:35:09 2010-02-12 18:35:41,093 INFO repeater task wh300_beamplots starting 17.6 seconds late 2010-02-12 18:35:44,396 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 18:35:44,397 INFO repeater task os75bb_beamplots starting 20.9 seconds late 2010-02-12 18:35:48,364 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-02-12 18:35:48,378 INFO repeater task os75nb_beamplots starting 24.8 seconds late 2010-02-12 18:35:52,031 INFO repeater task os75nb_beamplots ended after 3.7 seconds 2010-02-12 18:36:02,045 INFO repeater task os75nb_lastensq starting 3.0 seconds late 2010-02-12 18:36:21,134 INFO repeater task os75nb_lastensq ended after 19.1 seconds 2010-02-12 18:36:21,135 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:35:52 2010-02-12 18:37:22,158 INFO repeater task wh300_lastensq starting 3.2 seconds late 2010-02-12 18:37:42,171 INFO repeater task wh300_lastensq ended after 20.0 seconds 2010-02-12 18:37:42,171 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:37:09 2010-02-12 18:37:42,205 INFO repeater task wh300_beamplots starting 18.7 seconds late 2010-02-12 18:37:45,567 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 18:37:45,569 INFO repeater task os75bb_beamplots starting 22.0 seconds late 2010-02-12 18:37:49,555 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-02-12 18:37:49,556 INFO repeater task os75nb_beamplots starting 26.0 seconds late 2010-02-12 18:37:53,594 INFO repeater task os75nb_beamplots ended after 4.0 seconds 2010-02-12 18:38:43,960 INFO repeater task os75bb_lastensq starting 5.0 seconds late 2010-02-12 18:39:00,917 INFO repeater task os75bb_lastensq ended after 17.0 seconds 2010-02-12 18:39:00,918 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:38:32 2010-02-12 18:39:21,039 INFO repeater task wh300_lastensq starting 2.0 seconds late 2010-02-12 18:39:41,347 INFO repeater task wh300_lastensq ended after 20.3 seconds 2010-02-12 18:39:41,348 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:39:09 2010-02-12 18:39:41,349 INFO repeater task wh300_quick starting 17.8 seconds late 2010-02-12 18:40:57,226 INFO repeater task wh300_quick ended after 75.9 seconds 2010-02-12 18:40:57,227 INFO repeater task wh300_3dayplots starting 93.7 seconds late 2010-02-12 18:41:10,467 INFO repeater task wh300_3dayplots ended after 13.2 seconds 2010-02-12 18:41:10,469 INFO repeater task os75bb_quick starting 106.9 seconds late 2010-02-12 18:42:10,866 INFO repeater task os75bb_quick ended after 60.4 seconds 2010-02-12 18:42:10,875 INFO repeater task os75bb_3dayplots starting 167.3 seconds late 2010-02-12 18:42:25,961 INFO repeater task os75bb_3dayplots ended after 15.1 seconds 2010-02-12 18:42:25,970 INFO repeater task os75nb_lastensq starting 84.0 seconds late 2010-02-12 18:42:45,038 INFO repeater task os75nb_lastensq ended after 19.1 seconds 2010-02-12 18:42:45,039 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:40:54 2010-02-12 18:42:45,040 INFO repeater task os75nb_quick starting 201.5 seconds late 2010-02-12 18:43:44,472 INFO repeater task os75nb_quick ended after 59.4 seconds 2010-02-12 18:43:44,473 INFO repeater task os75nb_3dayplots starting 260.9 seconds late 2010-02-12 18:44:06,167 INFO repeater task os75nb_3dayplots ended after 21.7 seconds 2010-02-12 18:44:06,168 INFO repeater task wh300_beamplots starting 282.6 seconds late 2010-02-12 18:44:09,400 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-02-12 18:44:09,400 INFO repeater Extra increment added to time_to_run 2010-02-12 18:44:09,401 INFO repeater Extra increment added to time_to_run 2010-02-12 18:44:09,401 INFO repeater task os75bb_beamplots starting 285.9 seconds late 2010-02-12 18:44:12,816 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 18:44:12,816 INFO repeater Extra increment added to time_to_run 2010-02-12 18:44:12,816 INFO repeater Extra increment added to time_to_run 2010-02-12 18:44:12,817 INFO repeater task os75nb_beamplots starting 289.3 seconds late 2010-02-12 18:44:16,697 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-02-12 18:44:16,698 INFO repeater Extra increment added to time_to_run 2010-02-12 18:44:16,699 INFO repeater Extra increment added to time_to_run 2010-02-12 18:44:16,700 INFO repeater task wh300_beamplots starting 293.2 seconds late 2010-02-12 18:44:34,708 INFO repeater task wh300_beamplots ended after 18.0 seconds 2010-02-12 18:44:34,719 INFO repeater task os75bb_beamplots starting 311.2 seconds late 2010-02-12 18:44:53,661 INFO repeater task os75bb_beamplots ended after 18.9 seconds 2010-02-12 18:44:53,662 INFO repeater task os75nb_beamplots starting 330.1 seconds late 2010-02-12 18:45:11,259 INFO repeater task os75nb_beamplots ended after 17.6 seconds 2010-02-12 18:45:11,261 INFO repeater task posmv_hcorrstats starting 347.7 seconds late 2010-02-12 18:45:37,329 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:45:37,347 INFO repeater task posmv_hcorrstats ended after 26.1 seconds 2010-02-12 18:45:37,347 INFO repeater Extra increment added to time_to_run 2010-02-12 18:45:37,355 INFO repeater task phins_hcorrstats starting 373.8 seconds late 2010-02-12 18:46:04,189 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:46:04,190 INFO repeater task phins_hcorrstats ended after 26.8 seconds 2010-02-12 18:46:04,190 INFO repeater Extra increment added to time_to_run 2010-02-12 18:46:09,191 INFO repeater task wh300_lastensq starting 290.2 seconds late 2010-02-12 18:46:36,229 INFO repeater task wh300_lastensq ended after 27.0 seconds 2010-02-12 18:46:36,230 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:45:10 2010-02-12 18:46:36,247 INFO repeater task os75bb_lastensq starting 174.2 seconds late 2010-02-12 18:46:56,025 INFO repeater task os75bb_lastensq ended after 19.8 seconds 2010-02-12 18:46:56,026 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:43:34 2010-02-12 18:46:56,027 INFO repeater task os75nb_lastensq starting 52.0 seconds late 2010-02-12 18:47:13,874 INFO repeater task os75nb_lastensq ended after 17.8 seconds 2010-02-12 18:47:13,875 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:45:57 2010-02-12 18:47:13,877 INFO repeater task wh300_beamplots starting 110.3 seconds late 2010-02-12 18:47:17,115 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-02-12 18:47:17,116 INFO repeater task os75bb_beamplots starting 113.6 seconds late 2010-02-12 18:47:21,898 INFO repeater task os75bb_beamplots ended after 4.8 seconds 2010-02-12 18:47:21,899 INFO repeater task os75nb_beamplots starting 118.4 seconds late 2010-02-12 18:47:25,779 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-02-12 18:47:25,780 INFO repeater Extra increment added to time_to_run 2010-02-12 18:47:30,783 INFO repeater task wh300_lastensq starting 10.8 seconds late 2010-02-12 18:47:50,585 INFO repeater task wh300_lastensq ended after 19.8 seconds 2010-02-12 18:47:50,586 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:47:10 2010-02-12 18:47:50,664 INFO repeater task wh300_beamplots starting 27.1 seconds late 2010-02-12 18:47:54,436 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 18:47:54,437 INFO repeater task os75bb_beamplots starting 30.9 seconds late 2010-02-12 18:47:58,231 INFO repeater task os75bb_beamplots ended after 3.8 seconds 2010-02-12 18:48:48,424 INFO repeater task os75bb_lastensq starting 4.4 seconds late 2010-02-12 18:49:07,352 INFO repeater task os75bb_lastensq ended after 18.9 seconds 2010-02-12 18:49:07,353 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:48:36 2010-02-12 18:49:22,385 INFO repeater task wh300_lastensq starting 2.4 seconds late 2010-02-12 18:49:43,034 INFO repeater task wh300_lastensq ended after 20.6 seconds 2010-02-12 18:49:43,035 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:49:10 2010-02-12 18:49:43,039 INFO repeater task wh300_beamplots starting 19.5 seconds late 2010-02-12 18:49:46,171 INFO repeater task wh300_beamplots ended after 3.1 seconds 2010-02-12 18:49:46,185 INFO repeater task os75bb_beamplots starting 22.7 seconds late 2010-02-12 18:49:49,828 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-02-12 18:49:49,829 INFO repeater task os75nb_beamplots starting 26.3 seconds late 2010-02-12 18:49:53,992 INFO repeater task os75nb_beamplots ended after 4.2 seconds 2010-02-12 18:49:53,993 INFO repeater task wh300_beamplots starting 30.5 seconds late 2010-02-12 18:50:10,311 INFO repeater task wh300_beamplots ended after 16.3 seconds 2010-02-12 18:50:10,312 INFO repeater task os75bb_beamplots starting 46.8 seconds late 2010-02-12 18:50:29,756 INFO repeater task os75bb_beamplots ended after 19.4 seconds 2010-02-12 18:50:29,757 INFO repeater task os75nb_beamplots starting 66.2 seconds late 2010-02-12 18:50:48,333 INFO repeater task os75nb_beamplots ended after 18.6 seconds 2010-02-12 18:50:48,367 INFO repeater task posmv_hcorrstats starting 84.8 seconds late 2010-02-12 18:51:13,581 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:51:13,582 INFO repeater task posmv_hcorrstats ended after 25.2 seconds 2010-02-12 18:51:13,583 INFO repeater task phins_hcorrstats starting 110.0 seconds late 2010-02-12 18:51:36,588 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:51:36,589 INFO repeater task phins_hcorrstats ended after 23.0 seconds 2010-02-12 18:51:41,587 INFO repeater task wh300_lastensq starting 21.6 seconds late 2010-02-12 18:52:02,401 INFO repeater task wh300_lastensq ended after 20.8 seconds 2010-02-12 18:52:02,401 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:51:11 2010-02-12 18:52:02,413 INFO repeater task os75nb_lastensq starting 55.4 seconds late 2010-02-12 18:52:19,546 INFO repeater task os75nb_lastensq ended after 17.1 seconds 2010-02-12 18:52:19,547 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:50:59 2010-02-12 18:52:19,571 INFO repeater task wh300_beamplots starting 56.0 seconds late 2010-02-12 18:52:22,997 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 18:52:22,999 INFO repeater task os75bb_beamplots starting 59.5 seconds late 2010-02-12 18:52:27,888 INFO repeater task os75bb_beamplots ended after 4.9 seconds 2010-02-12 18:52:27,889 INFO repeater task os75nb_beamplots starting 64.4 seconds late 2010-02-12 18:52:32,006 INFO repeater task os75nb_beamplots ended after 4.1 seconds 2010-02-12 18:53:22,679 INFO repeater task wh300_lastensq starting 1.7 seconds late 2010-02-12 18:53:44,008 INFO repeater task wh300_lastensq ended after 21.3 seconds 2010-02-12 18:53:44,009 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:53:11 2010-02-12 18:53:44,012 INFO repeater task wh300_beamplots starting 20.5 seconds late 2010-02-12 18:53:47,317 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 18:53:47,318 INFO repeater task os75bb_beamplots starting 23.8 seconds late 2010-02-12 18:53:50,811 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 18:53:50,823 INFO repeater task os75nb_beamplots starting 27.3 seconds late 2010-02-12 18:53:54,499 INFO repeater task os75nb_beamplots ended after 3.7 seconds 2010-02-12 18:53:59,500 INFO repeater task os75bb_lastensq starting 13.5 seconds late 2010-02-12 18:54:18,155 INFO repeater task os75bb_lastensq ended after 18.7 seconds 2010-02-12 18:54:18,156 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:53:39 2010-02-12 18:54:28,178 INFO repeater task posmv_hcorrstats starting 4.6 seconds late 2010-02-12 18:54:54,978 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:54:54,979 INFO repeater task posmv_hcorrstats ended after 26.8 seconds 2010-02-12 18:54:54,980 INFO repeater task phins_hcorrstats starting 31.4 seconds late 2010-02-12 18:55:16,577 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 18:55:16,577 INFO repeater task phins_hcorrstats ended after 21.6 seconds 2010-02-12 18:55:21,575 INFO repeater task wh300_lastensq starting 0.6 seconds late 2010-02-12 18:55:44,000 INFO repeater task wh300_lastensq ended after 22.4 seconds 2010-02-12 18:55:44,001 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:55:11 2010-02-12 18:55:44,005 INFO repeater task wh300_beamplots starting 20.5 seconds late 2010-02-12 18:55:47,384 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 18:55:47,385 INFO repeater task os75bb_beamplots starting 23.9 seconds late 2010-02-12 18:55:51,315 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-02-12 18:55:51,316 INFO repeater task os75nb_beamplots starting 27.8 seconds late 2010-02-12 18:55:54,541 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-02-12 18:56:09,630 INFO repeater task os75nb_lastensq starting 0.6 seconds late 2010-02-12 18:56:26,483 INFO repeater task os75nb_lastensq ended after 16.9 seconds 2010-02-12 18:56:26,484 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 18:56:01 2010-02-12 18:57:22,134 INFO repeater task wh300_lastensq starting 1.1 seconds late 2010-02-12 18:57:43,977 INFO repeater task wh300_lastensq ended after 21.8 seconds 2010-02-12 18:57:43,978 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:57:11 2010-02-12 18:57:43,982 INFO repeater task wh300_beamplots starting 20.5 seconds late 2010-02-12 18:57:47,339 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 18:57:47,355 INFO repeater task os75bb_beamplots starting 23.8 seconds late 2010-02-12 18:57:50,844 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 18:57:50,845 INFO repeater task os75nb_beamplots starting 27.3 seconds late 2010-02-12 18:57:54,473 INFO repeater task os75nb_beamplots ended after 3.6 seconds 2010-02-12 18:58:49,532 INFO repeater task os75bb_lastensq starting 0.5 seconds late 2010-02-12 18:59:10,242 INFO repeater task os75bb_lastensq ended after 20.7 seconds 2010-02-12 18:59:10,243 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 18:58:41 2010-02-12 18:59:25,274 INFO repeater task wh300_lastensq starting 4.3 seconds late 2010-02-12 18:59:54,375 INFO repeater task wh300_lastensq ended after 29.1 seconds 2010-02-12 18:59:54,497 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 18:59:11 2010-02-12 18:59:54,502 INFO repeater task wh300_beamplots starting 31.0 seconds late 2010-02-12 19:00:00,990 INFO repeater task wh300_beamplots ended after 6.5 seconds 2010-02-12 19:00:01,001 INFO repeater task os75bb_beamplots starting 37.5 seconds late 2010-02-12 19:00:04,481 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 19:00:04,482 INFO repeater task os75nb_beamplots starting 40.9 seconds late 2010-02-12 19:00:08,707 INFO repeater task os75nb_beamplots ended after 4.2 seconds 2010-02-12 19:00:08,708 INFO repeater task wh300_beamplots starting 45.2 seconds late 2010-02-12 19:00:24,954 INFO repeater task wh300_beamplots ended after 16.2 seconds 2010-02-12 19:00:24,955 INFO repeater task os75bb_beamplots starting 61.4 seconds late 2010-02-12 19:00:45,596 INFO repeater task os75bb_beamplots ended after 20.6 seconds 2010-02-12 19:00:45,597 INFO repeater task os75nb_beamplots starting 82.1 seconds late 2010-02-12 19:01:02,757 INFO repeater task os75nb_beamplots ended after 17.2 seconds 2010-02-12 19:01:02,759 INFO repeater task posmv_hcorrstats starting 99.2 seconds late 2010-02-12 19:01:34,369 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:01:34,369 INFO repeater task posmv_hcorrstats ended after 31.6 seconds 2010-02-12 19:01:34,370 INFO repeater task phins_hcorrstats starting 130.8 seconds late 2010-02-12 19:02:02,476 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:02:02,476 INFO repeater task phins_hcorrstats ended after 28.1 seconds 2010-02-12 19:02:07,486 INFO repeater task wh300_lastensq starting 46.5 seconds late 2010-02-12 19:02:29,433 INFO repeater task wh300_lastensq ended after 21.9 seconds 2010-02-12 19:02:29,434 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:01:12 2010-02-12 19:02:29,436 INFO repeater task os75nb_lastensq starting 78.4 seconds late 2010-02-12 19:02:50,096 INFO repeater task os75nb_lastensq ended after 20.7 seconds 2010-02-12 19:02:50,096 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:01:03 2010-02-12 19:02:50,098 INFO repeater task wh300_beamplots starting 86.6 seconds late 2010-02-12 19:02:53,431 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 19:02:53,432 INFO repeater task os75bb_beamplots starting 89.9 seconds late 2010-02-12 19:02:56,833 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 19:02:56,834 INFO repeater task os75nb_beamplots starting 93.3 seconds late 2010-02-12 19:03:00,719 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-02-12 19:03:25,743 INFO repeater task wh300_lastensq starting 3.7 seconds late 2010-02-12 19:03:47,346 INFO repeater task wh300_lastensq ended after 21.6 seconds 2010-02-12 19:03:47,347 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:03:12 2010-02-12 19:03:47,351 INFO repeater task wh300_beamplots starting 23.8 seconds late 2010-02-12 19:03:51,161 INFO repeater task wh300_beamplots ended after 3.8 seconds 2010-02-12 19:03:51,162 INFO repeater task os75bb_beamplots starting 27.6 seconds late 2010-02-12 19:03:54,409 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-02-12 19:03:54,416 INFO repeater task os75nb_beamplots starting 30.9 seconds late 2010-02-12 19:03:57,895 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-02-12 19:04:02,900 INFO repeater task os75bb_lastensq starting 11.9 seconds late 2010-02-12 19:04:22,151 INFO repeater task os75bb_lastensq ended after 19.3 seconds 2010-02-12 19:04:22,152 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 19:03:43 2010-02-12 19:04:27,166 INFO repeater task posmv_hcorrstats starting 3.6 seconds late 2010-02-12 19:04:54,126 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:04:54,126 INFO repeater task posmv_hcorrstats ended after 27.0 seconds 2010-02-12 19:04:54,127 INFO repeater task phins_hcorrstats starting 30.6 seconds late 2010-02-12 19:05:15,893 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:05:15,894 INFO repeater task phins_hcorrstats ended after 21.8 seconds 2010-02-12 19:05:25,907 INFO repeater task wh300_lastensq starting 3.9 seconds late 2010-02-12 19:05:47,972 INFO repeater task wh300_lastensq ended after 22.1 seconds 2010-02-12 19:05:47,973 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:05:12 2010-02-12 19:05:47,977 INFO repeater task wh300_beamplots starting 24.4 seconds late 2010-02-12 19:05:51,416 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-02-12 19:05:51,417 INFO repeater task os75bb_beamplots starting 27.9 seconds late 2010-02-12 19:05:55,167 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-02-12 19:05:55,168 INFO repeater task os75nb_beamplots starting 31.6 seconds late 2010-02-12 19:05:59,004 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-02-12 19:06:14,052 INFO repeater task os75nb_lastensq starting 1.1 seconds late 2010-02-12 19:06:31,285 INFO repeater task os75nb_lastensq ended after 17.2 seconds 2010-02-12 19:06:31,286 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:06:06 2010-02-12 19:07:26,339 INFO repeater task wh300_lastensq starting 4.3 seconds late 2010-02-12 19:07:47,743 INFO repeater task wh300_lastensq ended after 21.4 seconds 2010-02-12 19:07:47,744 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:07:12 2010-02-12 19:07:47,748 INFO repeater task wh300_beamplots starting 24.2 seconds late 2010-02-12 19:07:51,983 INFO repeater task wh300_beamplots ended after 4.2 seconds 2010-02-12 19:07:51,984 INFO repeater task os75bb_beamplots starting 28.5 seconds late 2010-02-12 19:07:55,847 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-02-12 19:07:55,857 INFO repeater task os75nb_beamplots starting 32.3 seconds late 2010-02-12 19:07:58,962 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-02-12 19:08:54,831 INFO repeater task os75bb_lastensq starting 1.8 seconds late 2010-02-12 19:09:12,277 INFO repeater task os75bb_lastensq ended after 17.4 seconds 2010-02-12 19:09:12,278 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 19:08:46 2010-02-12 19:09:22,287 INFO repeater task wh300_lastensq starting 0.3 seconds late 2010-02-12 19:09:42,317 INFO repeater task wh300_lastensq ended after 20.0 seconds 2010-02-12 19:09:42,318 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:09:13 2010-02-12 19:09:42,319 INFO repeater task wh300_quick starting 18.8 seconds late 2010-02-12 19:10:59,607 INFO repeater task wh300_quick ended after 77.3 seconds 2010-02-12 19:10:59,608 INFO repeater task wh300_3dayplots starting 96.1 seconds late 2010-02-12 19:11:13,218 INFO repeater task wh300_3dayplots ended after 13.6 seconds 2010-02-12 19:11:13,219 INFO repeater task os75bb_quick starting 109.7 seconds late 2010-02-12 19:12:09,195 INFO repeater task os75bb_quick ended after 56.0 seconds 2010-02-12 19:12:09,196 INFO repeater task os75bb_3dayplots starting 165.7 seconds late 2010-02-12 19:12:24,116 INFO repeater task os75bb_3dayplots ended after 14.9 seconds 2010-02-12 19:12:24,126 INFO repeater task os75nb_lastensq starting 68.1 seconds late 2010-02-12 19:12:41,489 INFO repeater task os75nb_lastensq ended after 17.4 seconds 2010-02-12 19:12:41,490 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:11:08 2010-02-12 19:12:41,499 INFO repeater task os75nb_quick starting 198.0 seconds late 2010-02-12 19:13:41,515 INFO repeater task os75nb_quick ended after 60.0 seconds 2010-02-12 19:13:41,525 INFO repeater task os75nb_3dayplots starting 258.0 seconds late 2010-02-12 19:14:04,170 INFO repeater task os75nb_3dayplots ended after 22.6 seconds 2010-02-12 19:14:04,171 INFO repeater task wh300_beamplots starting 280.6 seconds late 2010-02-12 19:14:07,478 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-02-12 19:14:07,479 INFO repeater Extra increment added to time_to_run 2010-02-12 19:14:07,479 INFO repeater Extra increment added to time_to_run 2010-02-12 19:14:07,480 INFO repeater task os75bb_beamplots starting 283.9 seconds late 2010-02-12 19:14:10,846 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 19:14:10,846 INFO repeater Extra increment added to time_to_run 2010-02-12 19:14:10,847 INFO repeater Extra increment added to time_to_run 2010-02-12 19:14:10,847 INFO repeater task os75nb_beamplots starting 287.3 seconds late 2010-02-12 19:14:14,720 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-02-12 19:14:14,721 INFO repeater Extra increment added to time_to_run 2010-02-12 19:14:14,721 INFO repeater Extra increment added to time_to_run 2010-02-12 19:14:14,723 INFO repeater task wh300_beamplots starting 291.2 seconds late 2010-02-12 19:14:31,865 INFO repeater task wh300_beamplots ended after 17.1 seconds 2010-02-12 19:14:31,874 INFO repeater task os75bb_beamplots starting 308.3 seconds late 2010-02-12 19:14:51,935 INFO repeater task os75bb_beamplots ended after 20.1 seconds 2010-02-12 19:14:51,951 INFO repeater task os75nb_beamplots starting 328.4 seconds late 2010-02-12 19:15:09,395 INFO repeater task os75nb_beamplots ended after 17.4 seconds 2010-02-12 19:15:09,396 INFO repeater task wh300_beamhistory starting 345.9 seconds late 2010-02-12 19:15:12,245 ERROR repeater task wh300_beamhistory returncode 1 2010-02-12 19:15:12,271 INFO repeater task wh300_beamhistory output: Traceback (most recent call last): File "/home/currents/programs/uhdas/scripts/beam_history.py", line 151, in yearbase=yearbase) File "/home/currents/programs/pycurrents/py_utils/adcp_diagnostics.py", line 734, in beam_stats_byfile data=m.read(ends=1) File "/home/currents/programs/pycurrents/adcp/rdiraw.py", line 1647, in read binlist.append(a[ilist]) IndexError: index (6407) out of range (0<=index<6406) in dimension 0 2010-02-12 19:15:12,271 INFO repeater task wh300_beamhistory ended after 2.9 seconds 2010-02-12 19:15:12,272 INFO repeater task os75bb_beamhistory starting 348.7 seconds late 2010-02-12 19:16:26,376 INFO repeater task os75bb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75bb_bin_history.png 2010-02-12 19:16:26,394 INFO repeater task os75bb_beamhistory ended after 74.1 seconds 2010-02-12 19:16:26,394 INFO repeater task os75nb_beamhistory starting 422.9 seconds late 2010-02-12 19:17:19,186 INFO repeater task os75nb_beamhistory output: Warning: invalid value encountered in divide saving file to os75nb_bin_history.png 2010-02-12 19:17:19,187 INFO repeater task os75nb_beamhistory ended after 52.8 seconds 2010-02-12 19:17:19,188 INFO repeater task posmv_hcorrstats starting 475.6 seconds late 2010-02-12 19:17:56,171 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:17:56,191 INFO repeater task posmv_hcorrstats ended after 37.0 seconds 2010-02-12 19:17:56,191 INFO repeater Extra increment added to time_to_run 2010-02-12 19:17:56,200 INFO repeater task phins_hcorrstats starting 512.6 seconds late 2010-02-12 19:18:25,549 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:18:25,550 INFO repeater task phins_hcorrstats ended after 29.3 seconds 2010-02-12 19:18:25,550 INFO repeater Extra increment added to time_to_run 2010-02-12 19:18:30,551 INFO repeater task wh300_lastensq starting 427.6 seconds late 2010-02-12 19:19:02,407 INFO repeater task wh300_lastensq ended after 31.9 seconds 2010-02-12 19:19:02,408 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:17:14 2010-02-12 19:19:02,410 INFO repeater task os75bb_lastensq starting 306.4 seconds late 2010-02-12 19:19:22,930 INFO repeater task os75bb_lastensq ended after 20.5 seconds 2010-02-12 19:19:22,931 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 19:18:50 2010-02-12 19:19:22,957 INFO repeater task os75nb_lastensq starting 185.0 seconds late 2010-02-12 19:19:40,678 INFO repeater task os75nb_lastensq ended after 17.7 seconds 2010-02-12 19:19:40,679 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:16:11 2010-02-12 19:19:40,680 INFO repeater task wh300_beamplots starting 257.2 seconds late 2010-02-12 19:19:44,399 INFO repeater task wh300_beamplots ended after 3.7 seconds 2010-02-12 19:19:44,399 INFO repeater Extra increment added to time_to_run 2010-02-12 19:19:44,399 INFO repeater Extra increment added to time_to_run 2010-02-12 19:19:44,400 INFO repeater task os75bb_beamplots starting 260.9 seconds late 2010-02-12 19:19:47,856 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-02-12 19:19:47,856 INFO repeater Extra increment added to time_to_run 2010-02-12 19:19:47,856 INFO repeater Extra increment added to time_to_run 2010-02-12 19:19:47,857 INFO repeater task os75nb_beamplots starting 264.3 seconds late 2010-02-12 19:19:51,610 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-02-12 19:19:51,611 INFO repeater Extra increment added to time_to_run 2010-02-12 19:19:51,611 INFO repeater Extra increment added to time_to_run 2010-02-12 19:19:51,618 INFO repeater task wh300_beamplots starting 28.1 seconds late 2010-02-12 19:20:08,811 INFO repeater task wh300_beamplots ended after 17.2 seconds 2010-02-12 19:20:08,825 INFO repeater task os75bb_beamplots starting 45.3 seconds late 2010-02-12 19:20:26,634 INFO repeater task os75bb_beamplots ended after 17.8 seconds 2010-02-12 19:20:26,635 INFO repeater task os75nb_beamplots starting 63.1 seconds late 2010-02-12 19:20:44,124 INFO repeater task os75nb_beamplots ended after 17.5 seconds 2010-02-12 19:20:44,126 INFO repeater task posmv_hcorrstats starting 80.6 seconds late 2010-02-12 19:21:15,709 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:21:15,710 INFO repeater task posmv_hcorrstats ended after 31.6 seconds 2010-02-12 19:21:15,719 INFO repeater task phins_hcorrstats starting 112.2 seconds late 2010-02-12 19:21:44,512 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:21:44,512 INFO repeater task phins_hcorrstats ended after 28.8 seconds 2010-02-12 19:21:49,511 INFO repeater task wh300_lastensq starting 145.5 seconds late 2010-02-12 19:22:15,654 INFO repeater task wh300_lastensq ended after 26.1 seconds 2010-02-12 19:22:15,655 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:21:14 2010-02-12 19:22:15,658 INFO repeater task os75nb_lastensq starting 54.7 seconds late 2010-02-12 19:22:37,265 INFO repeater task os75nb_lastensq ended after 21.6 seconds 2010-02-12 19:22:37,266 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:21:13 2010-02-12 19:22:37,268 INFO repeater task wh300_beamplots starting 73.7 seconds late 2010-02-12 19:22:41,428 INFO repeater task wh300_beamplots ended after 4.2 seconds 2010-02-12 19:22:41,429 INFO repeater task os75bb_beamplots starting 77.9 seconds late 2010-02-12 19:22:44,843 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 19:22:44,844 INFO repeater task os75nb_beamplots starting 81.3 seconds late 2010-02-12 19:22:49,430 INFO repeater task os75nb_beamplots ended after 4.6 seconds 2010-02-12 19:23:24,771 INFO repeater task wh300_lastensq starting 0.8 seconds late 2010-02-12 19:23:51,322 INFO repeater task wh300_lastensq ended after 26.6 seconds 2010-02-12 19:23:51,323 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:23:15 2010-02-12 19:23:51,425 INFO repeater task wh300_beamplots starting 27.9 seconds late 2010-02-12 19:23:55,035 INFO repeater task wh300_beamplots ended after 3.6 seconds 2010-02-12 19:23:55,036 INFO repeater task os75bb_beamplots starting 31.5 seconds late 2010-02-12 19:24:00,035 INFO repeater task os75bb_beamplots ended after 5.0 seconds 2010-02-12 19:24:00,036 INFO repeater task os75nb_beamplots starting 36.5 seconds late 2010-02-12 19:24:05,002 INFO repeater task os75nb_beamplots ended after 5.0 seconds 2010-02-12 19:24:10,004 INFO repeater task os75bb_lastensq starting 10.0 seconds late 2010-02-12 19:24:31,474 INFO repeater task os75bb_lastensq ended after 21.5 seconds 2010-02-12 19:24:31,475 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 19:23:53 2010-02-12 19:24:31,482 INFO repeater task posmv_hcorrstats starting 7.9 seconds late 2010-02-12 19:25:02,727 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:25:02,727 INFO repeater task posmv_hcorrstats ended after 31.2 seconds 2010-02-12 19:25:02,728 INFO repeater task phins_hcorrstats starting 39.2 seconds late 2010-02-12 19:25:31,135 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:25:31,135 INFO repeater task phins_hcorrstats ended after 28.4 seconds 2010-02-12 19:25:36,135 INFO repeater task wh300_lastensq starting 11.1 seconds late 2010-02-12 19:26:02,181 INFO repeater task wh300_lastensq ended after 26.0 seconds 2010-02-12 19:26:02,182 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:25:15 2010-02-12 19:26:02,186 INFO repeater task wh300_beamplots starting 38.7 seconds late 2010-02-12 19:26:06,065 INFO repeater task wh300_beamplots ended after 3.9 seconds 2010-02-12 19:26:06,067 INFO repeater task os75bb_beamplots starting 42.5 seconds late 2010-02-12 19:26:09,465 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-02-12 19:26:09,475 INFO repeater task os75nb_beamplots starting 45.9 seconds late 2010-02-12 19:26:12,881 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-02-12 19:26:27,901 INFO repeater task os75nb_lastensq starting 4.9 seconds late 2010-02-12 19:26:46,886 INFO repeater task os75nb_lastensq ended after 19.0 seconds 2010-02-12 19:26:46,887 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:26:15 2010-02-12 19:27:27,254 INFO repeater task wh300_lastensq starting 2.3 seconds late 2010-02-12 19:27:49,833 INFO repeater task wh300_lastensq ended after 22.6 seconds 2010-02-12 19:27:49,834 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:27:15 2010-02-12 19:27:49,838 INFO repeater task wh300_beamplots starting 26.3 seconds late 2010-02-12 19:27:54,365 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-12 19:27:54,366 INFO repeater task os75bb_beamplots starting 30.8 seconds late 2010-02-12 19:27:58,232 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-02-12 19:27:58,233 INFO repeater task os75nb_beamplots starting 34.7 seconds late 2010-02-12 19:28:02,082 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-02-12 19:29:07,649 INFO repeater task os75bb_lastensq starting 4.6 seconds late 2010-02-12 19:29:25,726 INFO repeater task os75bb_lastensq ended after 18.1 seconds 2010-02-12 19:29:25,727 INFO repeater Time from /home/data/kn197_03/proc/os75bb/load/wrote_ens_time: 2010/02/12 19:28:55 2010-02-12 19:29:25,730 INFO repeater task wh300_beamplots starting 2.2 seconds late 2010-02-12 19:29:30,231 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-02-12 19:29:30,232 INFO repeater task os75bb_beamplots starting 6.7 seconds late 2010-02-12 19:29:35,197 INFO repeater task os75bb_beamplots ended after 5.0 seconds 2010-02-12 19:29:35,198 INFO repeater task os75nb_beamplots starting 11.7 seconds late 2010-02-12 19:29:40,116 INFO repeater task os75nb_beamplots ended after 4.9 seconds 2010-02-12 19:29:40,126 INFO repeater task wh300_beamplots starting 16.6 seconds late 2010-02-12 19:30:01,137 INFO repeater task wh300_beamplots ended after 21.0 seconds 2010-02-12 19:30:01,144 INFO repeater task os75bb_beamplots starting 37.6 seconds late 2010-02-12 19:30:23,037 INFO repeater task os75bb_beamplots ended after 21.9 seconds 2010-02-12 19:30:23,039 INFO repeater task os75nb_beamplots starting 59.5 seconds late 2010-02-12 19:30:42,624 INFO repeater task os75nb_beamplots ended after 19.6 seconds 2010-02-12 19:30:42,626 INFO repeater task posmv_hcorrstats starting 79.1 seconds late 2010-02-12 19:31:13,118 INFO repeater task posmv_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:31:13,119 INFO repeater task posmv_hcorrstats ended after 30.5 seconds 2010-02-12 19:31:13,120 INFO repeater task phins_hcorrstats starting 109.6 seconds late 2010-02-12 19:31:39,424 INFO repeater task phins_hcorrstats output: failed to get position. nonmonotonic m_dday? 2010-02-12 19:31:39,424 INFO repeater task phins_hcorrstats ended after 26.3 seconds 2010-02-12 19:31:44,423 INFO repeater task wh300_lastensq starting 139.4 seconds late 2010-02-12 19:32:14,125 INFO repeater task wh300_lastensq ended after 29.7 seconds 2010-02-12 19:32:14,126 INFO repeater Time from /home/data/kn197_03/proc/wh300/load/wrote_ens_time: 2010/02/12 19:31:16 2010-02-12 19:32:14,128 INFO repeater task os75nb_lastensq starting 49.1 seconds late 2010-02-12 19:32:35,598 INFO repeater task os75nb_lastensq ended after 21.5 seconds 2010-02-12 19:32:35,600 INFO repeater Time from /home/data/kn197_03/proc/os75nb/load/wrote_ens_time: 2010/02/12 19:31:18 2010-02-12 19:32:35,602 INFO repeater task wh300_beamplots starting 72.1 seconds late 2010-02-12 19:32:39,596 INFO repeater task wh300_beamplots ended after 4.0 seconds 2010-02-12 19:32:39,597 INFO repeater task os75bb_beamplots starting 76.1 seconds late 2010-02-12 19:32:43,519 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-02-12 19:32:43,529 INFO repeater task os75nb_beamplots starting 80.0 seconds late 2010-02-12 19:32:48,476 INFO repeater task os75nb_beamplots ended after 4.9 seconds 2010-02-12 19:33:23,982 INFO repeater task wh300_beamplots starting 0.5 seconds late 2010-02-12 19:33:28,750 INFO repeater task wh300_beamplots ended after 4.8 seconds 2010-02-12 19:33:28,751 INFO repeater task os75bb_beamplots starting 5.2 seconds late 2010-02-12 19:33:34,465 INFO repeater task os75bb_beamplots ended after 5.7 seconds 2010-02-12 19:33:34,466 INFO repeater task os75nb_beamplots starting 10.9 seconds late 2010-02-12 19:33:38,940 INFO repeater task os75nb_beamplots ended after 4.5 seconds 2010-02-12 19:33:43,946 INFO repeater task wh300_lastensq starting 17.9 seconds late 2010-02-12 19:34:05,767 INFO repeater task wh300_lastensq ended after 21.8 seconds