2010-04-21 18:36:45,409 INFO repeater task os75bb_beamplots ended after 3.8 seconds 2010-04-21 18:36:45,460 INFO repeater Extra increment added to time_to_run 2010-04-21 18:36:45,460 INFO repeater Extra increment added to time_to_run 2010-04-21 18:36:45,469 INFO repeater task os75nb_beamplots starting 266.4 seconds late 2010-04-21 18:36:48,918 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 18:36:48,919 INFO repeater Extra increment added to time_to_run 2010-04-21 18:36:48,919 INFO repeater Extra increment added to time_to_run 2010-04-21 18:36:48,920 INFO repeater task wh300_beamplots starting 269.8 seconds late 2010-04-21 18:37:03,131 INFO repeater task wh300_beamplots ended after 14.2 seconds 2010-04-21 18:37:03,139 INFO repeater task os75bb_beamplots starting 284.0 seconds late 2010-04-21 18:37:18,764 INFO repeater task os75bb_beamplots ended after 15.6 seconds 2010-04-21 18:37:18,765 INFO repeater task os75nb_beamplots starting 299.6 seconds late 2010-04-21 18:37:33,973 INFO repeater task os75nb_beamplots ended after 15.2 seconds 2010-04-21 18:37:33,975 INFO repeater task posmv_hcorrstats starting 314.8 seconds late 2010-04-21 18:37:56,736 INFO repeater task posmv_hcorrstats ended after 22.8 seconds 2010-04-21 18:37:56,751 INFO repeater Extra increment added to time_to_run 2010-04-21 18:37:56,753 INFO repeater task phins_hcorrstats starting 337.6 seconds late 2010-04-21 18:38:25,689 INFO repeater task phins_hcorrstats ended after 28.9 seconds 2010-04-21 18:38:25,690 INFO repeater Extra increment added to time_to_run 2010-04-21 18:38:30,688 INFO repeater task wh300_lastensq starting 335.7 seconds late 2010-04-21 18:38:51,095 INFO repeater task wh300_lastensq ended after 20.4 seconds 2010-04-21 18:38:51,096 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:36:47 2010-04-21 18:38:51,099 INFO repeater task os75nb_lastensq starting 121.1 seconds late 2010-04-21 18:39:06,922 INFO repeater task os75nb_lastensq ended after 15.8 seconds 2010-04-21 18:39:06,923 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 18:36:43 2010-04-21 18:39:06,925 INFO repeater task wh300_beamplots starting 47.8 seconds late 2010-04-21 18:39:09,981 INFO repeater task wh300_beamplots ended after 3.1 seconds 2010-04-21 18:39:09,982 INFO repeater task os75bb_beamplots starting 50.9 seconds late 2010-04-21 18:39:13,524 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 18:39:13,531 INFO repeater task os75nb_beamplots starting 54.4 seconds late 2010-04-21 18:39:17,438 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-04-21 18:39:22,440 INFO repeater task wh300_lastensq starting 25.4 seconds late 2010-04-21 18:39:40,793 INFO repeater task wh300_lastensq ended after 18.4 seconds 2010-04-21 18:39:40,794 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:38:47 2010-04-21 18:39:40,796 INFO repeater task os75bb_lastensq starting 36.8 seconds late 2010-04-21 18:39:58,940 INFO repeater task os75bb_lastensq ended after 18.1 seconds 2010-04-21 18:39:58,941 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 18:38:57 2010-04-21 18:40:23,971 INFO repeater task wh300_beamplots starting 4.9 seconds late 2010-04-21 18:40:27,015 INFO repeater task wh300_beamplots ended after 3.0 seconds 2010-04-21 18:40:27,025 INFO repeater task os75bb_beamplots starting 7.9 seconds late 2010-04-21 18:40:30,436 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 18:40:30,437 INFO repeater task os75nb_beamplots starting 11.3 seconds late 2010-04-21 18:40:33,744 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:40:58,984 INFO repeater task wh300_lastensq starting 2.0 seconds late 2010-04-21 18:41:18,147 INFO repeater task wh300_lastensq ended after 19.2 seconds 2010-04-21 18:41:18,148 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:40:48 2010-04-21 18:41:53,515 INFO repeater task os75nb_lastensq starting 0.5 seconds late 2010-04-21 18:42:11,417 INFO repeater task os75nb_lastensq ended after 17.9 seconds 2010-04-21 18:42:11,418 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 18:41:46 2010-04-21 18:42:21,475 INFO repeater task wh300_beamplots starting 2.4 seconds late 2010-04-21 18:42:24,722 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 18:42:24,723 INFO repeater task os75bb_beamplots starting 5.6 seconds late 2010-04-21 18:42:27,931 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 18:42:27,932 INFO repeater task os75nb_beamplots starting 8.8 seconds late 2010-04-21 18:42:31,236 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:42:31,238 INFO repeater task wh300_beamplots starting 12.1 seconds late 2010-04-21 18:42:45,730 INFO repeater task wh300_beamplots ended after 14.5 seconds 2010-04-21 18:42:45,731 INFO repeater task os75bb_beamplots starting 26.6 seconds late 2010-04-21 18:43:01,621 INFO repeater task os75bb_beamplots ended after 15.9 seconds 2010-04-21 18:43:01,622 INFO repeater task os75nb_beamplots starting 42.5 seconds late 2010-04-21 18:43:18,078 INFO repeater task os75nb_beamplots ended after 16.5 seconds 2010-04-21 18:43:18,100 INFO repeater task posmv_hcorrstats starting 59.0 seconds late 2010-04-21 18:43:38,441 INFO repeater task posmv_hcorrstats ended after 20.3 seconds 2010-04-21 18:43:38,450 INFO repeater task phins_hcorrstats starting 79.3 seconds late 2010-04-21 18:43:57,945 INFO repeater task phins_hcorrstats ended after 19.5 seconds 2010-04-21 18:44:02,948 INFO repeater task wh300_lastensq starting 64.9 seconds late 2010-04-21 18:44:23,478 INFO repeater task wh300_lastensq ended after 20.5 seconds 2010-04-21 18:44:23,479 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:42:48 2010-04-21 18:44:23,481 INFO repeater task os75bb_lastensq starting 16.5 seconds late 2010-04-21 18:44:39,913 INFO repeater task os75bb_lastensq ended after 16.4 seconds 2010-04-21 18:44:39,914 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 18:44:00 2010-04-21 18:44:39,917 INFO repeater task wh300_beamplots starting 20.8 seconds late 2010-04-21 18:44:43,249 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 18:44:43,249 INFO repeater task os75bb_beamplots starting 24.1 seconds late 2010-04-21 18:44:46,360 INFO repeater task os75bb_beamplots ended after 3.1 seconds 2010-04-21 18:44:46,362 INFO repeater task os75nb_beamplots starting 27.2 seconds late 2010-04-21 18:44:50,142 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-04-21 18:45:00,156 INFO repeater task wh300_lastensq starting 2.2 seconds late 2010-04-21 18:45:20,654 INFO repeater task wh300_lastensq ended after 20.5 seconds 2010-04-21 18:45:20,655 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:44:49 2010-04-21 18:46:21,604 INFO repeater task wh300_beamplots starting 2.5 seconds late 2010-04-21 18:46:25,002 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 18:46:25,003 INFO repeater task os75bb_beamplots starting 5.9 seconds late 2010-04-21 18:46:28,070 INFO repeater task os75bb_beamplots ended after 3.1 seconds 2010-04-21 18:46:28,071 INFO repeater task os75nb_beamplots starting 9.0 seconds late 2010-04-21 18:46:31,401 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:46:56,439 INFO repeater task os75nb_lastensq starting 0.4 seconds late 2010-04-21 18:47:12,313 INFO repeater task os75nb_lastensq ended after 15.9 seconds 2010-04-21 18:47:12,314 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 18:46:50 2010-04-21 18:47:17,316 INFO repeater task wh300_lastensq starting 18.3 seconds late 2010-04-21 18:47:36,802 INFO repeater task wh300_lastensq ended after 19.5 seconds 2010-04-21 18:47:36,802 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:46:49 2010-04-21 18:47:36,810 INFO repeater task posmv_hcorrstats starting 17.7 seconds late 2010-04-21 18:47:57,670 INFO repeater task posmv_hcorrstats ended after 20.9 seconds 2010-04-21 18:47:57,671 INFO repeater task phins_hcorrstats starting 38.5 seconds late 2010-04-21 18:48:18,428 INFO repeater task phins_hcorrstats ended after 20.8 seconds 2010-04-21 18:48:23,431 INFO repeater task wh300_beamplots starting 4.3 seconds late 2010-04-21 18:48:26,729 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 18:48:26,730 INFO repeater task os75bb_beamplots starting 7.6 seconds late 2010-04-21 18:48:29,973 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 18:48:29,974 INFO repeater task os75nb_beamplots starting 10.9 seconds late 2010-04-21 18:48:33,276 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:49:03,300 INFO repeater task wh300_lastensq starting 4.3 seconds late 2010-04-21 18:49:21,751 INFO repeater task wh300_lastensq ended after 18.5 seconds 2010-04-21 18:49:21,752 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:48:50 2010-04-21 18:49:21,753 INFO repeater task os75bb_lastensq starting 11.8 seconds late 2010-04-21 18:49:39,385 INFO repeater task os75bb_lastensq ended after 17.6 seconds 2010-04-21 18:49:39,385 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 18:49:04 2010-04-21 18:50:19,587 INFO repeater task wh300_beamplots starting 0.5 seconds late 2010-04-21 18:50:22,504 INFO repeater task wh300_beamplots ended after 2.9 seconds 2010-04-21 18:50:22,505 INFO repeater task os75bb_beamplots starting 3.4 seconds late 2010-04-21 18:50:26,508 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-04-21 18:50:26,509 INFO repeater task os75nb_beamplots starting 7.4 seconds late 2010-04-21 18:50:29,813 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:51:00,068 INFO repeater task wh300_lastensq starting 0.1 seconds late 2010-04-21 18:51:18,326 INFO repeater task wh300_lastensq ended after 18.3 seconds 2010-04-21 18:51:18,327 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:50:50 2010-04-21 18:52:03,378 INFO repeater task os75nb_lastensq starting 3.4 seconds late 2010-04-21 18:52:21,323 INFO repeater task os75nb_lastensq ended after 17.9 seconds 2010-04-21 18:52:21,324 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 18:51:53 2010-04-21 18:52:21,325 INFO repeater task wh300_beamplots starting 2.2 seconds late 2010-04-21 18:52:24,685 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 18:52:24,686 INFO repeater task os75bb_beamplots starting 5.6 seconds late 2010-04-21 18:52:28,295 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-04-21 18:52:28,305 INFO repeater task os75nb_beamplots starting 9.2 seconds late 2010-04-21 18:52:31,562 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:52:31,563 INFO repeater task wh300_beamplots starting 12.4 seconds late 2010-04-21 18:52:45,694 INFO repeater task wh300_beamplots ended after 14.1 seconds 2010-04-21 18:52:45,703 INFO repeater task os75bb_beamplots starting 26.6 seconds late 2010-04-21 18:53:01,477 INFO repeater task os75bb_beamplots ended after 15.8 seconds 2010-04-21 18:53:01,478 INFO repeater task os75nb_beamplots starting 42.4 seconds late 2010-04-21 18:53:16,804 INFO repeater task os75nb_beamplots ended after 15.3 seconds 2010-04-21 18:53:16,806 INFO repeater task posmv_hcorrstats starting 57.7 seconds late 2010-04-21 18:53:37,623 INFO repeater task posmv_hcorrstats ended after 20.8 seconds 2010-04-21 18:53:37,624 INFO repeater task phins_hcorrstats starting 78.5 seconds late 2010-04-21 18:53:56,322 INFO repeater task phins_hcorrstats ended after 18.7 seconds 2010-04-21 18:54:01,320 INFO repeater task wh300_lastensq starting 61.3 seconds late 2010-04-21 18:54:22,086 INFO repeater task wh300_lastensq ended after 20.8 seconds 2010-04-21 18:54:22,087 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:52:51 2010-04-21 18:54:22,089 INFO repeater task os75bb_lastensq starting 8.1 seconds late 2010-04-21 18:54:39,687 INFO repeater task os75bb_lastensq ended after 17.6 seconds 2010-04-21 18:54:39,688 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 18:54:07 2010-04-21 18:54:39,691 INFO repeater task wh300_beamplots starting 20.6 seconds late 2010-04-21 18:54:42,588 INFO repeater task wh300_beamplots ended after 2.9 seconds 2010-04-21 18:54:42,589 INFO repeater task os75bb_beamplots starting 23.5 seconds late 2010-04-21 18:54:46,113 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 18:54:46,114 INFO repeater task os75nb_beamplots starting 27.0 seconds late 2010-04-21 18:54:49,441 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 18:55:04,572 INFO repeater task wh300_lastensq starting 3.6 seconds late 2010-04-21 18:55:24,321 INFO repeater task wh300_lastensq ended after 19.7 seconds 2010-04-21 18:55:24,322 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:54:51 2010-04-21 18:56:19,902 INFO repeater task wh300_beamplots starting 0.8 seconds late 2010-04-21 18:56:23,013 INFO repeater task wh300_beamplots ended after 3.1 seconds 2010-04-21 18:56:23,014 INFO repeater task os75bb_beamplots starting 3.9 seconds late 2010-04-21 18:56:26,882 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-04-21 18:56:26,883 INFO repeater task os75nb_beamplots starting 7.8 seconds late 2010-04-21 18:56:30,063 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-04-21 18:57:05,372 INFO repeater task wh300_lastensq starting 4.4 seconds late 2010-04-21 18:57:24,979 INFO repeater task wh300_lastensq ended after 19.6 seconds 2010-04-21 18:57:24,980 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:56:51 2010-04-21 18:57:25,028 INFO repeater task os75nb_lastensq starting 22.0 seconds late 2010-04-21 18:57:42,221 INFO repeater task os75nb_lastensq ended after 17.2 seconds 2010-04-21 18:57:42,222 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 18:56:56 2010-04-21 18:57:42,227 INFO repeater task posmv_hcorrstats starting 23.1 seconds late 2010-04-21 18:58:03,068 INFO repeater task posmv_hcorrstats ended after 20.8 seconds 2010-04-21 18:58:03,069 INFO repeater task phins_hcorrstats starting 43.9 seconds late 2010-04-21 18:58:23,981 INFO repeater task phins_hcorrstats ended after 20.9 seconds 2010-04-21 18:58:28,983 INFO repeater task wh300_beamplots starting 9.9 seconds late 2010-04-21 18:58:31,895 INFO repeater task wh300_beamplots ended after 2.9 seconds 2010-04-21 18:58:31,896 INFO repeater task os75bb_beamplots starting 12.8 seconds late 2010-04-21 18:58:35,393 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 18:58:35,394 INFO repeater task os75nb_beamplots starting 16.3 seconds late 2010-04-21 18:58:38,793 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 18:59:04,040 INFO repeater task wh300_lastensq starting 3.0 seconds late 2010-04-21 18:59:22,539 INFO repeater task wh300_lastensq ended after 18.5 seconds 2010-04-21 18:59:22,540 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 18:58:52 2010-04-21 18:59:22,542 INFO repeater task os75bb_lastensq starting 5.5 seconds late 2010-04-21 18:59:39,737 INFO repeater task os75bb_lastensq ended after 17.2 seconds 2010-04-21 18:59:39,738 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 18:59:10 2010-04-21 19:00:20,469 INFO repeater task wh300_beamplots starting 1.4 seconds late 2010-04-21 19:00:23,432 INFO repeater task wh300_beamplots ended after 3.0 seconds 2010-04-21 19:00:23,433 INFO repeater task os75bb_beamplots starting 4.3 seconds late 2010-04-21 19:00:27,084 INFO repeater task os75bb_beamplots ended after 3.7 seconds 2010-04-21 19:00:27,085 INFO repeater task os75nb_beamplots starting 8.0 seconds late 2010-04-21 19:00:30,891 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-04-21 19:01:06,108 INFO repeater task wh300_lastensq starting 4.1 seconds late 2010-04-21 19:01:24,628 INFO repeater task wh300_lastensq ended after 18.5 seconds 2010-04-21 19:01:24,629 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:00:52 2010-04-21 19:02:09,752 INFO repeater task os75nb_lastensq starting 3.8 seconds late 2010-04-21 19:02:32,801 INFO repeater task os75nb_lastensq ended after 23.0 seconds 2010-04-21 19:02:32,802 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:01:59 2010-04-21 19:02:32,803 INFO repeater task os75nb_quick starting 13.7 seconds late 2010-04-21 19:03:33,368 INFO repeater task os75nb_quick ended after 60.6 seconds 2010-04-21 19:03:33,377 INFO repeater task os75nb_3dayplots starting 74.3 seconds late 2010-04-21 19:03:57,231 INFO repeater task os75nb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 19:03:57,231 INFO repeater task os75nb_3dayplots ended after 23.9 seconds 2010-04-21 19:03:57,232 INFO repeater task wh300_beamplots starting 98.1 seconds late 2010-04-21 19:04:00,634 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 19:04:00,635 INFO repeater task os75bb_beamplots starting 101.5 seconds late 2010-04-21 19:04:04,550 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-04-21 19:04:04,551 INFO repeater task os75nb_beamplots starting 105.4 seconds late 2010-04-21 19:04:08,429 INFO repeater task os75nb_beamplots ended after 3.9 seconds 2010-04-21 19:04:08,452 INFO repeater task wh300_beamplots starting 109.3 seconds late 2010-04-21 19:04:25,011 INFO repeater task wh300_beamplots ended after 16.6 seconds 2010-04-21 19:04:25,013 INFO repeater task os75bb_beamplots starting 125.9 seconds late 2010-04-21 19:04:41,731 INFO repeater task os75bb_beamplots ended after 16.7 seconds 2010-04-21 19:04:41,733 INFO repeater task os75nb_beamplots starting 142.6 seconds late 2010-04-21 19:04:57,144 INFO repeater task os75nb_beamplots ended after 15.4 seconds 2010-04-21 19:04:57,144 INFO repeater task wh300_beamhistory starting 158.0 seconds late 2010-04-21 19:06:53,214 INFO repeater task wh300_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to wh300_bin_history.png 2010-04-21 19:06:53,214 INFO repeater task wh300_beamhistory ended after 116.1 seconds 2010-04-21 19:06:53,215 INFO repeater task os75bb_beamhistory starting 274.1 seconds late 2010-04-21 19:07:56,540 INFO repeater task os75bb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75bb_bin_history.png 2010-04-21 19:07:56,540 INFO repeater task os75bb_beamhistory ended after 63.3 seconds 2010-04-21 19:07:56,541 INFO repeater task os75nb_beamhistory starting 337.4 seconds late 2010-04-21 19:08:43,378 INFO repeater task os75nb_beamhistory output: saving file to os75nb_bin_history.png 2010-04-21 19:08:43,379 INFO repeater task os75nb_beamhistory ended after 46.8 seconds 2010-04-21 19:08:43,380 INFO repeater task posmv_hcorrstats starting 384.2 seconds late 2010-04-21 19:09:32,585 INFO repeater task posmv_hcorrstats ended after 49.2 seconds 2010-04-21 19:09:32,600 INFO repeater Extra increment added to time_to_run 2010-04-21 19:09:32,601 INFO repeater task phins_hcorrstats starting 433.5 seconds late 2010-04-21 19:10:06,605 INFO repeater task phins_hcorrstats ended after 34.0 seconds 2010-04-21 19:10:06,605 INFO repeater Extra increment added to time_to_run 2010-04-21 19:10:11,605 INFO repeater task wh300_lastensq starting 429.6 seconds late 2010-04-21 19:10:40,933 INFO repeater task wh300_lastensq ended after 29.3 seconds 2010-04-21 19:10:40,934 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:08:53 2010-04-21 19:10:40,935 INFO repeater task wh300_quick starting 501.8 seconds late 2010-04-21 19:11:50,910 INFO repeater task wh300_quick ended after 70.0 seconds 2010-04-21 19:11:50,911 INFO repeater task wh300_3dayplots starting 571.8 seconds late 2010-04-21 19:12:24,090 INFO repeater task wh300_3dayplots ended after 33.2 seconds 2010-04-21 19:12:24,101 INFO repeater task os75bb_lastensq starting 484.1 seconds late 2010-04-21 19:12:44,957 INFO repeater task os75bb_lastensq ended after 20.9 seconds 2010-04-21 19:12:44,957 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:09:17 2010-04-21 19:12:44,958 INFO repeater task os75bb_quick starting 625.8 seconds late 2010-04-21 19:13:44,154 INFO repeater task os75bb_quick ended after 59.2 seconds 2010-04-21 19:13:44,155 INFO repeater task os75bb_3dayplots starting 685.0 seconds late 2010-04-21 19:13:56,909 INFO repeater task os75bb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 19:13:56,910 INFO repeater task os75bb_3dayplots ended after 12.8 seconds 2010-04-21 19:13:56,910 INFO repeater task os75nb_lastensq starting 407.9 seconds late 2010-04-21 19:14:17,856 INFO repeater task os75nb_lastensq ended after 20.9 seconds 2010-04-21 19:14:17,857 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:12:06 2010-04-21 19:14:17,880 INFO repeater task wh300_beamplots starting 598.8 seconds late 2010-04-21 19:14:21,617 INFO repeater task wh300_beamplots ended after 3.7 seconds 2010-04-21 19:14:21,618 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:21,618 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:21,618 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:21,618 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:21,619 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:21,619 INFO repeater task os75bb_beamplots starting 602.5 seconds late 2010-04-21 19:14:25,134 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 19:14:25,135 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:25,135 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:25,135 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:25,135 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:25,136 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:25,136 INFO repeater task os75nb_beamplots starting 606.0 seconds late 2010-04-21 19:14:28,550 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 19:14:28,551 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:28,551 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:28,551 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:28,551 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:28,552 INFO repeater Extra increment added to time_to_run 2010-04-21 19:14:28,552 INFO repeater task wh300_beamplots starting 129.4 seconds late 2010-04-21 19:14:45,143 INFO repeater task wh300_beamplots ended after 16.6 seconds 2010-04-21 19:14:45,144 INFO repeater task os75bb_beamplots starting 146.0 seconds late 2010-04-21 19:15:00,729 INFO repeater task os75bb_beamplots ended after 15.6 seconds 2010-04-21 19:15:00,730 INFO repeater task os75nb_beamplots starting 161.6 seconds late 2010-04-21 19:15:16,579 INFO repeater task os75nb_beamplots ended after 15.8 seconds 2010-04-21 19:15:16,581 INFO repeater task posmv_hcorrstats starting 177.4 seconds late 2010-04-21 19:15:45,117 INFO repeater task posmv_hcorrstats ended after 28.5 seconds 2010-04-21 19:15:45,130 INFO repeater task phins_hcorrstats starting 206.0 seconds late 2010-04-21 19:16:06,814 INFO repeater task phins_hcorrstats ended after 21.7 seconds 2010-04-21 19:16:11,816 INFO repeater task wh300_lastensq starting 308.8 seconds late 2010-04-21 19:16:37,952 INFO repeater task wh300_lastensq ended after 26.1 seconds 2010-04-21 19:16:37,953 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:14:54 2010-04-21 19:16:37,954 INFO repeater task os75bb_lastensq starting 131.0 seconds late 2010-04-21 19:16:54,883 INFO repeater task os75bb_lastensq ended after 16.9 seconds 2010-04-21 19:16:54,884 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:14:20 2010-04-21 19:16:54,939 INFO repeater task wh300_beamplots starting 35.8 seconds late 2010-04-21 19:16:58,205 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 19:16:58,206 INFO repeater task os75bb_beamplots starting 39.1 seconds late 2010-04-21 19:17:01,678 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 19:17:01,679 INFO repeater task os75nb_beamplots starting 42.6 seconds late 2010-04-21 19:17:04,606 INFO repeater task os75nb_beamplots ended after 2.9 seconds 2010-04-21 19:17:09,608 INFO repeater task wh300_lastensq starting 5.6 seconds late 2010-04-21 19:17:29,181 INFO repeater task wh300_lastensq ended after 19.6 seconds 2010-04-21 19:17:29,182 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:16:55 2010-04-21 19:17:29,184 INFO repeater task os75nb_lastensq starting 13.2 seconds late 2010-04-21 19:17:47,100 INFO repeater task os75nb_lastensq ended after 17.9 seconds 2010-04-21 19:17:47,101 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:17:09 2010-04-21 19:17:47,106 INFO repeater task posmv_hcorrstats starting 28.0 seconds late 2010-04-21 19:18:08,031 INFO repeater task posmv_hcorrstats ended after 20.9 seconds 2010-04-21 19:18:08,032 INFO repeater task phins_hcorrstats starting 48.9 seconds late 2010-04-21 19:18:29,708 INFO repeater task phins_hcorrstats ended after 21.7 seconds 2010-04-21 19:18:34,772 INFO repeater task wh300_beamplots starting 15.7 seconds late 2010-04-21 19:18:38,176 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 19:18:38,177 INFO repeater task os75bb_beamplots starting 19.1 seconds late 2010-04-21 19:18:41,684 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 19:18:41,686 INFO repeater task os75nb_beamplots starting 22.6 seconds late 2010-04-21 19:18:45,081 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 19:19:05,204 INFO repeater task wh300_lastensq starting 0.2 seconds late 2010-04-21 19:19:25,097 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 19:19:25,098 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:18:55 2010-04-21 19:19:30,105 INFO repeater task os75bb_lastensq starting 0.1 seconds late 2010-04-21 19:19:47,360 INFO repeater task os75bb_lastensq ended after 17.3 seconds 2010-04-21 19:19:47,361 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:19:23 2010-04-21 19:20:22,399 INFO repeater task wh300_beamplots starting 3.3 seconds late 2010-04-21 19:20:25,699 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 19:20:25,700 INFO repeater task os75bb_beamplots starting 6.6 seconds late 2010-04-21 19:20:28,711 INFO repeater task os75bb_beamplots ended after 3.0 seconds 2010-04-21 19:20:28,712 INFO repeater task os75nb_beamplots starting 9.6 seconds late 2010-04-21 19:20:32,235 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-04-21 19:21:07,412 INFO repeater task wh300_lastensq starting 2.4 seconds late 2010-04-21 19:21:27,275 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 19:21:27,275 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:20:55 2010-04-21 19:22:22,346 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2010-04-21 19:22:39,207 INFO repeater task os75nb_lastensq ended after 16.9 seconds 2010-04-21 19:22:39,208 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:22:12 2010-04-21 19:22:39,224 INFO repeater task wh300_beamplots starting 20.1 seconds late 2010-04-21 19:22:43,767 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-04-21 19:22:43,768 INFO repeater task os75bb_beamplots starting 24.7 seconds late 2010-04-21 19:22:47,205 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 19:22:47,206 INFO repeater task os75nb_beamplots starting 28.1 seconds late 2010-04-21 19:22:50,230 INFO repeater task os75nb_beamplots ended after 3.0 seconds 2010-04-21 19:22:50,231 INFO repeater task wh300_beamplots starting 31.1 seconds late 2010-04-21 19:23:04,603 INFO repeater task wh300_beamplots ended after 14.4 seconds 2010-04-21 19:23:04,604 INFO repeater task os75bb_beamplots starting 45.5 seconds late 2010-04-21 19:23:20,896 INFO repeater task os75bb_beamplots ended after 16.3 seconds 2010-04-21 19:23:20,897 INFO repeater task os75nb_beamplots starting 61.8 seconds late 2010-04-21 19:23:36,332 INFO repeater task os75nb_beamplots ended after 15.4 seconds 2010-04-21 19:23:36,334 INFO repeater task posmv_hcorrstats starting 77.2 seconds late 2010-04-21 19:23:57,359 INFO repeater task posmv_hcorrstats ended after 21.0 seconds 2010-04-21 19:23:57,360 INFO repeater task phins_hcorrstats starting 98.2 seconds late 2010-04-21 19:24:16,569 INFO repeater task phins_hcorrstats ended after 19.2 seconds 2010-04-21 19:24:21,572 INFO repeater task wh300_lastensq starting 76.6 seconds late 2010-04-21 19:24:41,132 INFO repeater task wh300_lastensq ended after 19.6 seconds 2010-04-21 19:24:41,133 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:22:56 2010-04-21 19:24:41,134 INFO repeater task os75bb_lastensq starting 8.1 seconds late 2010-04-21 19:24:58,399 INFO repeater task os75bb_lastensq ended after 17.3 seconds 2010-04-21 19:24:58,400 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:24:26 2010-04-21 19:24:58,403 INFO repeater task wh300_beamplots starting 39.3 seconds late 2010-04-21 19:25:01,799 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 19:25:01,800 INFO repeater task os75bb_beamplots starting 42.7 seconds late 2010-04-21 19:25:05,183 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 19:25:05,184 INFO repeater task os75nb_beamplots starting 46.1 seconds late 2010-04-21 19:25:08,246 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-04-21 19:25:13,248 INFO repeater task wh300_lastensq starting 7.2 seconds late 2010-04-21 19:25:33,103 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 19:25:33,104 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:24:56 2010-04-21 19:26:23,550 INFO repeater task wh300_beamplots starting 4.4 seconds late 2010-04-21 19:26:26,809 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 19:26:26,810 INFO repeater task os75bb_beamplots starting 7.7 seconds late 2010-04-21 19:26:29,951 INFO repeater task os75bb_beamplots ended after 3.1 seconds 2010-04-21 19:26:29,953 INFO repeater task os75nb_beamplots starting 10.8 seconds late 2010-04-21 19:26:33,354 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 19:27:08,732 INFO repeater task wh300_lastensq starting 2.7 seconds late 2010-04-21 19:27:28,595 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 19:27:28,596 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:26:56 2010-04-21 19:27:28,598 INFO repeater task os75nb_lastensq starting 6.6 seconds late 2010-04-21 19:27:45,421 INFO repeater task os75nb_lastensq ended after 16.8 seconds 2010-04-21 19:27:45,422 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:27:16 2010-04-21 19:27:45,427 INFO repeater task posmv_hcorrstats starting 26.3 seconds late 2010-04-21 19:28:05,971 INFO repeater task posmv_hcorrstats ended after 20.5 seconds 2010-04-21 19:28:05,972 INFO repeater task phins_hcorrstats starting 46.8 seconds late 2010-04-21 19:28:27,520 INFO repeater task phins_hcorrstats ended after 21.5 seconds 2010-04-21 19:28:32,523 INFO repeater task wh300_beamplots starting 13.4 seconds late 2010-04-21 19:28:36,030 INFO repeater task wh300_beamplots ended after 3.5 seconds 2010-04-21 19:28:36,031 INFO repeater task os75bb_beamplots starting 16.9 seconds late 2010-04-21 19:28:39,392 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 19:28:39,403 INFO repeater task os75nb_beamplots starting 20.3 seconds late 2010-04-21 19:28:42,397 INFO repeater task os75nb_beamplots ended after 3.0 seconds 2010-04-21 19:29:07,420 INFO repeater task wh300_lastensq starting 1.4 seconds late 2010-04-21 19:29:27,545 INFO repeater task wh300_lastensq ended after 20.1 seconds 2010-04-21 19:29:27,546 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:28:57 2010-04-21 19:29:37,561 INFO repeater task os75bb_lastensq starting 1.6 seconds late 2010-04-21 19:29:55,015 INFO repeater task os75bb_lastensq ended after 17.5 seconds 2010-04-21 19:29:55,016 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:29:29 2010-04-21 19:30:20,397 INFO repeater task wh300_beamplots starting 1.3 seconds late 2010-04-21 19:30:24,744 INFO repeater task wh300_beamplots ended after 4.3 seconds 2010-04-21 19:30:24,754 INFO repeater task os75bb_beamplots starting 5.6 seconds late 2010-04-21 19:30:28,799 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-04-21 19:30:28,800 INFO repeater task os75nb_beamplots starting 9.7 seconds late 2010-04-21 19:30:32,093 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 19:31:07,128 INFO repeater task wh300_lastensq starting 0.1 seconds late 2010-04-21 19:31:26,655 INFO repeater task wh300_lastensq ended after 19.5 seconds 2010-04-21 19:31:26,656 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:30:57 2010-04-21 19:32:22,024 INFO repeater task wh300_quick starting 2.9 seconds late 2010-04-21 19:33:34,892 INFO repeater task wh300_quick ended after 72.9 seconds 2010-04-21 19:33:34,893 INFO repeater task wh300_3dayplots starting 75.8 seconds late 2010-04-21 19:34:06,251 INFO repeater task wh300_3dayplots ended after 31.4 seconds 2010-04-21 19:34:06,253 INFO repeater task os75bb_quick starting 107.1 seconds late 2010-04-21 19:35:02,442 INFO repeater task os75bb_quick ended after 56.2 seconds 2010-04-21 19:35:02,443 INFO repeater task os75bb_3dayplots starting 163.3 seconds late 2010-04-21 19:35:15,453 INFO repeater task os75bb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 19:35:15,453 INFO repeater task os75bb_3dayplots ended after 13.0 seconds 2010-04-21 19:35:15,454 INFO repeater task os75nb_lastensq starting 169.5 seconds late 2010-04-21 19:35:33,001 INFO repeater task os75nb_lastensq ended after 17.5 seconds 2010-04-21 19:35:33,002 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:32:19 2010-04-21 19:35:33,003 INFO repeater task os75nb_quick starting 193.9 seconds late 2010-04-21 19:36:20,961 INFO repeater task os75nb_quick ended after 48.0 seconds 2010-04-21 19:36:20,962 INFO repeater task os75nb_3dayplots starting 241.8 seconds late 2010-04-21 19:36:43,099 INFO repeater task os75nb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 19:36:43,100 INFO repeater task os75nb_3dayplots ended after 22.1 seconds 2010-04-21 19:36:43,101 INFO repeater task wh300_beamplots starting 264.0 seconds late 2010-04-21 19:36:46,321 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 19:36:46,321 INFO repeater Extra increment added to time_to_run 2010-04-21 19:36:46,321 INFO repeater Extra increment added to time_to_run 2010-04-21 19:36:46,322 INFO repeater task os75bb_beamplots starting 267.2 seconds late 2010-04-21 19:36:49,715 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 19:36:49,715 INFO repeater Extra increment added to time_to_run 2010-04-21 19:36:49,716 INFO repeater Extra increment added to time_to_run 2010-04-21 19:36:49,719 INFO repeater task os75nb_beamplots starting 270.6 seconds late 2010-04-21 19:36:52,943 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-04-21 19:36:52,943 INFO repeater Extra increment added to time_to_run 2010-04-21 19:36:52,944 INFO repeater Extra increment added to time_to_run 2010-04-21 19:36:52,950 INFO repeater task wh300_beamplots starting 273.8 seconds late 2010-04-21 19:37:07,827 INFO repeater task wh300_beamplots ended after 14.9 seconds 2010-04-21 19:37:07,828 INFO repeater task os75bb_beamplots starting 288.7 seconds late 2010-04-21 19:37:23,132 INFO repeater task os75bb_beamplots ended after 15.3 seconds 2010-04-21 19:37:23,133 INFO repeater task os75nb_beamplots starting 304.0 seconds late 2010-04-21 19:37:39,024 INFO repeater task os75nb_beamplots ended after 15.9 seconds 2010-04-21 19:37:39,052 INFO repeater task posmv_hcorrstats starting 319.9 seconds late 2010-04-21 19:38:02,247 INFO repeater task posmv_hcorrstats ended after 23.2 seconds 2010-04-21 19:38:02,262 INFO repeater Extra increment added to time_to_run 2010-04-21 19:38:02,263 INFO repeater task phins_hcorrstats starting 343.1 seconds late 2010-04-21 19:38:28,880 INFO repeater task phins_hcorrstats ended after 26.6 seconds 2010-04-21 19:38:28,881 INFO repeater Extra increment added to time_to_run 2010-04-21 19:38:33,880 INFO repeater task wh300_lastensq starting 326.9 seconds late 2010-04-21 19:38:58,871 INFO repeater task wh300_lastensq ended after 25.0 seconds 2010-04-21 19:38:58,873 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:36:58 2010-04-21 19:38:58,874 INFO repeater task os75bb_lastensq starting 259.9 seconds late 2010-04-21 19:39:18,849 INFO repeater task os75bb_lastensq ended after 20.0 seconds 2010-04-21 19:39:18,850 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:39:00 2010-04-21 19:39:18,852 INFO repeater task os75nb_lastensq starting 109.9 seconds late 2010-04-21 19:39:35,513 INFO repeater task os75nb_lastensq ended after 16.7 seconds 2010-04-21 19:39:35,514 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:37:22 2010-04-21 19:39:35,515 INFO repeater task wh300_beamplots starting 76.4 seconds late 2010-04-21 19:39:39,057 INFO repeater task wh300_beamplots ended after 3.5 seconds 2010-04-21 19:39:39,058 INFO repeater task os75bb_beamplots starting 79.9 seconds late 2010-04-21 19:39:42,430 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 19:39:42,431 INFO repeater task os75nb_beamplots starting 83.3 seconds late 2010-04-21 19:39:45,502 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-04-21 19:39:50,504 INFO repeater task wh300_lastensq starting 42.5 seconds late 2010-04-21 19:40:10,727 INFO repeater task wh300_lastensq ended after 20.2 seconds 2010-04-21 19:40:10,728 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:38:59 2010-04-21 19:40:20,743 INFO repeater task wh300_beamplots starting 1.6 seconds late 2010-04-21 19:40:23,965 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 19:40:23,974 INFO repeater task os75bb_beamplots starting 4.9 seconds late 2010-04-21 19:40:27,874 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-04-21 19:40:27,875 INFO repeater task os75nb_beamplots starting 8.8 seconds late 2010-04-21 19:40:31,062 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-04-21 19:41:11,244 INFO repeater task wh300_lastensq starting 2.2 seconds late 2010-04-21 19:41:31,357 INFO repeater task wh300_lastensq ended after 20.1 seconds 2010-04-21 19:41:31,357 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:40:59 2010-04-21 19:42:21,593 INFO repeater task wh300_beamplots starting 2.5 seconds late 2010-04-21 19:42:24,840 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 19:42:24,841 INFO repeater task os75bb_beamplots starting 5.7 seconds late 2010-04-21 19:42:28,403 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-04-21 19:42:28,404 INFO repeater task os75nb_beamplots starting 9.3 seconds late 2010-04-21 19:42:31,908 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-04-21 19:42:31,909 INFO repeater task wh300_beamplots starting 12.8 seconds late 2010-04-21 19:42:48,641 INFO repeater task wh300_beamplots ended after 16.7 seconds 2010-04-21 19:42:48,642 INFO repeater task os75bb_beamplots starting 29.5 seconds late 2010-04-21 19:43:05,521 INFO repeater task os75bb_beamplots ended after 16.9 seconds 2010-04-21 19:43:05,522 INFO repeater task os75nb_beamplots starting 46.4 seconds late 2010-04-21 19:43:21,012 INFO repeater task os75nb_beamplots ended after 15.5 seconds 2010-04-21 19:43:21,015 INFO repeater task posmv_hcorrstats starting 61.9 seconds late 2010-04-21 19:43:42,190 INFO repeater task posmv_hcorrstats ended after 21.2 seconds 2010-04-21 19:43:42,191 INFO repeater task phins_hcorrstats starting 83.1 seconds late 2010-04-21 19:44:02,758 INFO repeater task phins_hcorrstats ended after 20.6 seconds 2010-04-21 19:44:07,756 INFO repeater task wh300_lastensq starting 58.8 seconds late 2010-04-21 19:44:27,607 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 19:44:27,608 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:42:59 2010-04-21 19:44:27,609 INFO repeater task os75bb_lastensq starting 17.6 seconds late 2010-04-21 19:44:47,769 INFO repeater task os75bb_lastensq ended after 20.2 seconds 2010-04-21 19:44:47,770 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:44:08 2010-04-21 19:44:47,772 INFO repeater task os75nb_lastensq starting 135.8 seconds late 2010-04-21 19:45:05,650 INFO repeater task os75nb_lastensq ended after 17.9 seconds 2010-04-21 19:45:05,651 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:42:25 2010-04-21 19:45:05,654 INFO repeater task wh300_beamplots starting 46.5 seconds late 2010-04-21 19:45:08,890 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 19:45:08,892 INFO repeater task os75bb_beamplots starting 49.8 seconds late 2010-04-21 19:45:12,069 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 19:45:12,070 INFO repeater task os75nb_beamplots starting 53.0 seconds late 2010-04-21 19:45:15,415 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 19:45:20,469 INFO repeater task wh300_lastensq starting 11.5 seconds late 2010-04-21 19:45:40,898 INFO repeater task wh300_lastensq ended after 20.4 seconds 2010-04-21 19:45:40,899 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:45:00 2010-04-21 19:46:21,549 INFO repeater task wh300_beamplots starting 2.4 seconds late 2010-04-21 19:46:24,868 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 19:46:24,869 INFO repeater task os75bb_beamplots starting 5.8 seconds late 2010-04-21 19:46:28,093 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 19:46:28,094 INFO repeater task os75nb_beamplots starting 9.0 seconds late 2010-04-21 19:46:31,406 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 19:47:12,036 INFO repeater task wh300_lastensq starting 2.0 seconds late 2010-04-21 19:47:32,136 INFO repeater task wh300_lastensq ended after 20.1 seconds 2010-04-21 19:47:32,137 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:47:00 2010-04-21 19:47:32,145 INFO repeater task posmv_hcorrstats starting 13.0 seconds late 2010-04-21 19:47:53,192 INFO repeater task posmv_hcorrstats ended after 21.0 seconds 2010-04-21 19:47:53,193 INFO repeater task phins_hcorrstats starting 34.1 seconds late 2010-04-21 19:48:12,336 INFO repeater task phins_hcorrstats ended after 19.1 seconds 2010-04-21 19:48:17,338 INFO repeater task os75nb_lastensq starting 42.3 seconds late 2010-04-21 19:48:33,836 INFO repeater task os75nb_lastensq ended after 16.5 seconds 2010-04-21 19:48:33,837 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:47:28 2010-04-21 19:48:33,838 INFO repeater task wh300_beamplots starting 14.7 seconds late 2010-04-21 19:48:37,309 INFO repeater task wh300_beamplots ended after 3.5 seconds 2010-04-21 19:48:37,322 INFO repeater task os75bb_beamplots starting 18.2 seconds late 2010-04-21 19:48:42,601 INFO repeater task os75bb_beamplots ended after 5.3 seconds 2010-04-21 19:48:42,621 INFO repeater task os75nb_beamplots starting 23.5 seconds late 2010-04-21 19:48:46,367 INFO repeater task os75nb_beamplots ended after 3.7 seconds 2010-04-21 19:49:11,388 INFO repeater task wh300_lastensq starting 1.4 seconds late 2010-04-21 19:49:31,395 INFO repeater task wh300_lastensq ended after 20.0 seconds 2010-04-21 19:49:31,395 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:49:00 2010-04-21 19:49:31,397 INFO repeater task os75bb_lastensq starting 13.4 seconds late 2010-04-21 19:49:49,300 INFO repeater task os75bb_lastensq ended after 17.9 seconds 2010-04-21 19:49:49,301 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:49:11 2010-04-21 19:50:19,343 INFO repeater task wh300_beamplots starting 0.2 seconds late 2010-04-21 19:50:22,375 INFO repeater task wh300_beamplots ended after 3.0 seconds 2010-04-21 19:50:22,376 INFO repeater task os75bb_beamplots starting 3.3 seconds late 2010-04-21 19:50:25,776 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 19:50:25,777 INFO repeater task os75nb_beamplots starting 6.7 seconds late 2010-04-21 19:50:29,129 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 19:51:14,528 INFO repeater task wh300_lastensq starting 4.5 seconds late 2010-04-21 19:51:34,354 INFO repeater task wh300_lastensq ended after 19.8 seconds 2010-04-21 19:51:34,355 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:51:01 2010-04-21 19:52:20,307 INFO repeater task wh300_beamplots starting 1.2 seconds late 2010-04-21 19:52:23,412 INFO repeater task wh300_beamplots ended after 3.1 seconds 2010-04-21 19:52:23,413 INFO repeater task os75bb_beamplots starting 4.3 seconds late 2010-04-21 19:52:26,710 INFO repeater task os75bb_beamplots ended after 3.3 seconds 2010-04-21 19:52:26,711 INFO repeater task os75nb_beamplots starting 7.6 seconds late 2010-04-21 19:52:29,757 INFO repeater task os75nb_beamplots ended after 3.0 seconds 2010-04-21 19:52:29,758 INFO repeater task wh300_beamplots starting 10.6 seconds late 2010-04-21 19:52:47,366 INFO repeater task wh300_beamplots ended after 17.6 seconds 2010-04-21 19:52:47,367 INFO repeater task os75bb_beamplots starting 28.2 seconds late 2010-04-21 19:53:02,760 INFO repeater task os75bb_beamplots ended after 15.4 seconds 2010-04-21 19:53:02,761 INFO repeater task os75nb_beamplots starting 43.6 seconds late 2010-04-21 19:53:18,531 INFO repeater task os75nb_beamplots ended after 15.8 seconds 2010-04-21 19:53:18,534 INFO repeater task posmv_hcorrstats starting 59.4 seconds late 2010-04-21 19:53:39,900 INFO repeater task posmv_hcorrstats ended after 21.4 seconds 2010-04-21 19:53:39,901 INFO repeater task phins_hcorrstats starting 80.8 seconds late 2010-04-21 19:53:58,599 INFO repeater task phins_hcorrstats ended after 18.7 seconds 2010-04-21 19:54:03,604 INFO repeater task wh300_lastensq starting 52.6 seconds late 2010-04-21 19:54:24,573 INFO repeater task wh300_lastensq ended after 21.0 seconds 2010-04-21 19:54:24,574 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:53:01 2010-04-21 19:54:24,575 INFO repeater task os75bb_lastensq starting 3.6 seconds late 2010-04-21 19:54:43,116 INFO repeater task os75bb_lastensq ended after 18.5 seconds 2010-04-21 19:54:43,116 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:54:14 2010-04-21 19:54:43,147 INFO repeater task os75nb_lastensq starting 125.1 seconds late 2010-04-21 19:55:01,869 INFO repeater task os75nb_lastensq ended after 18.7 seconds 2010-04-21 19:55:01,870 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:52:32 2010-04-21 19:55:01,872 INFO repeater task wh300_beamplots starting 42.8 seconds late 2010-04-21 19:55:05,133 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 19:55:05,134 INFO repeater task os75bb_beamplots starting 46.0 seconds late 2010-04-21 19:55:09,060 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-04-21 19:55:09,068 INFO repeater task os75nb_beamplots starting 50.0 seconds late 2010-04-21 19:55:12,373 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 19:55:17,376 INFO repeater task wh300_lastensq starting 6.4 seconds late 2010-04-21 19:55:38,166 INFO repeater task wh300_lastensq ended after 20.8 seconds 2010-04-21 19:55:38,167 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:55:01 2010-04-21 19:56:23,259 INFO repeater task wh300_beamplots starting 4.1 seconds late 2010-04-21 19:56:26,471 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 19:56:26,472 INFO repeater task os75bb_beamplots starting 7.4 seconds late 2010-04-21 19:56:29,708 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 19:56:29,709 INFO repeater task os75nb_beamplots starting 10.6 seconds late 2010-04-21 19:56:33,162 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-04-21 19:57:13,664 INFO repeater task wh300_lastensq starting 2.7 seconds late 2010-04-21 19:57:33,555 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 19:57:33,555 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:57:02 2010-04-21 19:57:33,565 INFO repeater task posmv_hcorrstats starting 14.4 seconds late 2010-04-21 19:57:54,917 INFO repeater task posmv_hcorrstats ended after 21.4 seconds 2010-04-21 19:57:54,919 INFO repeater task phins_hcorrstats starting 35.8 seconds late 2010-04-21 19:58:14,106 INFO repeater task phins_hcorrstats ended after 19.2 seconds 2010-04-21 19:58:19,163 INFO repeater task os75nb_lastensq starting 37.2 seconds late 2010-04-21 19:58:36,109 INFO repeater task os75nb_lastensq ended after 16.9 seconds 2010-04-21 19:58:36,110 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 19:57:35 2010-04-21 19:58:36,111 INFO repeater task wh300_beamplots starting 17.0 seconds late 2010-04-21 19:58:39,795 INFO repeater task wh300_beamplots ended after 3.7 seconds 2010-04-21 19:58:39,796 INFO repeater task os75bb_beamplots starting 20.7 seconds late 2010-04-21 19:58:44,068 INFO repeater task os75bb_beamplots ended after 4.3 seconds 2010-04-21 19:58:44,069 INFO repeater task os75nb_beamplots starting 25.0 seconds late 2010-04-21 19:58:48,812 INFO repeater task os75nb_beamplots ended after 4.7 seconds 2010-04-21 19:59:14,067 INFO repeater task wh300_lastensq starting 2.1 seconds late 2010-04-21 19:59:33,586 INFO repeater task wh300_lastensq ended after 19.5 seconds 2010-04-21 19:59:33,587 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 19:59:02 2010-04-21 19:59:33,588 INFO repeater task os75bb_lastensq starting 9.6 seconds late 2010-04-21 19:59:51,510 INFO repeater task os75bb_lastensq ended after 17.9 seconds 2010-04-21 19:59:51,511 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 19:59:18 2010-04-21 20:00:21,987 INFO repeater task wh300_beamplots starting 2.9 seconds late 2010-04-21 20:00:25,217 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:00:25,218 INFO repeater task os75bb_beamplots starting 6.1 seconds late 2010-04-21 20:00:28,453 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 20:00:28,454 INFO repeater task os75nb_beamplots starting 9.3 seconds late 2010-04-21 20:00:31,740 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 20:01:12,080 INFO repeater task wh300_lastensq starting 0.1 seconds late 2010-04-21 20:01:32,902 INFO repeater task wh300_lastensq ended after 20.8 seconds 2010-04-21 20:01:32,903 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:01:03 2010-04-21 20:02:23,371 INFO repeater task wh300_quick starting 4.3 seconds late 2010-04-21 20:03:54,605 INFO repeater task wh300_quick ended after 91.2 seconds 2010-04-21 20:03:54,622 INFO repeater task wh300_3dayplots starting 95.5 seconds late 2010-04-21 20:04:25,564 INFO repeater task wh300_3dayplots ended after 30.9 seconds 2010-04-21 20:04:25,566 INFO repeater task os75bb_quick starting 126.5 seconds late 2010-04-21 20:05:26,715 INFO repeater task os75bb_quick ended after 61.1 seconds 2010-04-21 20:05:26,716 INFO repeater task os75bb_3dayplots starting 187.6 seconds late 2010-04-21 20:05:39,146 INFO repeater task os75bb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 20:05:39,146 INFO repeater task os75bb_3dayplots ended after 12.4 seconds 2010-04-21 20:05:39,147 INFO repeater task os75nb_lastensq starting 174.1 seconds late 2010-04-21 20:05:57,628 INFO repeater task os75nb_lastensq ended after 18.5 seconds 2010-04-21 20:05:57,629 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:02:38 2010-04-21 20:05:57,630 INFO repeater task os75nb_quick starting 218.5 seconds late 2010-04-21 20:06:45,849 INFO repeater task os75nb_quick ended after 48.2 seconds 2010-04-21 20:06:45,850 INFO repeater task os75nb_3dayplots starting 266.7 seconds late 2010-04-21 20:07:07,743 INFO repeater task os75nb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 20:07:07,743 INFO repeater task os75nb_3dayplots ended after 21.9 seconds 2010-04-21 20:07:07,757 INFO repeater task wh300_beamplots starting 288.6 seconds late 2010-04-21 20:07:11,045 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 20:07:11,046 INFO repeater Extra increment added to time_to_run 2010-04-21 20:07:11,046 INFO repeater Extra increment added to time_to_run 2010-04-21 20:07:11,046 INFO repeater task os75bb_beamplots starting 291.9 seconds late 2010-04-21 20:07:14,948 INFO repeater task os75bb_beamplots ended after 3.9 seconds 2010-04-21 20:07:14,949 INFO repeater Extra increment added to time_to_run 2010-04-21 20:07:14,949 INFO repeater Extra increment added to time_to_run 2010-04-21 20:07:14,950 INFO repeater task os75nb_beamplots starting 295.8 seconds late 2010-04-21 20:07:18,299 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 20:07:18,299 INFO repeater Extra increment added to time_to_run 2010-04-21 20:07:18,299 INFO repeater Extra increment added to time_to_run 2010-04-21 20:07:18,300 INFO repeater task wh300_beamplots starting 299.2 seconds late 2010-04-21 20:07:32,341 INFO repeater task wh300_beamplots ended after 14.0 seconds 2010-04-21 20:07:32,342 INFO repeater task os75bb_beamplots starting 313.2 seconds late 2010-04-21 20:07:48,533 INFO repeater task os75bb_beamplots ended after 16.2 seconds 2010-04-21 20:07:48,546 INFO repeater task os75nb_beamplots starting 329.4 seconds late 2010-04-21 20:08:03,814 INFO repeater task os75nb_beamplots ended after 15.3 seconds 2010-04-21 20:08:03,827 INFO repeater task wh300_beamhistory starting 344.7 seconds late 2010-04-21 20:09:57,586 INFO repeater task wh300_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to wh300_bin_history.png 2010-04-21 20:09:57,586 INFO repeater task wh300_beamhistory ended after 113.8 seconds 2010-04-21 20:09:57,587 INFO repeater task os75bb_beamhistory starting 458.5 seconds late 2010-04-21 20:11:03,107 INFO repeater task os75bb_beamhistory output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide Warning: invalid value encountered in divide /usr/lib/python2.5/site-packages/numpy/ma/core.py:3740: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") saving file to os75bb_bin_history.png 2010-04-21 20:11:03,107 INFO repeater task os75bb_beamhistory ended after 65.5 seconds 2010-04-21 20:11:03,108 INFO repeater task os75nb_beamhistory starting 524.0 seconds late 2010-04-21 20:11:46,093 INFO repeater task os75nb_beamhistory output: saving file to os75nb_bin_history.png 2010-04-21 20:11:46,093 INFO repeater task os75nb_beamhistory ended after 43.0 seconds 2010-04-21 20:11:46,103 INFO repeater task posmv_hcorrstats starting 567.0 seconds late 2010-04-21 20:12:33,075 INFO repeater task posmv_hcorrstats ended after 47.0 seconds 2010-04-21 20:12:33,088 INFO repeater Extra increment added to time_to_run 2010-04-21 20:12:33,089 INFO repeater Extra increment added to time_to_run 2010-04-21 20:12:33,089 INFO repeater task phins_hcorrstats starting 614.0 seconds late 2010-04-21 20:13:07,400 INFO repeater task phins_hcorrstats ended after 34.3 seconds 2010-04-21 20:13:07,400 INFO repeater Extra increment added to time_to_run 2010-04-21 20:13:07,400 INFO repeater Extra increment added to time_to_run 2010-04-21 20:13:12,400 INFO repeater task wh300_lastensq starting 599.4 seconds late 2010-04-21 20:13:43,617 INFO repeater task wh300_lastensq ended after 31.2 seconds 2010-04-21 20:13:43,618 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:13:05 2010-04-21 20:13:43,620 INFO repeater task os75bb_lastensq starting 555.6 seconds late 2010-04-21 20:14:03,858 INFO repeater task os75bb_lastensq ended after 20.2 seconds 2010-04-21 20:14:03,859 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:09:24 2010-04-21 20:14:03,860 INFO repeater task os75nb_lastensq starting 375.9 seconds late 2010-04-21 20:14:23,356 INFO repeater task os75nb_lastensq ended after 19.5 seconds 2010-04-21 20:14:23,357 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:12:45 2010-04-21 20:14:23,359 INFO repeater task wh300_beamplots starting 364.2 seconds late 2010-04-21 20:14:26,551 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:14:26,552 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:26,552 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:26,553 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:26,554 INFO repeater task os75bb_beamplots starting 367.4 seconds late 2010-04-21 20:14:30,054 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 20:14:30,055 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:30,055 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:30,055 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:30,056 INFO repeater task os75nb_beamplots starting 370.9 seconds late 2010-04-21 20:14:33,377 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 20:14:33,377 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:33,378 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:33,378 INFO repeater Extra increment added to time_to_run 2010-04-21 20:14:33,378 INFO repeater task wh300_beamplots starting 134.3 seconds late 2010-04-21 20:14:47,626 INFO repeater task wh300_beamplots ended after 14.2 seconds 2010-04-21 20:14:47,634 INFO repeater task os75bb_beamplots starting 148.5 seconds late 2010-04-21 20:15:06,220 INFO repeater task os75bb_beamplots ended after 18.6 seconds 2010-04-21 20:15:06,221 INFO repeater task os75nb_beamplots starting 167.1 seconds late 2010-04-21 20:15:22,628 INFO repeater task os75nb_beamplots ended after 16.4 seconds 2010-04-21 20:15:27,628 INFO repeater task wh300_lastensq starting 12.6 seconds late 2010-04-21 20:15:46,617 INFO repeater task wh300_lastensq ended after 19.0 seconds 2010-04-21 20:15:46,618 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:15:06 2010-04-21 20:15:46,620 INFO repeater task os75bb_lastensq starting 72.6 seconds late 2010-04-21 20:16:03,542 INFO repeater task os75bb_lastensq ended after 16.9 seconds 2010-04-21 20:16:03,543 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:14:27 2010-04-21 20:16:23,575 INFO repeater task wh300_beamplots starting 4.5 seconds late 2010-04-21 20:16:26,963 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 20:16:26,965 INFO repeater task os75bb_beamplots starting 7.8 seconds late 2010-04-21 20:16:30,379 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 20:16:30,380 INFO repeater task os75nb_beamplots starting 11.3 seconds late 2010-04-21 20:16:33,477 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-04-21 20:17:18,592 INFO repeater task wh300_lastensq starting 2.6 seconds late 2010-04-21 20:17:37,769 INFO repeater task wh300_lastensq ended after 19.2 seconds 2010-04-21 20:17:37,770 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:17:06 2010-04-21 20:17:37,778 INFO repeater task posmv_hcorrstats starting 18.6 seconds late 2010-04-21 20:17:59,127 INFO repeater task posmv_hcorrstats ended after 21.3 seconds 2010-04-21 20:17:59,129 INFO repeater task phins_hcorrstats starting 40.0 seconds late 2010-04-21 20:18:17,692 INFO repeater task phins_hcorrstats ended after 18.6 seconds 2010-04-21 20:18:22,694 INFO repeater task os75nb_lastensq starting 27.7 seconds late 2010-04-21 20:18:39,863 INFO repeater task os75nb_lastensq ended after 17.2 seconds 2010-04-21 20:18:39,864 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:17:48 2010-04-21 20:18:39,866 INFO repeater task wh300_beamplots starting 20.8 seconds late 2010-04-21 20:18:43,197 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 20:18:43,198 INFO repeater task os75bb_beamplots starting 24.1 seconds late 2010-04-21 20:18:46,335 INFO repeater task os75bb_beamplots ended after 3.1 seconds 2010-04-21 20:18:46,336 INFO repeater task os75nb_beamplots starting 27.2 seconds late 2010-04-21 20:18:49,702 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 20:19:19,880 INFO repeater task wh300_lastensq starting 3.9 seconds late 2010-04-21 20:19:39,125 INFO repeater task wh300_lastensq ended after 19.2 seconds 2010-04-21 20:19:39,126 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:19:06 2010-04-21 20:19:39,127 INFO repeater task os75bb_lastensq starting 2.1 seconds late 2010-04-21 20:19:55,207 INFO repeater task os75bb_lastensq ended after 16.1 seconds 2010-04-21 20:19:55,207 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:19:31 2010-04-21 20:20:20,446 INFO repeater task wh300_beamplots starting 1.3 seconds late 2010-04-21 20:20:23,616 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:20:23,628 INFO repeater task os75bb_beamplots starting 4.5 seconds late 2010-04-21 20:20:28,148 INFO repeater task os75bb_beamplots ended after 4.5 seconds 2010-04-21 20:20:28,149 INFO repeater task os75nb_beamplots starting 9.0 seconds late 2010-04-21 20:20:31,568 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 20:21:16,860 INFO repeater task wh300_lastensq starting 0.9 seconds late 2010-04-21 20:21:36,868 INFO repeater task wh300_lastensq ended after 20.0 seconds 2010-04-21 20:21:36,869 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:21:07 2010-04-21 20:22:22,397 INFO repeater task wh300_beamplots starting 3.3 seconds late 2010-04-21 20:22:25,557 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:22:25,567 INFO repeater task os75bb_beamplots starting 6.5 seconds late 2010-04-21 20:22:29,804 INFO repeater task os75bb_beamplots ended after 4.2 seconds 2010-04-21 20:22:29,812 INFO repeater task os75nb_beamplots starting 10.7 seconds late 2010-04-21 20:22:33,484 INFO repeater task os75nb_beamplots ended after 3.7 seconds 2010-04-21 20:22:33,485 INFO repeater task wh300_beamplots starting 14.4 seconds late 2010-04-21 20:22:47,765 INFO repeater task wh300_beamplots ended after 14.3 seconds 2010-04-21 20:22:47,766 INFO repeater task os75bb_beamplots starting 28.6 seconds late 2010-04-21 20:23:06,143 INFO repeater task os75bb_beamplots ended after 18.4 seconds 2010-04-21 20:23:06,147 INFO repeater task os75nb_beamplots starting 47.0 seconds late 2010-04-21 20:23:21,364 INFO repeater task os75nb_beamplots ended after 15.2 seconds 2010-04-21 20:23:21,366 INFO repeater task posmv_hcorrstats starting 62.2 seconds late 2010-04-21 20:23:43,443 INFO repeater task posmv_hcorrstats ended after 22.1 seconds 2010-04-21 20:23:43,444 INFO repeater task phins_hcorrstats starting 84.3 seconds late 2010-04-21 20:24:01,547 INFO repeater task phins_hcorrstats ended after 18.1 seconds 2010-04-21 20:24:06,548 INFO repeater task wh300_lastensq starting 49.5 seconds late 2010-04-21 20:24:25,354 INFO repeater task wh300_lastensq ended after 18.8 seconds 2010-04-21 20:24:25,354 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:23:07 2010-04-21 20:24:25,357 INFO repeater task os75nb_lastensq starting 87.4 seconds late 2010-04-21 20:24:41,834 INFO repeater task os75nb_lastensq ended after 16.5 seconds 2010-04-21 20:24:41,835 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:22:51 2010-04-21 20:24:41,837 INFO repeater task wh300_beamplots starting 22.7 seconds late 2010-04-21 20:24:45,234 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 20:24:45,235 INFO repeater task os75bb_beamplots starting 26.1 seconds late 2010-04-21 20:24:48,863 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-04-21 20:24:48,868 INFO repeater task os75nb_beamplots starting 29.8 seconds late 2010-04-21 20:24:51,903 INFO repeater task os75nb_beamplots ended after 3.0 seconds 2010-04-21 20:24:57,004 INFO repeater task os75bb_lastensq starting 16.0 seconds late 2010-04-21 20:25:15,828 INFO repeater task os75bb_lastensq ended after 18.8 seconds 2010-04-21 20:25:15,829 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:24:34 2010-04-21 20:25:20,836 INFO repeater task wh300_lastensq starting 3.8 seconds late 2010-04-21 20:25:40,918 INFO repeater task wh300_lastensq ended after 20.1 seconds 2010-04-21 20:25:40,919 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:25:08 2010-04-21 20:26:21,260 INFO repeater task wh300_beamplots starting 2.1 seconds late 2010-04-21 20:26:24,302 INFO repeater task wh300_beamplots ended after 3.0 seconds 2010-04-21 20:26:24,303 INFO repeater task os75bb_beamplots starting 5.2 seconds late 2010-04-21 20:26:27,735 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 20:26:27,736 INFO repeater task os75nb_beamplots starting 8.6 seconds late 2010-04-21 20:26:31,031 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 20:27:21,469 INFO repeater task wh300_lastensq starting 3.5 seconds late 2010-04-21 20:27:41,392 INFO repeater task wh300_lastensq ended after 19.9 seconds 2010-04-21 20:27:41,393 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:27:08 2010-04-21 20:27:41,401 INFO repeater task posmv_hcorrstats starting 22.3 seconds late 2010-04-21 20:28:01,746 INFO repeater task posmv_hcorrstats ended after 20.3 seconds 2010-04-21 20:28:01,747 INFO repeater task phins_hcorrstats starting 42.6 seconds late 2010-04-21 20:28:20,708 INFO repeater task phins_hcorrstats ended after 19.0 seconds 2010-04-21 20:28:25,710 INFO repeater task os75nb_lastensq starting 24.7 seconds late 2010-04-21 20:28:42,065 INFO repeater task os75nb_lastensq ended after 16.4 seconds 2010-04-21 20:28:42,066 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:27:54 2010-04-21 20:28:42,067 INFO repeater task wh300_beamplots starting 23.0 seconds late 2010-04-21 20:28:45,382 INFO repeater task wh300_beamplots ended after 3.3 seconds 2010-04-21 20:28:45,382 INFO repeater task os75bb_beamplots starting 26.3 seconds late 2010-04-21 20:28:48,751 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 20:28:48,752 INFO repeater task os75nb_beamplots starting 29.6 seconds late 2010-04-21 20:28:51,867 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-04-21 20:29:22,048 INFO repeater task wh300_lastensq starting 4.0 seconds late 2010-04-21 20:29:42,307 INFO repeater task wh300_lastensq ended after 20.3 seconds 2010-04-21 20:29:42,308 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:29:09 2010-04-21 20:29:47,352 INFO repeater task os75bb_lastensq starting 3.4 seconds late 2010-04-21 20:30:03,727 INFO repeater task os75bb_lastensq ended after 16.4 seconds 2010-04-21 20:30:03,728 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:29:37 2010-04-21 20:30:23,759 INFO repeater task wh300_beamplots starting 4.6 seconds late 2010-04-21 20:30:27,122 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 20:30:27,123 INFO repeater task os75bb_beamplots starting 8.0 seconds late 2010-04-21 20:30:30,535 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 20:30:30,536 INFO repeater task os75nb_beamplots starting 11.4 seconds late 2010-04-21 20:30:33,650 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-04-21 20:31:19,172 INFO repeater task wh300_lastensq starting 0.2 seconds late 2010-04-21 20:31:38,139 INFO repeater task wh300_lastensq ended after 19.0 seconds 2010-04-21 20:31:38,140 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:31:09 2010-04-21 20:32:23,716 INFO repeater task wh300_quick starting 4.6 seconds late 2010-04-21 20:33:36,566 INFO repeater task wh300_quick ended after 72.9 seconds 2010-04-21 20:33:36,567 INFO repeater task wh300_3dayplots starting 77.5 seconds late 2010-04-21 20:34:07,943 INFO repeater task wh300_3dayplots ended after 31.4 seconds 2010-04-21 20:34:07,944 INFO repeater task os75bb_quick starting 108.8 seconds late 2010-04-21 20:35:07,071 INFO repeater task os75bb_quick ended after 59.1 seconds 2010-04-21 20:35:07,080 INFO repeater task os75bb_3dayplots starting 168.0 seconds late 2010-04-21 20:35:21,635 INFO repeater task os75bb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 20:35:21,636 INFO repeater task os75bb_3dayplots ended after 14.6 seconds 2010-04-21 20:35:21,637 INFO repeater task os75nb_lastensq starting 137.6 seconds late 2010-04-21 20:35:37,409 INFO repeater task os75nb_lastensq ended after 15.8 seconds 2010-04-21 20:35:37,410 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:32:58 2010-04-21 20:35:37,411 INFO repeater task os75nb_quick starting 198.3 seconds late 2010-04-21 20:36:26,035 INFO repeater task os75nb_quick ended after 48.6 seconds 2010-04-21 20:36:26,036 INFO repeater task os75nb_3dayplots starting 246.9 seconds late 2010-04-21 20:36:45,023 INFO repeater task os75nb_3dayplots output: Warning: invalid value encountered in divide Warning: invalid value encountered in divide 2010-04-21 20:36:45,024 INFO repeater task os75nb_3dayplots ended after 19.0 seconds 2010-04-21 20:36:45,025 INFO repeater task wh300_beamplots starting 265.9 seconds late 2010-04-21 20:36:49,519 INFO repeater task wh300_beamplots ended after 4.5 seconds 2010-04-21 20:36:49,519 INFO repeater Extra increment added to time_to_run 2010-04-21 20:36:49,520 INFO repeater Extra increment added to time_to_run 2010-04-21 20:36:49,525 INFO repeater task os75bb_beamplots starting 270.4 seconds late 2010-04-21 20:36:53,103 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-04-21 20:36:53,103 INFO repeater Extra increment added to time_to_run 2010-04-21 20:36:53,104 INFO repeater Extra increment added to time_to_run 2010-04-21 20:36:53,105 INFO repeater task os75nb_beamplots starting 274.0 seconds late 2010-04-21 20:36:56,231 INFO repeater task os75nb_beamplots ended after 3.1 seconds 2010-04-21 20:36:56,232 INFO repeater Extra increment added to time_to_run 2010-04-21 20:36:56,232 INFO repeater Extra increment added to time_to_run 2010-04-21 20:36:56,233 INFO repeater task wh300_beamplots starting 277.1 seconds late 2010-04-21 20:37:13,331 INFO repeater task wh300_beamplots ended after 17.1 seconds 2010-04-21 20:37:13,333 INFO repeater task os75bb_beamplots starting 294.2 seconds late 2010-04-21 20:37:28,998 INFO repeater task os75bb_beamplots ended after 15.7 seconds 2010-04-21 20:37:29,008 INFO repeater task os75nb_beamplots starting 309.9 seconds late 2010-04-21 20:37:44,207 INFO repeater task os75nb_beamplots ended after 15.2 seconds 2010-04-21 20:37:44,209 INFO repeater task posmv_hcorrstats starting 325.1 seconds late 2010-04-21 20:38:07,074 INFO repeater task posmv_hcorrstats ended after 22.9 seconds 2010-04-21 20:38:07,091 INFO repeater Extra increment added to time_to_run 2010-04-21 20:38:07,092 INFO repeater task phins_hcorrstats starting 348.0 seconds late 2010-04-21 20:38:35,735 INFO repeater task phins_hcorrstats ended after 28.6 seconds 2010-04-21 20:38:35,736 INFO repeater Extra increment added to time_to_run 2010-04-21 20:38:40,736 INFO repeater task wh300_lastensq starting 321.7 seconds late 2010-04-21 20:39:01,398 INFO repeater task wh300_lastensq ended after 20.7 seconds 2010-04-21 20:39:01,398 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:37:11 2010-04-21 20:39:01,402 INFO repeater task os75bb_lastensq starting 254.4 seconds late 2010-04-21 20:39:22,990 INFO repeater task os75bb_lastensq ended after 21.6 seconds 2010-04-21 20:39:22,991 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:39:03 2010-04-21 20:39:22,993 INFO repeater task os75nb_lastensq starting 75.0 seconds late 2010-04-21 20:39:39,133 INFO repeater task os75nb_lastensq ended after 16.1 seconds 2010-04-21 20:39:39,134 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:38:01 2010-04-21 20:39:39,135 INFO repeater task wh300_beamplots starting 80.0 seconds late 2010-04-21 20:39:42,528 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 20:39:42,529 INFO repeater task os75bb_beamplots starting 83.4 seconds late 2010-04-21 20:39:45,723 INFO repeater task os75bb_beamplots ended after 3.2 seconds 2010-04-21 20:39:45,725 INFO repeater task os75nb_beamplots starting 86.6 seconds late 2010-04-21 20:39:49,275 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-04-21 20:39:54,360 INFO repeater task wh300_lastensq starting 33.4 seconds late 2010-04-21 20:40:15,913 INFO repeater task wh300_lastensq ended after 21.6 seconds 2010-04-21 20:40:15,914 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:39:11 2010-04-21 20:40:20,998 INFO repeater task wh300_beamplots starting 1.9 seconds late 2010-04-21 20:40:25,436 INFO repeater task wh300_beamplots ended after 4.4 seconds 2010-04-21 20:40:25,438 INFO repeater task os75bb_beamplots starting 6.3 seconds late 2010-04-21 20:40:29,577 INFO repeater task os75bb_beamplots ended after 4.1 seconds 2010-04-21 20:40:29,578 INFO repeater task os75nb_beamplots starting 10.5 seconds late 2010-04-21 20:40:34,955 INFO repeater task os75nb_beamplots ended after 5.4 seconds 2010-04-21 20:41:25,208 INFO repeater task wh300_lastensq starting 4.2 seconds late 2010-04-21 20:41:44,186 INFO repeater task wh300_lastensq ended after 19.0 seconds 2010-04-21 20:41:44,187 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:41:12 2010-04-21 20:42:19,740 INFO repeater task wh300_beamplots starting 0.6 seconds late 2010-04-21 20:42:23,269 INFO repeater task wh300_beamplots ended after 3.5 seconds 2010-04-21 20:42:23,270 INFO repeater task os75bb_beamplots starting 4.2 seconds late 2010-04-21 20:42:26,838 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-04-21 20:42:26,839 INFO repeater task os75nb_beamplots starting 7.7 seconds late 2010-04-21 20:42:30,210 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 20:42:30,211 INFO repeater task wh300_beamplots starting 11.1 seconds late 2010-04-21 20:42:44,601 INFO repeater task wh300_beamplots ended after 14.4 seconds 2010-04-21 20:42:44,602 INFO repeater task os75bb_beamplots starting 25.5 seconds late 2010-04-21 20:43:00,723 INFO repeater task os75bb_beamplots ended after 16.1 seconds 2010-04-21 20:43:00,725 INFO repeater task os75nb_beamplots starting 41.6 seconds late 2010-04-21 20:43:18,716 INFO repeater task os75nb_beamplots ended after 18.0 seconds 2010-04-21 20:43:18,719 INFO repeater task posmv_hcorrstats starting 59.6 seconds late 2010-04-21 20:43:45,068 INFO repeater task posmv_hcorrstats ended after 26.3 seconds 2010-04-21 20:43:45,082 INFO repeater task phins_hcorrstats starting 85.9 seconds late 2010-04-21 20:44:15,111 INFO repeater task phins_hcorrstats ended after 30.0 seconds 2010-04-21 20:44:20,108 INFO repeater task wh300_lastensq starting 58.1 seconds late 2010-04-21 20:44:40,335 INFO repeater task wh300_lastensq ended after 20.2 seconds 2010-04-21 20:44:40,335 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:43:12 2010-04-21 20:44:40,337 INFO repeater task os75bb_lastensq starting 27.3 seconds late 2010-04-21 20:44:58,523 INFO repeater task os75bb_lastensq ended after 18.2 seconds 2010-04-21 20:44:58,523 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:44:11 2010-04-21 20:44:58,525 INFO repeater task os75nb_lastensq starting 107.5 seconds late 2010-04-21 20:45:17,618 INFO repeater task os75nb_lastensq ended after 19.1 seconds 2010-04-21 20:45:17,619 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:43:04 2010-04-21 20:45:17,625 INFO repeater task wh300_beamplots starting 58.5 seconds late 2010-04-21 20:45:20,869 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:45:20,870 INFO repeater task os75bb_beamplots starting 61.8 seconds late 2010-04-21 20:45:24,357 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 20:45:24,358 INFO repeater task os75nb_beamplots starting 65.2 seconds late 2010-04-21 20:45:28,147 INFO repeater task os75nb_beamplots ended after 3.8 seconds 2010-04-21 20:45:33,148 INFO repeater task wh300_lastensq starting 11.1 seconds late 2010-04-21 20:45:51,874 INFO repeater task wh300_lastensq ended after 18.7 seconds 2010-04-21 20:45:51,875 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:45:13 2010-04-21 20:46:22,158 INFO repeater task wh300_beamplots starting 3.0 seconds late 2010-04-21 20:46:25,327 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:46:25,328 INFO repeater task os75bb_beamplots starting 6.2 seconds late 2010-04-21 20:46:29,279 INFO repeater task os75bb_beamplots ended after 4.0 seconds 2010-04-21 20:46:29,280 INFO repeater task os75nb_beamplots starting 10.2 seconds late 2010-04-21 20:46:32,487 INFO repeater task os75nb_beamplots ended after 3.2 seconds 2010-04-21 20:47:23,104 INFO repeater task wh300_lastensq starting 0.1 seconds late 2010-04-21 20:47:42,560 INFO repeater task wh300_lastensq ended after 19.5 seconds 2010-04-21 20:47:42,560 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:47:13 2010-04-21 20:47:42,569 INFO repeater task posmv_hcorrstats starting 23.4 seconds late 2010-04-21 20:48:05,015 INFO repeater task posmv_hcorrstats ended after 22.4 seconds 2010-04-21 20:48:05,016 INFO repeater task phins_hcorrstats starting 45.9 seconds late 2010-04-21 20:48:26,147 INFO repeater task phins_hcorrstats ended after 21.1 seconds 2010-04-21 20:48:31,146 INFO repeater task os75nb_lastensq starting 17.1 seconds late 2010-04-21 20:48:47,541 INFO repeater task os75nb_lastensq ended after 16.4 seconds 2010-04-21 20:48:47,542 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:48:07 2010-04-21 20:48:47,543 INFO repeater task wh300_beamplots starting 28.4 seconds late 2010-04-21 20:48:50,604 INFO repeater task wh300_beamplots ended after 3.1 seconds 2010-04-21 20:48:50,605 INFO repeater task os75bb_beamplots starting 31.5 seconds late 2010-04-21 20:48:54,089 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 20:48:54,090 INFO repeater task os75nb_beamplots starting 35.0 seconds late 2010-04-21 20:48:57,414 INFO repeater task os75nb_beamplots ended after 3.3 seconds 2010-04-21 20:49:22,536 INFO repeater task os75bb_lastensq starting 1.5 seconds late 2010-04-21 20:49:39,286 INFO repeater task os75bb_lastensq ended after 16.8 seconds 2010-04-21 20:49:39,287 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:49:14 2010-04-21 20:49:44,292 INFO repeater task wh300_lastensq starting 21.3 seconds late 2010-04-21 20:50:03,947 INFO repeater task wh300_lastensq ended after 19.7 seconds 2010-04-21 20:50:03,948 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:49:14 2010-04-21 20:50:23,973 INFO repeater task wh300_beamplots starting 4.9 seconds late 2010-04-21 20:50:27,060 INFO repeater task wh300_beamplots ended after 3.1 seconds 2010-04-21 20:50:27,061 INFO repeater task os75bb_beamplots starting 7.9 seconds late 2010-04-21 20:50:30,706 INFO repeater task os75bb_beamplots ended after 3.6 seconds 2010-04-21 20:50:30,707 INFO repeater task os75nb_beamplots starting 11.6 seconds late 2010-04-21 20:50:34,324 INFO repeater task os75nb_beamplots ended after 3.6 seconds 2010-04-21 20:51:24,556 INFO repeater task wh300_lastensq starting 0.6 seconds late 2010-04-21 20:51:43,510 INFO repeater task wh300_lastensq ended after 19.0 seconds 2010-04-21 20:51:43,511 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:51:14 2010-04-21 20:52:23,564 INFO repeater task wh300_beamplots starting 4.4 seconds late 2010-04-21 20:52:26,735 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:52:26,736 INFO repeater task os75bb_beamplots starting 7.6 seconds late 2010-04-21 20:52:30,238 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 20:52:30,239 INFO repeater task os75nb_beamplots starting 11.1 seconds late 2010-04-21 20:52:33,740 INFO repeater task os75nb_beamplots ended after 3.5 seconds 2010-04-21 20:52:33,750 INFO repeater task wh300_beamplots starting 14.6 seconds late 2010-04-21 20:52:48,084 INFO repeater task wh300_beamplots ended after 14.3 seconds 2010-04-21 20:52:48,085 INFO repeater task os75bb_beamplots starting 29.0 seconds late 2010-04-21 20:53:04,447 INFO repeater task os75bb_beamplots ended after 16.4 seconds 2010-04-21 20:53:04,448 INFO repeater task os75nb_beamplots starting 45.3 seconds late 2010-04-21 20:53:22,524 INFO repeater task os75nb_beamplots ended after 18.1 seconds 2010-04-21 20:53:22,527 INFO repeater task posmv_hcorrstats starting 63.4 seconds late 2010-04-21 20:53:43,371 INFO repeater task posmv_hcorrstats ended after 20.8 seconds 2010-04-21 20:53:43,372 INFO repeater task phins_hcorrstats starting 84.2 seconds late 2010-04-21 20:54:06,218 INFO repeater task phins_hcorrstats ended after 22.8 seconds 2010-04-21 20:54:11,216 INFO repeater task wh300_lastensq starting 47.2 seconds late 2010-04-21 20:54:30,274 INFO repeater task wh300_lastensq ended after 19.1 seconds 2010-04-21 20:54:30,275 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:53:15 2010-04-21 20:54:30,276 INFO repeater task os75bb_lastensq starting 6.3 seconds late 2010-04-21 20:54:47,689 INFO repeater task os75bb_lastensq ended after 17.4 seconds 2010-04-21 20:54:47,690 INFO repeater Time from /home/data/kn197_6/proc/os75bb/load/wrote_ens_time: 2010/04/21 20:54:17 2010-04-21 20:54:47,720 INFO repeater task os75nb_lastensq starting 90.7 seconds late 2010-04-21 20:55:03,978 INFO repeater task os75nb_lastensq ended after 16.3 seconds 2010-04-21 20:55:03,978 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:53:11 2010-04-21 20:55:03,980 INFO repeater task wh300_beamplots starting 44.9 seconds late 2010-04-21 20:55:07,425 INFO repeater task wh300_beamplots ended after 3.4 seconds 2010-04-21 20:55:07,434 INFO repeater task os75bb_beamplots starting 48.3 seconds late 2010-04-21 20:55:10,896 INFO repeater task os75bb_beamplots ended after 3.5 seconds 2010-04-21 20:55:10,897 INFO repeater task os75nb_beamplots starting 51.8 seconds late 2010-04-21 20:55:15,077 INFO repeater task os75nb_beamplots ended after 4.2 seconds 2010-04-21 20:55:25,084 INFO repeater task wh300_lastensq starting 0.1 seconds late 2010-04-21 20:55:44,793 INFO repeater task wh300_lastensq ended after 19.7 seconds 2010-04-21 20:55:44,794 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:55:15 2010-04-21 20:56:20,270 INFO repeater task wh300_beamplots starting 1.2 seconds late 2010-04-21 20:56:23,300 INFO repeater task wh300_beamplots ended after 3.0 seconds 2010-04-21 20:56:23,300 INFO repeater task os75bb_beamplots starting 4.2 seconds late 2010-04-21 20:56:26,584 INFO repeater task os75bb_beamplots ended after 3.3 seconds 2010-04-21 20:56:26,585 INFO repeater task os75nb_beamplots starting 7.5 seconds late 2010-04-21 20:56:30,029 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 20:57:20,295 INFO repeater task posmv_hcorrstats starting 1.2 seconds late 2010-04-21 20:57:41,113 INFO repeater task posmv_hcorrstats ended after 20.8 seconds 2010-04-21 20:57:41,114 INFO repeater task phins_hcorrstats starting 22.0 seconds late 2010-04-21 20:58:00,986 INFO repeater task phins_hcorrstats ended after 19.9 seconds 2010-04-21 20:58:05,992 INFO repeater task wh300_lastensq starting 41.0 seconds late 2010-04-21 20:58:25,478 INFO repeater task wh300_lastensq ended after 19.5 seconds 2010-04-21 20:58:25,479 INFO repeater Time from /home/data/kn197_6/proc/wh300/load/wrote_ens_time: 2010/04/21 20:57:16 2010-04-21 20:58:25,481 INFO repeater task os75nb_lastensq starting 4.5 seconds late 2010-04-21 20:58:42,592 INFO repeater task os75nb_lastensq ended after 17.1 seconds 2010-04-21 20:58:42,593 INFO repeater Time from /home/data/kn197_6/proc/os75nb/load/wrote_ens_time: 2010/04/21 20:58:14 2010-04-21 20:58:42,594 INFO repeater task wh300_beamplots starting 23.5 seconds late 2010-04-21 20:58:45,797 INFO repeater task wh300_beamplots ended after 3.2 seconds 2010-04-21 20:58:45,798 INFO repeater task os75bb_beamplots starting 26.7 seconds late 2010-04-21 20:58:49,217 INFO repeater task os75bb_beamplots ended after 3.4 seconds 2010-04-21 20:58:49,218 INFO repeater task os75nb_beamplots starting 30.1 seconds late 2010-04-21 20:58:52,659 INFO repeater task os75nb_beamplots ended after 3.4 seconds 2010-04-21 20:59:27,712 INFO repeater task wh300_lastensq starting 1.7 seconds late 2010-04-21 20:59:47,103 INFO repeater task wh300_lastensq ended after 19.4 seconds