2015-09-16 04:01:55,342 INFO repeater task nb300_beamplots starting 16.6 seconds late 2015-09-16 04:01:57,220 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:01:57,220 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 04:01:57,221 INFO repeater task nb300_beamplots starting 18.5 seconds late 2015-09-16 04:01:59,146 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:01:59,147 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 04:01:59,147 INFO repeater task ashtech_hcorrstats starting 20.4 seconds late 2015-09-16 04:02:00,216 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 04:02:05,222 INFO repeater task nb300_10pings starting 6.5 seconds late 2015-09-16 04:02:05,411 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:02:50,474 INFO repeater task nb300_lastensq starting 1.5 seconds late 2015-09-16 04:02:53,069 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:02:53,070 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:02:39 2015-09-16 04:04:23,193 INFO repeater task nb300_10pings starting 4.4 seconds late 2015-09-16 04:04:23,374 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:04:53,411 INFO repeater task nb300_lastensq starting 4.4 seconds late 2015-09-16 04:04:56,055 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:04:56,055 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:04:39 2015-09-16 04:06:41,172 INFO repeater task nb300_10pings starting 2.4 seconds late 2015-09-16 04:06:41,396 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:06:41,400 INFO repeater task ashtech_hcorrstats starting 2.6 seconds late 2015-09-16 04:06:42,461 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 04:06:52,474 INFO repeater task nb300_lastensq starting 3.5 seconds late 2015-09-16 04:06:55,059 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:06:55,059 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:06:39 2015-09-16 04:08:50,228 INFO repeater task nb300_lastensq starting 1.2 seconds late 2015-09-16 04:08:52,857 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:08:52,858 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:08:39 2015-09-16 04:09:02,872 INFO repeater task nb300_10pings starting 4.1 seconds late 2015-09-16 04:09:03,051 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:10:53,205 INFO repeater task nb300_lastensq starting 4.2 seconds late 2015-09-16 04:10:55,767 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:10:55,768 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:10:39 2015-09-16 04:11:20,806 INFO repeater task nb300_10pings starting 2.1 seconds late 2015-09-16 04:11:20,990 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:11:41,017 INFO repeater task ashtech_hcorrstats starting 2.3 seconds late 2015-09-16 04:11:42,064 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 04:12:52,165 INFO repeater task nb300_lastensq starting 3.2 seconds late 2015-09-16 04:12:54,797 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:12:54,797 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:12:39 2015-09-16 04:13:39,862 INFO repeater task nb300_10pings starting 1.1 seconds late 2015-09-16 04:13:40,043 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:14:50,146 INFO repeater task nb300_lastensq starting 1.1 seconds late 2015-09-16 04:14:52,732 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:14:52,732 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:14:39 2015-09-16 04:16:02,834 INFO repeater task nb300_10pings starting 4.1 seconds late 2015-09-16 04:16:03,014 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:16:43,075 INFO repeater task nb300_quick starting 4.3 seconds late 2015-09-16 04:16:49,120 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 04:16:49,120 INFO repeater task nb300_3dayplots starting 10.4 seconds late 2015-09-16 04:16:54,930 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 04:16:54,930 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 04:16:54,931 INFO repeater task nb300_tsstats starting 16.2 seconds late 2015-09-16 04:16:55,947 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 04:16:55,947 INFO repeater task nb300_beamplots starting 17.2 seconds late 2015-09-16 04:16:57,804 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:16:57,804 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 04:16:57,804 INFO repeater task nb300_beamplots starting 19.1 seconds late 2015-09-16 04:16:59,846 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:16:59,846 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 04:16:59,846 INFO repeater task ashtech_hcorrstats starting 21.1 seconds late 2015-09-16 04:17:00,893 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 04:17:05,898 INFO repeater task nb300_lastensq starting 16.9 seconds late 2015-09-16 04:17:08,499 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:17:08,499 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:16:39 2015-09-16 04:18:23,609 INFO repeater task nb300_10pings starting 4.9 seconds late 2015-09-16 04:18:23,787 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:18:53,830 INFO repeater task nb300_lastensq starting 4.8 seconds late 2015-09-16 04:18:56,476 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:18:56,476 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:18:39 2015-09-16 04:20:41,632 INFO repeater task nb300_10pings starting 2.9 seconds late 2015-09-16 04:20:41,811 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:20:51,826 INFO repeater task nb300_lastensq starting 2.8 seconds late 2015-09-16 04:20:54,426 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:20:54,426 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:20:39 2015-09-16 04:21:39,492 INFO repeater task ashtech_hcorrstats starting 0.7 seconds late 2015-09-16 04:21:40,581 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 04:22:50,676 INFO repeater task nb300_lastensq starting 1.7 seconds late 2015-09-16 04:22:53,271 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:22:53,272 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:22:39 2015-09-16 04:23:03,286 INFO repeater task nb300_10pings starting 4.5 seconds late 2015-09-16 04:23:03,471 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:24:53,623 INFO repeater task nb300_lastensq starting 4.6 seconds late 2015-09-16 04:24:56,277 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:24:56,277 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:24:39 2015-09-16 04:25:21,308 INFO repeater task nb300_10pings starting 2.6 seconds late 2015-09-16 04:25:21,500 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:26:41,598 INFO repeater task ashtech_hcorrstats starting 2.8 seconds late 2015-09-16 04:26:42,660 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 04:26:52,671 INFO repeater task nb300_lastensq starting 3.7 seconds late 2015-09-16 04:26:55,278 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:26:55,278 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:26:39 2015-09-16 04:27:40,340 INFO repeater task nb300_10pings starting 1.6 seconds late 2015-09-16 04:27:40,523 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:28:50,625 INFO repeater task nb300_lastensq starting 1.6 seconds late 2015-09-16 04:28:53,269 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:28:53,270 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:28:39 2015-09-16 04:30:03,366 INFO repeater task nb300_10pings starting 4.6 seconds late 2015-09-16 04:30:03,548 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:30:53,622 INFO repeater task nb300_lastensq starting 4.6 seconds late 2015-09-16 04:30:56,232 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:30:56,232 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:30:39 2015-09-16 04:31:41,296 INFO repeater task nb300_quick starting 2.5 seconds late 2015-09-16 04:31:47,249 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 04:31:47,250 INFO repeater task nb300_3dayplots starting 8.5 seconds late 2015-09-16 04:31:53,129 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 04:31:53,129 INFO repeater task nb300_3dayplots ended after 5.9 seconds 2015-09-16 04:31:53,132 INFO repeater task nb300_tsstats starting 14.4 seconds late 2015-09-16 04:31:54,139 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 04:31:54,140 INFO repeater task nb300_beamplots starting 15.4 seconds late 2015-09-16 04:31:56,011 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:31:56,011 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 04:31:56,012 INFO repeater task nb300_beamplots starting 17.3 seconds late 2015-09-16 04:31:57,933 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:31:57,933 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 04:31:57,933 INFO repeater task ashtech_hcorrstats starting 19.2 seconds late 2015-09-16 04:31:58,999 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 04:32:19,028 INFO repeater task nb300_10pings starting 0.3 seconds late 2015-09-16 04:32:19,210 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:32:49,255 INFO repeater task nb300_lastensq starting 0.3 seconds late 2015-09-16 04:32:51,849 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:32:51,849 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:32:39 2015-09-16 04:34:42,010 INFO repeater task nb300_10pings starting 3.3 seconds late 2015-09-16 04:34:42,194 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:34:52,209 INFO repeater task nb300_lastensq starting 3.2 seconds late 2015-09-16 04:34:54,863 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:34:54,864 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:34:39 2015-09-16 04:36:39,989 INFO repeater task ashtech_hcorrstats starting 1.2 seconds late 2015-09-16 04:36:41,034 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 04:36:51,043 INFO repeater task nb300_lastensq starting 2.0 seconds late 2015-09-16 04:36:53,637 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:36:53,638 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:36:39 2015-09-16 04:37:03,652 INFO repeater task nb300_10pings starting 4.9 seconds late 2015-09-16 04:37:03,833 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:38:53,996 INFO repeater task nb300_lastensq starting 5.0 seconds late 2015-09-16 04:38:56,643 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:38:56,644 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:38:39 2015-09-16 04:39:21,672 INFO repeater task nb300_10pings starting 2.9 seconds late 2015-09-16 04:39:21,855 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:40:51,976 INFO repeater task nb300_lastensq starting 3.0 seconds late 2015-09-16 04:40:54,574 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:40:54,575 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:40:39 2015-09-16 04:41:39,631 INFO repeater task nb300_10pings starting 0.9 seconds late 2015-09-16 04:41:39,817 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:41:39,818 INFO repeater task ashtech_hcorrstats starting 1.1 seconds late 2015-09-16 04:41:40,907 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 04:42:51,003 INFO repeater task nb300_lastensq starting 2.0 seconds late 2015-09-16 04:42:53,586 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:42:53,587 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:42:39 2015-09-16 04:44:03,689 INFO repeater task nb300_10pings starting 4.9 seconds late 2015-09-16 04:44:03,871 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:44:53,943 INFO repeater task nb300_lastensq starting 4.9 seconds late 2015-09-16 04:44:56,641 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:44:56,642 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:44:39 2015-09-16 04:46:21,742 INFO repeater task nb300_10pings starting 3.0 seconds late 2015-09-16 04:46:21,946 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:46:41,968 INFO repeater task nb300_quick starting 3.2 seconds late 2015-09-16 04:46:47,963 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 04:46:47,964 INFO repeater task nb300_3dayplots starting 9.2 seconds late 2015-09-16 04:46:53,731 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 04:46:53,732 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 04:46:53,732 INFO repeater task nb300_tsstats starting 15.0 seconds late 2015-09-16 04:46:54,743 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 04:46:54,744 INFO repeater task nb300_beamplots starting 16.0 seconds late 2015-09-16 04:46:56,628 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:46:56,629 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 04:46:56,629 INFO repeater task nb300_beamplots starting 17.9 seconds late 2015-09-16 04:46:58,625 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 04:46:58,625 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 04:46:58,625 INFO repeater task ashtech_hcorrstats starting 19.9 seconds late 2015-09-16 04:46:59,673 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 04:47:04,679 INFO repeater task nb300_lastensq starting 15.7 seconds late 2015-09-16 04:47:07,332 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:47:07,332 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:46:39 2015-09-16 04:48:42,464 INFO repeater task nb300_10pings starting 3.7 seconds late 2015-09-16 04:48:42,648 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:48:52,661 INFO repeater task nb300_lastensq starting 3.7 seconds late 2015-09-16 04:48:55,259 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:48:55,260 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:48:39 2015-09-16 04:50:50,429 INFO repeater task nb300_lastensq starting 1.4 seconds late 2015-09-16 04:50:53,119 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:50:53,120 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:50:39 2015-09-16 04:51:03,131 INFO repeater task nb300_10pings starting 4.4 seconds late 2015-09-16 04:51:03,318 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:51:43,375 INFO repeater task ashtech_hcorrstats starting 4.6 seconds late 2015-09-16 04:51:44,412 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 04:52:49,503 INFO repeater task nb300_lastensq starting 0.5 seconds late 2015-09-16 04:52:52,134 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:52:52,135 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:52:39 2015-09-16 04:53:22,171 INFO repeater task nb300_10pings starting 3.4 seconds late 2015-09-16 04:53:22,352 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:54:52,490 INFO repeater task nb300_lastensq starting 3.5 seconds late 2015-09-16 04:54:55,171 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:54:55,171 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:54:39 2015-09-16 04:55:40,240 INFO repeater task nb300_10pings starting 1.5 seconds late 2015-09-16 04:55:40,418 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:56:40,498 INFO repeater task ashtech_hcorrstats starting 1.7 seconds late 2015-09-16 04:56:41,548 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 04:56:51,561 INFO repeater task nb300_lastensq starting 2.6 seconds late 2015-09-16 04:56:54,156 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 04:56:54,157 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:56:39 2015-09-16 04:57:59,253 INFO repeater task nb300_10pings starting 0.5 seconds late 2015-09-16 04:57:59,432 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 04:58:49,504 INFO repeater task nb300_lastensq starting 0.5 seconds late 2015-09-16 04:58:52,195 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 04:58:52,196 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 04:58:39 2015-09-16 05:00:22,323 INFO repeater task nb300_10pings starting 3.6 seconds late 2015-09-16 05:00:22,506 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:00:52,547 INFO repeater task nb300_lastensq starting 3.5 seconds late 2015-09-16 05:00:55,161 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:00:55,161 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:00:39 2015-09-16 05:01:40,213 INFO repeater task nb300_quick starting 1.5 seconds late 2015-09-16 05:01:46,261 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 05:01:46,261 INFO repeater task nb300_3dayplots starting 7.5 seconds late 2015-09-16 05:01:52,045 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 05:01:52,045 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 05:01:52,046 INFO repeater task nb300_tsstats starting 13.3 seconds late 2015-09-16 05:01:53,073 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 05:01:53,074 INFO repeater task nb300_beamplots starting 14.3 seconds late 2015-09-16 05:01:54,963 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:01:54,963 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 05:01:54,963 INFO repeater task nb300_beamplots starting 16.2 seconds late 2015-09-16 05:01:56,993 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:01:56,994 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 05:01:56,994 INFO repeater task ashtech_hcorrstats starting 18.2 seconds late 2015-09-16 05:01:58,036 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:02:43,087 INFO repeater task nb300_10pings starting 4.3 seconds late 2015-09-16 05:02:43,270 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:02:53,284 INFO repeater task nb300_lastensq starting 4.3 seconds late 2015-09-16 05:02:55,890 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:02:55,890 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:02:39 2015-09-16 05:04:50,826 INFO repeater task nb300_lastensq starting 1.8 seconds late 2015-09-16 05:04:53,479 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:04:53,480 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:04:39 2015-09-16 05:05:03,494 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 05:05:03,677 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:06:38,817 INFO repeater task ashtech_hcorrstats starting 0.1 seconds late 2015-09-16 05:06:39,867 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 05:06:49,881 INFO repeater task nb300_lastensq starting 0.9 seconds late 2015-09-16 05:06:52,536 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:06:52,537 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:06:39 2015-09-16 05:07:22,580 INFO repeater task nb300_10pings starting 3.8 seconds late 2015-09-16 05:07:22,763 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:08:52,900 INFO repeater task nb300_lastensq starting 3.9 seconds late 2015-09-16 05:08:55,516 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:08:55,516 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:08:39 2015-09-16 05:09:40,580 INFO repeater task nb300_10pings starting 1.8 seconds late 2015-09-16 05:09:40,764 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:10:50,868 INFO repeater task nb300_lastensq starting 1.9 seconds late 2015-09-16 05:10:53,536 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:10:53,537 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:10:39 2015-09-16 05:11:43,601 INFO repeater task ashtech_hcorrstats starting 4.8 seconds late 2015-09-16 05:11:44,636 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:11:59,657 INFO repeater task nb300_10pings starting 0.9 seconds late 2015-09-16 05:11:59,839 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:12:49,910 INFO repeater task nb300_lastensq starting 0.9 seconds late 2015-09-16 05:12:52,514 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:12:52,515 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:12:39 2015-09-16 05:14:22,640 INFO repeater task nb300_10pings starting 3.9 seconds late 2015-09-16 05:14:22,823 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:14:52,867 INFO repeater task nb300_lastensq starting 3.9 seconds late 2015-09-16 05:14:55,528 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:14:55,529 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:14:39 2015-09-16 05:16:40,668 INFO repeater task nb300_10pings starting 1.9 seconds late 2015-09-16 05:16:40,851 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:16:40,851 INFO repeater task nb300_quick starting 2.1 seconds late 2015-09-16 05:16:46,857 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 05:16:46,857 INFO repeater task nb300_3dayplots starting 8.1 seconds late 2015-09-16 05:16:52,664 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 05:16:52,665 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 05:16:52,665 INFO repeater task nb300_tsstats starting 13.9 seconds late 2015-09-16 05:16:53,690 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 05:16:53,691 INFO repeater task nb300_beamplots starting 14.9 seconds late 2015-09-16 05:16:55,594 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:16:55,594 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 05:16:55,595 INFO repeater task nb300_beamplots starting 16.8 seconds late 2015-09-16 05:16:57,547 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:16:57,548 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 05:16:57,548 INFO repeater task ashtech_hcorrstats starting 18.8 seconds late 2015-09-16 05:16:58,591 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:17:03,597 INFO repeater task nb300_lastensq starting 14.6 seconds late 2015-09-16 05:17:06,243 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:17:06,244 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:16:39 2015-09-16 05:18:51,395 INFO repeater task nb300_lastensq starting 2.4 seconds late 2015-09-16 05:18:54,010 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:18:54,010 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:18:39 2015-09-16 05:18:59,017 INFO repeater task nb300_10pings starting 0.3 seconds late 2015-09-16 05:18:59,197 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:20:49,330 INFO repeater task nb300_lastensq starting 0.3 seconds late 2015-09-16 05:20:52,004 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:20:52,005 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:20:39 2015-09-16 05:21:22,047 INFO repeater task nb300_10pings starting 3.3 seconds late 2015-09-16 05:21:22,231 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:21:42,262 INFO repeater task ashtech_hcorrstats starting 3.5 seconds late 2015-09-16 05:21:43,306 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:22:53,408 INFO repeater task nb300_lastensq starting 4.4 seconds late 2015-09-16 05:22:56,026 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:22:56,026 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:22:39 2015-09-16 05:23:41,090 INFO repeater task nb300_10pings starting 2.3 seconds late 2015-09-16 05:23:41,272 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:24:51,377 INFO repeater task nb300_lastensq starting 2.4 seconds late 2015-09-16 05:24:54,055 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:24:54,055 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:24:39 2015-09-16 05:25:59,153 INFO repeater task nb300_10pings starting 0.4 seconds late 2015-09-16 05:25:59,335 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:26:39,393 INFO repeater task ashtech_hcorrstats starting 0.6 seconds late 2015-09-16 05:26:40,450 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 05:26:50,463 INFO repeater task nb300_lastensq starting 1.5 seconds late 2015-09-16 05:26:53,156 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2015-09-16 05:26:53,157 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:26:53,157 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:26:39 2015-09-16 05:28:23,293 INFO repeater task nb300_10pings starting 4.5 seconds late 2015-09-16 05:28:23,471 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:28:53,516 INFO repeater task nb300_lastensq starting 4.5 seconds late 2015-09-16 05:28:56,130 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:28:56,130 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:28:39 2015-09-16 05:30:41,287 INFO repeater task nb300_10pings starting 2.5 seconds late 2015-09-16 05:30:41,470 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:30:51,484 INFO repeater task nb300_lastensq starting 2.5 seconds late 2015-09-16 05:30:54,163 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:30:54,163 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:30:39 2015-09-16 05:31:39,231 INFO repeater task nb300_quick starting 0.5 seconds late 2015-09-16 05:31:45,240 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 05:31:45,240 INFO repeater task nb300_3dayplots starting 6.5 seconds late 2015-09-16 05:31:51,057 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 05:31:51,058 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 05:31:51,058 INFO repeater task nb300_tsstats starting 12.3 seconds late 2015-09-16 05:31:52,077 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 05:31:52,077 INFO repeater task nb300_beamplots starting 13.3 seconds late 2015-09-16 05:31:54,035 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:31:54,036 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 05:31:54,036 INFO repeater task nb300_beamplots starting 15.3 seconds late 2015-09-16 05:31:55,978 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:31:55,978 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 05:31:55,979 INFO repeater task ashtech_hcorrstats starting 17.2 seconds late 2015-09-16 05:31:57,014 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:32:52,095 INFO repeater task nb300_lastensq starting 3.1 seconds late 2015-09-16 05:32:54,773 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:32:54,773 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:32:39 2015-09-16 05:32:59,780 INFO repeater task nb300_10pings starting 1.0 seconds late 2015-09-16 05:32:59,962 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:34:50,100 INFO repeater task nb300_lastensq starting 1.1 seconds late 2015-09-16 05:34:52,720 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:34:52,720 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:34:39 2015-09-16 05:35:22,758 INFO repeater task nb300_10pings starting 4.0 seconds late 2015-09-16 05:35:22,937 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:36:43,032 INFO repeater task ashtech_hcorrstats starting 4.3 seconds late 2015-09-16 05:36:44,087 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 05:36:49,093 INFO repeater task nb300_lastensq starting 0.1 seconds late 2015-09-16 05:36:51,766 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:36:51,767 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:36:39 2015-09-16 05:37:41,838 INFO repeater task nb300_10pings starting 3.1 seconds late 2015-09-16 05:37:42,018 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:38:52,115 INFO repeater task nb300_lastensq starting 3.1 seconds late 2015-09-16 05:38:54,741 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:38:54,741 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:38:39 2015-09-16 05:39:59,835 INFO repeater task nb300_10pings starting 1.1 seconds late 2015-09-16 05:40:00,013 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:40:50,079 INFO repeater task nb300_lastensq starting 1.1 seconds late 2015-09-16 05:40:52,761 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:40:52,762 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:40:39 2015-09-16 05:41:42,831 INFO repeater task ashtech_hcorrstats starting 4.1 seconds late 2015-09-16 05:41:43,875 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:42:18,916 INFO repeater task nb300_10pings starting 0.2 seconds late 2015-09-16 05:42:19,096 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:42:49,139 INFO repeater task nb300_lastensq starting 0.1 seconds late 2015-09-16 05:42:51,822 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:42:51,822 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:42:39 2015-09-16 05:44:41,961 INFO repeater task nb300_10pings starting 3.2 seconds late 2015-09-16 05:44:42,165 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:44:52,183 INFO repeater task nb300_lastensq starting 3.2 seconds late 2015-09-16 05:44:54,817 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:44:54,817 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:44:39 2015-09-16 05:46:39,962 INFO repeater task nb300_quick starting 1.2 seconds late 2015-09-16 05:46:45,943 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 05:46:45,944 INFO repeater task nb300_3dayplots starting 7.2 seconds late 2015-09-16 05:46:51,764 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 05:46:51,765 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 05:46:51,765 INFO repeater task nb300_tsstats starting 13.0 seconds late 2015-09-16 05:46:52,836 INFO repeater task nb300_tsstats ended after 1.1 seconds 2015-09-16 05:46:52,837 INFO repeater task nb300_beamplots starting 14.1 seconds late 2015-09-16 05:46:54,730 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:46:54,731 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 05:46:54,731 INFO repeater task nb300_beamplots starting 16.0 seconds late 2015-09-16 05:46:56,660 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 05:46:56,660 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 05:46:56,660 INFO repeater task ashtech_hcorrstats starting 17.9 seconds late 2015-09-16 05:46:57,734 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 05:47:02,740 INFO repeater task nb300_lastensq starting 13.7 seconds late 2015-09-16 05:47:05,373 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:47:05,373 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:46:39 2015-09-16 05:47:05,374 INFO repeater task nb300_10pings starting 6.6 seconds late 2015-09-16 05:47:05,551 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:48:50,708 INFO repeater task nb300_lastensq starting 1.7 seconds late 2015-09-16 05:48:53,395 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:48:53,395 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:48:39 2015-09-16 05:49:23,437 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 05:49:23,617 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:50:53,736 INFO repeater task nb300_lastensq starting 4.7 seconds late 2015-09-16 05:50:56,379 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:50:56,380 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:50:39 2015-09-16 05:51:41,447 INFO repeater task nb300_10pings starting 2.7 seconds late 2015-09-16 05:51:41,629 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:51:41,630 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2015-09-16 05:51:42,675 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:52:52,778 INFO repeater task nb300_lastensq starting 3.8 seconds late 2015-09-16 05:52:55,468 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:52:55,468 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:52:39 2015-09-16 05:54:00,565 INFO repeater task nb300_10pings starting 1.8 seconds late 2015-09-16 05:54:00,745 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:54:50,816 INFO repeater task nb300_lastensq starting 1.8 seconds late 2015-09-16 05:54:53,483 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:54:53,485 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:54:39 2015-09-16 05:56:23,591 INFO repeater task nb300_10pings starting 4.8 seconds late 2015-09-16 05:56:23,774 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:56:38,795 INFO repeater task ashtech_hcorrstats starting 0.0 seconds late 2015-09-16 05:56:39,843 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 05:56:49,856 INFO repeater task nb300_lastensq starting 0.9 seconds late 2015-09-16 05:56:52,541 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 05:56:52,542 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:56:39 2015-09-16 05:58:42,694 INFO repeater task nb300_10pings starting 3.9 seconds late 2015-09-16 05:58:42,874 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 05:58:52,889 INFO repeater task nb300_lastensq starting 3.9 seconds late 2015-09-16 05:58:55,529 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 05:58:55,529 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 05:58:39 2015-09-16 06:00:50,695 INFO repeater task nb300_lastensq starting 1.7 seconds late 2015-09-16 06:00:53,427 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:00:53,427 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:00:39 2015-09-16 06:01:03,442 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 06:01:03,624 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:01:43,684 INFO repeater task nb300_quick starting 4.9 seconds late 2015-09-16 06:01:49,654 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 06:01:49,655 INFO repeater task nb300_3dayplots starting 10.9 seconds late 2015-09-16 06:01:55,490 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 06:01:55,491 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 06:01:55,491 INFO repeater task nb300_tsstats starting 16.7 seconds late 2015-09-16 06:01:56,534 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-09-16 06:01:56,534 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 06:01:56,534 INFO repeater task nb300_beamplots starting 17.8 seconds late 2015-09-16 06:01:58,445 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:01:58,445 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 06:01:58,446 INFO repeater task nb300_beamplots starting 19.7 seconds late 2015-09-16 06:02:00,399 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:02:00,400 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 06:02:00,400 INFO repeater task ashtech_hcorrstats starting 21.6 seconds late 2015-09-16 06:02:01,452 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:02:51,521 INFO repeater task nb300_lastensq starting 2.5 seconds late 2015-09-16 06:02:54,171 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:02:54,171 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:02:39 2015-09-16 06:03:19,208 INFO repeater task nb300_10pings starting 0.5 seconds late 2015-09-16 06:03:19,390 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:04:49,521 INFO repeater task nb300_lastensq starting 0.5 seconds late 2015-09-16 06:04:52,120 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:04:52,120 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:04:39 2015-09-16 06:05:42,196 INFO repeater task nb300_10pings starting 3.4 seconds late 2015-09-16 06:05:42,382 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:06:42,467 INFO repeater task ashtech_hcorrstats starting 3.7 seconds late 2015-09-16 06:06:43,576 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:06:53,588 INFO repeater task nb300_lastensq starting 4.6 seconds late 2015-09-16 06:06:56,176 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:06:56,177 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:06:39 2015-09-16 06:08:01,277 INFO repeater task nb300_10pings starting 2.5 seconds late 2015-09-16 06:08:01,460 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:08:51,535 INFO repeater task nb300_lastensq starting 2.5 seconds late 2015-09-16 06:08:54,164 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:08:54,165 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:08:39 2015-09-16 06:10:19,290 INFO repeater task nb300_10pings starting 0.5 seconds late 2015-09-16 06:10:19,472 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:10:49,518 INFO repeater task nb300_lastensq starting 0.5 seconds late 2015-09-16 06:10:52,099 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:10:52,099 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:10:39 2015-09-16 06:11:42,176 INFO repeater task ashtech_hcorrstats starting 3.4 seconds late 2015-09-16 06:11:43,225 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 06:12:43,307 INFO repeater task nb300_10pings starting 4.6 seconds late 2015-09-16 06:12:43,490 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:12:53,504 INFO repeater task nb300_lastensq starting 4.5 seconds late 2015-09-16 06:12:56,160 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:12:56,160 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:12:39 2015-09-16 06:14:51,324 INFO repeater task nb300_lastensq starting 2.3 seconds late 2015-09-16 06:14:53,911 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:14:53,912 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:14:39 2015-09-16 06:14:58,919 INFO repeater task nb300_10pings starting 0.2 seconds late 2015-09-16 06:14:59,101 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:16:39,246 INFO repeater task nb300_quick starting 0.5 seconds late 2015-09-16 06:16:45,215 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 06:16:45,215 INFO repeater task nb300_3dayplots starting 6.5 seconds late 2015-09-16 06:16:51,047 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 06:16:51,048 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 06:16:51,048 INFO repeater task nb300_tsstats starting 12.3 seconds late 2015-09-16 06:16:52,065 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 06:16:52,066 INFO repeater task nb300_beamplots starting 13.3 seconds late 2015-09-16 06:16:53,925 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:16:53,925 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 06:16:53,926 INFO repeater task nb300_beamplots starting 15.2 seconds late 2015-09-16 06:16:55,875 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:16:55,876 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 06:16:55,876 INFO repeater task ashtech_hcorrstats starting 17.1 seconds late 2015-09-16 06:16:56,915 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 06:17:01,921 INFO repeater task nb300_lastensq starting 12.9 seconds late 2015-09-16 06:17:04,496 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:17:04,496 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:16:39 2015-09-16 06:17:19,518 INFO repeater task nb300_10pings starting 0.8 seconds late 2015-09-16 06:17:19,703 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:18:49,830 INFO repeater task nb300_lastensq starting 0.8 seconds late 2015-09-16 06:18:52,470 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:18:52,471 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:18:39 2015-09-16 06:19:42,546 INFO repeater task nb300_10pings starting 3.8 seconds late 2015-09-16 06:19:42,732 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:20:52,832 INFO repeater task nb300_lastensq starting 3.8 seconds late 2015-09-16 06:20:55,442 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:20:55,442 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:20:39 2015-09-16 06:21:40,513 INFO repeater task ashtech_hcorrstats starting 1.8 seconds late 2015-09-16 06:21:41,563 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 06:22:01,591 INFO repeater task nb300_10pings starting 2.8 seconds late 2015-09-16 06:22:01,803 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:22:51,854 INFO repeater task nb300_lastensq starting 2.9 seconds late 2015-09-16 06:22:54,481 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:22:54,482 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:22:39 2015-09-16 06:24:19,599 INFO repeater task nb300_10pings starting 0.9 seconds late 2015-09-16 06:24:19,785 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:24:49,829 INFO repeater task nb300_lastensq starting 0.8 seconds late 2015-09-16 06:24:52,437 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:24:52,438 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:24:39 2015-09-16 06:26:42,584 INFO repeater task nb300_10pings starting 3.8 seconds late 2015-09-16 06:26:42,767 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:26:42,768 INFO repeater task ashtech_hcorrstats starting 4.0 seconds late 2015-09-16 06:26:43,872 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:26:53,885 INFO repeater task nb300_lastensq starting 4.9 seconds late 2015-09-16 06:26:56,455 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:26:56,455 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:26:39 2015-09-16 06:28:51,620 INFO repeater task nb300_lastensq starting 2.6 seconds late 2015-09-16 06:28:54,257 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:28:54,258 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:28:39 2015-09-16 06:28:59,265 INFO repeater task nb300_10pings starting 0.5 seconds late 2015-09-16 06:28:59,448 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:30:49,610 INFO repeater task nb300_lastensq starting 0.6 seconds late 2015-09-16 06:30:52,209 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:30:52,209 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:30:39 2015-09-16 06:31:22,250 INFO repeater task nb300_10pings starting 3.5 seconds late 2015-09-16 06:31:22,436 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:31:42,467 INFO repeater task nb300_quick starting 3.7 seconds late 2015-09-16 06:31:48,443 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 06:31:48,444 INFO repeater task nb300_3dayplots starting 9.7 seconds late 2015-09-16 06:31:54,252 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 06:31:54,252 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 06:31:54,252 INFO repeater task nb300_tsstats starting 15.5 seconds late 2015-09-16 06:31:55,324 INFO repeater task nb300_tsstats ended after 1.1 seconds 2015-09-16 06:31:55,325 INFO repeater task nb300_beamplots starting 16.6 seconds late 2015-09-16 06:31:57,211 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:31:57,211 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 06:31:57,212 INFO repeater task nb300_beamplots starting 18.5 seconds late 2015-09-16 06:31:59,180 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:31:59,180 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 06:31:59,180 INFO repeater task ashtech_hcorrstats starting 20.4 seconds late 2015-09-16 06:32:00,281 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:32:50,348 INFO repeater task nb300_lastensq starting 1.3 seconds late 2015-09-16 06:32:52,955 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:32:52,955 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:32:39 2015-09-16 06:33:43,029 INFO repeater task nb300_10pings starting 4.3 seconds late 2015-09-16 06:33:43,212 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:34:53,310 INFO repeater task nb300_lastensq starting 4.3 seconds late 2015-09-16 06:34:55,938 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:34:55,938 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:34:39 2015-09-16 06:36:01,027 INFO repeater task nb300_10pings starting 2.3 seconds late 2015-09-16 06:36:01,211 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:36:41,266 INFO repeater task ashtech_hcorrstats starting 2.5 seconds late 2015-09-16 06:36:42,316 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:36:52,329 INFO repeater task nb300_lastensq starting 3.3 seconds late 2015-09-16 06:36:54,927 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:36:54,927 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:36:39 2015-09-16 06:38:20,050 INFO repeater task nb300_10pings starting 1.3 seconds late 2015-09-16 06:38:20,232 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:38:50,273 INFO repeater task nb300_lastensq starting 1.3 seconds late 2015-09-16 06:38:52,920 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:38:52,921 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:38:39 2015-09-16 06:40:43,051 INFO repeater task nb300_10pings starting 4.3 seconds late 2015-09-16 06:40:43,234 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:40:53,248 INFO repeater task nb300_lastensq starting 4.2 seconds late 2015-09-16 06:40:55,851 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:40:55,851 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:40:39 2015-09-16 06:41:40,918 INFO repeater task ashtech_hcorrstats starting 2.2 seconds late 2015-09-16 06:41:41,980 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:42:52,074 INFO repeater task nb300_lastensq starting 3.1 seconds late 2015-09-16 06:42:54,715 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:42:54,716 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:42:39 2015-09-16 06:42:59,722 INFO repeater task nb300_10pings starting 1.0 seconds late 2015-09-16 06:42:59,903 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:44:50,035 INFO repeater task nb300_lastensq starting 1.0 seconds late 2015-09-16 06:44:52,642 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:44:52,642 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:44:39 2015-09-16 06:45:22,684 INFO repeater task nb300_10pings starting 3.9 seconds late 2015-09-16 06:45:22,883 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:46:42,990 INFO repeater task nb300_quick starting 4.2 seconds late 2015-09-16 06:46:49,029 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 06:46:49,029 INFO repeater task nb300_3dayplots starting 10.3 seconds late 2015-09-16 06:46:54,820 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 06:46:54,820 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 06:46:54,821 INFO repeater task nb300_tsstats starting 16.1 seconds late 2015-09-16 06:46:55,842 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 06:46:55,843 INFO repeater task nb300_beamplots starting 17.1 seconds late 2015-09-16 06:46:57,746 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:46:57,746 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 06:46:57,747 INFO repeater task nb300_beamplots starting 19.0 seconds late 2015-09-16 06:46:59,725 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 06:46:59,725 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 06:46:59,726 INFO repeater task ashtech_hcorrstats starting 21.0 seconds late 2015-09-16 06:47:00,766 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 06:47:05,771 INFO repeater task nb300_lastensq starting 16.8 seconds late 2015-09-16 06:47:08,394 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:47:08,394 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:46:39 2015-09-16 06:47:43,440 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 06:47:43,646 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:48:53,732 INFO repeater task nb300_lastensq starting 4.7 seconds late 2015-09-16 06:48:56,402 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:48:56,402 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:48:39 2015-09-16 06:50:01,488 INFO repeater task nb300_10pings starting 2.7 seconds late 2015-09-16 06:50:01,670 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:50:51,743 INFO repeater task nb300_lastensq starting 2.7 seconds late 2015-09-16 06:50:54,417 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:50:54,418 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:50:39 2015-09-16 06:51:39,481 INFO repeater task ashtech_hcorrstats starting 0.7 seconds late 2015-09-16 06:51:40,593 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:52:20,646 INFO repeater task nb300_10pings starting 1.9 seconds late 2015-09-16 06:52:20,830 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:52:50,874 INFO repeater task nb300_lastensq starting 1.9 seconds late 2015-09-16 06:52:53,471 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:52:53,471 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:52:39 2015-09-16 06:54:43,620 INFO repeater task nb300_10pings starting 4.9 seconds late 2015-09-16 06:54:43,803 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:54:53,812 INFO repeater task nb300_lastensq starting 4.8 seconds late 2015-09-16 06:54:56,494 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:54:56,494 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:54:39 2015-09-16 06:56:41,649 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2015-09-16 06:56:42,702 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 06:56:52,712 INFO repeater task nb300_lastensq starting 3.7 seconds late 2015-09-16 06:56:55,339 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 06:56:55,339 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:56:39 2015-09-16 06:57:00,346 INFO repeater task nb300_10pings starting 1.6 seconds late 2015-09-16 06:57:00,530 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 06:58:50,682 INFO repeater task nb300_lastensq starting 1.7 seconds late 2015-09-16 06:58:53,360 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 06:58:53,361 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 06:58:39 2015-09-16 06:59:23,401 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 06:59:23,581 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:00:53,712 INFO repeater task nb300_lastensq starting 4.7 seconds late 2015-09-16 07:00:56,329 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:00:56,329 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:00:39 2015-09-16 07:01:41,394 INFO repeater task nb300_10pings starting 2.6 seconds late 2015-09-16 07:01:41,579 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:01:41,580 INFO repeater task nb300_quick starting 2.8 seconds late 2015-09-16 07:01:47,598 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 07:01:47,599 INFO repeater task nb300_3dayplots starting 8.9 seconds late 2015-09-16 07:01:53,376 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 07:01:53,376 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 07:01:53,377 INFO repeater task nb300_tsstats starting 14.6 seconds late 2015-09-16 07:01:54,398 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 07:01:54,399 INFO repeater task nb300_beamplots starting 15.6 seconds late 2015-09-16 07:01:56,266 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:01:56,267 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 07:01:56,267 INFO repeater task nb300_beamplots starting 17.5 seconds late 2015-09-16 07:01:58,267 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:01:58,267 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 07:01:58,267 INFO repeater task ashtech_hcorrstats starting 19.5 seconds late 2015-09-16 07:01:59,310 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:02:49,379 INFO repeater task nb300_lastensq starting 0.4 seconds late 2015-09-16 07:02:51,981 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:02:51,981 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:02:39 2015-09-16 07:04:02,063 INFO repeater task nb300_10pings starting 3.3 seconds late 2015-09-16 07:04:02,245 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:04:52,308 INFO repeater task nb300_lastensq starting 3.3 seconds late 2015-09-16 07:04:54,978 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:04:54,979 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:04:39 2015-09-16 07:06:20,106 INFO repeater task nb300_10pings starting 1.4 seconds late 2015-09-16 07:06:20,290 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:06:40,321 INFO repeater task ashtech_hcorrstats starting 1.6 seconds late 2015-09-16 07:06:41,370 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:06:51,380 INFO repeater task nb300_lastensq starting 2.4 seconds late 2015-09-16 07:06:54,043 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:06:54,043 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:06:39 2015-09-16 07:08:39,175 INFO repeater task nb300_10pings starting 0.4 seconds late 2015-09-16 07:08:39,371 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:08:49,385 INFO repeater task nb300_lastensq starting 0.4 seconds late 2015-09-16 07:08:52,024 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:08:52,024 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:08:39 2015-09-16 07:10:52,198 INFO repeater task nb300_lastensq starting 3.2 seconds late 2015-09-16 07:10:54,875 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:10:54,876 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:10:39 2015-09-16 07:10:59,883 INFO repeater task nb300_10pings starting 1.1 seconds late 2015-09-16 07:11:00,070 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:11:40,126 INFO repeater task ashtech_hcorrstats starting 1.4 seconds late 2015-09-16 07:11:41,172 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:12:51,271 INFO repeater task nb300_lastensq starting 2.3 seconds late 2015-09-16 07:12:53,886 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:12:53,887 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:12:39 2015-09-16 07:13:18,924 INFO repeater task nb300_10pings starting 0.2 seconds late 2015-09-16 07:13:19,108 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:14:49,241 INFO repeater task nb300_lastensq starting 0.2 seconds late 2015-09-16 07:14:51,907 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:14:51,907 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:14:39 2015-09-16 07:15:41,967 INFO repeater task nb300_10pings starting 3.2 seconds late 2015-09-16 07:15:42,147 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:16:42,235 INFO repeater task nb300_quick starting 3.5 seconds late 2015-09-16 07:16:48,295 INFO repeater task nb300_quick ended after 6.1 seconds 2015-09-16 07:16:48,296 INFO repeater task nb300_3dayplots starting 9.5 seconds late 2015-09-16 07:16:54,092 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 07:16:54,093 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 07:16:54,093 INFO repeater task nb300_tsstats starting 15.3 seconds late 2015-09-16 07:16:55,102 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 07:16:55,103 INFO repeater task nb300_beamplots starting 16.4 seconds late 2015-09-16 07:16:57,037 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:16:57,037 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 07:16:57,037 INFO repeater task nb300_beamplots starting 18.3 seconds late 2015-09-16 07:16:58,948 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:16:58,948 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 07:16:58,948 INFO repeater task ashtech_hcorrstats starting 20.2 seconds late 2015-09-16 07:16:59,988 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:17:04,994 INFO repeater task nb300_lastensq starting 16.0 seconds late 2015-09-16 07:17:07,648 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:17:07,649 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:16:39 2015-09-16 07:18:02,728 INFO repeater task nb300_10pings starting 4.0 seconds late 2015-09-16 07:18:02,912 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:18:52,981 INFO repeater task nb300_lastensq starting 4.0 seconds late 2015-09-16 07:18:55,612 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:18:55,613 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:18:39 2015-09-16 07:20:20,738 INFO repeater task nb300_10pings starting 2.0 seconds late 2015-09-16 07:20:20,915 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:20:50,960 INFO repeater task nb300_lastensq starting 2.0 seconds late 2015-09-16 07:20:53,658 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:20:53,658 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:20:39 2015-09-16 07:21:43,729 INFO repeater task ashtech_hcorrstats starting 5.0 seconds late 2015-09-16 07:21:44,779 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 07:22:39,850 INFO repeater task nb300_10pings starting 1.1 seconds late 2015-09-16 07:22:40,030 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:22:50,045 INFO repeater task nb300_lastensq starting 1.0 seconds late 2015-09-16 07:22:52,704 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:22:52,704 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:22:39 2015-09-16 07:24:52,878 INFO repeater task nb300_lastensq starting 3.9 seconds late 2015-09-16 07:24:55,567 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:24:55,568 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:24:39 2015-09-16 07:25:00,575 INFO repeater task nb300_10pings starting 1.8 seconds late 2015-09-16 07:25:00,757 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:26:40,905 INFO repeater task ashtech_hcorrstats starting 2.2 seconds late 2015-09-16 07:26:41,953 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:26:51,966 INFO repeater task nb300_lastensq starting 3.0 seconds late 2015-09-16 07:26:54,658 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:26:54,659 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:26:39 2015-09-16 07:27:19,696 INFO repeater task nb300_10pings starting 0.9 seconds late 2015-09-16 07:27:19,881 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:28:50,012 INFO repeater task nb300_lastensq starting 1.0 seconds late 2015-09-16 07:28:52,632 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:28:52,632 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:28:39 2015-09-16 07:29:42,697 INFO repeater task nb300_10pings starting 3.9 seconds late 2015-09-16 07:29:42,878 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:30:52,987 INFO repeater task nb300_lastensq starting 4.0 seconds late 2015-09-16 07:30:55,656 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:30:55,656 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:30:39 2015-09-16 07:31:40,721 INFO repeater task nb300_quick starting 2.0 seconds late 2015-09-16 07:31:46,733 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 07:31:46,734 INFO repeater task nb300_3dayplots starting 8.0 seconds late 2015-09-16 07:31:52,525 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 07:31:52,526 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 07:31:52,526 INFO repeater task nb300_tsstats starting 13.8 seconds late 2015-09-16 07:31:53,533 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 07:31:53,534 INFO repeater task nb300_beamplots starting 14.8 seconds late 2015-09-16 07:31:55,445 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:31:55,445 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 07:31:55,446 INFO repeater task nb300_beamplots starting 16.7 seconds late 2015-09-16 07:31:57,410 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:31:57,410 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 07:31:57,410 INFO repeater task ashtech_hcorrstats starting 18.7 seconds late 2015-09-16 07:31:58,456 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:32:03,462 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 07:32:03,647 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:32:53,717 INFO repeater task nb300_lastensq starting 4.7 seconds late 2015-09-16 07:32:56,404 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:32:56,404 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:32:39 2015-09-16 07:34:21,520 INFO repeater task nb300_10pings starting 2.8 seconds late 2015-09-16 07:34:21,704 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:34:51,749 INFO repeater task nb300_lastensq starting 2.7 seconds late 2015-09-16 07:34:54,376 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:34:54,377 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:34:39 2015-09-16 07:36:39,518 INFO repeater task nb300_10pings starting 0.8 seconds late 2015-09-16 07:36:39,701 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:36:39,702 INFO repeater task ashtech_hcorrstats starting 0.9 seconds late 2015-09-16 07:36:40,748 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:36:50,761 INFO repeater task nb300_lastensq starting 1.8 seconds late 2015-09-16 07:36:53,450 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:36:53,450 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:36:39 2015-09-16 07:38:53,625 INFO repeater task nb300_lastensq starting 4.6 seconds late 2015-09-16 07:38:56,268 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:38:56,269 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:38:39 2015-09-16 07:39:01,276 INFO repeater task nb300_10pings starting 2.5 seconds late 2015-09-16 07:39:01,455 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:40:51,607 INFO repeater task nb300_lastensq starting 2.6 seconds late 2015-09-16 07:40:54,319 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:40:54,320 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:40:39 2015-09-16 07:41:19,355 INFO repeater task nb300_10pings starting 0.6 seconds late 2015-09-16 07:41:19,535 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:41:39,565 INFO repeater task ashtech_hcorrstats starting 0.8 seconds late 2015-09-16 07:41:40,606 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:42:50,707 INFO repeater task nb300_lastensq starting 1.7 seconds late 2015-09-16 07:42:53,394 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:42:53,395 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:42:39 2015-09-16 07:43:43,455 INFO repeater task nb300_10pings starting 4.7 seconds late 2015-09-16 07:43:43,643 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:44:53,735 INFO repeater task nb300_lastensq starting 4.7 seconds late 2015-09-16 07:44:56,388 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:44:56,388 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:44:39 2015-09-16 07:46:01,464 INFO repeater task nb300_10pings starting 2.7 seconds late 2015-09-16 07:46:01,645 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:46:41,707 INFO repeater task nb300_quick starting 3.0 seconds late 2015-09-16 07:46:47,732 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 07:46:47,733 INFO repeater task nb300_3dayplots starting 9.0 seconds late 2015-09-16 07:46:53,517 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 07:46:53,518 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 07:46:53,518 INFO repeater task nb300_tsstats starting 14.8 seconds late 2015-09-16 07:46:54,585 INFO repeater task nb300_tsstats ended after 1.1 seconds 2015-09-16 07:46:54,585 INFO repeater task nb300_beamplots starting 15.8 seconds late 2015-09-16 07:46:56,482 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:46:56,482 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 07:46:56,482 INFO repeater task nb300_beamplots starting 17.7 seconds late 2015-09-16 07:46:58,430 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 07:46:58,431 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 07:46:58,431 INFO repeater task ashtech_hcorrstats starting 19.7 seconds late 2015-09-16 07:46:59,526 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 07:47:04,531 INFO repeater task nb300_lastensq starting 15.5 seconds late 2015-09-16 07:47:07,170 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:47:07,170 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:46:39 2015-09-16 07:48:22,277 INFO repeater task nb300_10pings starting 3.5 seconds late 2015-09-16 07:48:22,463 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:48:52,508 INFO repeater task nb300_lastensq starting 3.5 seconds late 2015-09-16 07:48:55,208 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:48:55,208 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:48:39 2015-09-16 07:50:40,363 INFO repeater task nb300_10pings starting 1.6 seconds late 2015-09-16 07:50:40,545 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:50:50,559 INFO repeater task nb300_lastensq starting 1.6 seconds late 2015-09-16 07:50:53,212 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:50:53,212 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:50:39 2015-09-16 07:51:43,285 INFO repeater task ashtech_hcorrstats starting 4.5 seconds late 2015-09-16 07:51:44,327 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:52:49,420 INFO repeater task nb300_lastensq starting 0.4 seconds late 2015-09-16 07:52:52,130 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:52:52,130 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:52:39 2015-09-16 07:53:02,145 INFO repeater task nb300_10pings starting 3.4 seconds late 2015-09-16 07:53:02,326 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:54:52,476 INFO repeater task nb300_lastensq starting 3.5 seconds late 2015-09-16 07:54:55,119 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 07:54:55,119 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:54:39 2015-09-16 07:55:20,155 INFO repeater task nb300_10pings starting 1.4 seconds late 2015-09-16 07:55:20,338 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:56:40,455 INFO repeater task ashtech_hcorrstats starting 1.7 seconds late 2015-09-16 07:56:41,504 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2015-09-16 07:56:51,513 INFO repeater task nb300_lastensq starting 2.5 seconds late 2015-09-16 07:56:54,210 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:56:54,211 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:56:39 2015-09-16 07:57:39,268 INFO repeater task nb300_10pings starting 0.5 seconds late 2015-09-16 07:57:39,452 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 07:58:49,533 INFO repeater task nb300_lastensq starting 0.5 seconds late 2015-09-16 07:58:52,188 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 07:58:52,189 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 07:58:39 2015-09-16 08:00:02,276 INFO repeater task nb300_10pings starting 3.5 seconds late 2015-09-16 08:00:02,477 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:00:52,537 INFO repeater task nb300_lastensq starting 3.5 seconds late 2015-09-16 08:00:55,260 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 08:00:55,261 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:00:39 2015-09-16 08:01:40,328 INFO repeater task nb300_quick starting 1.6 seconds late 2015-09-16 08:01:46,348 INFO repeater task nb300_quick ended after 6.0 seconds 2015-09-16 08:01:46,349 INFO repeater task nb300_3dayplots starting 7.6 seconds late 2015-09-16 08:01:52,128 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 08:01:52,129 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 08:01:52,129 INFO repeater task nb300_tsstats starting 13.4 seconds late 2015-09-16 08:01:53,214 INFO repeater task nb300_tsstats ended after 1.1 seconds 2015-09-16 08:01:53,214 INFO repeater task nb300_beamplots starting 14.5 seconds late 2015-09-16 08:01:55,097 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 08:01:55,097 INFO repeater task nb300_beamplots ended after 1.9 seconds 2015-09-16 08:01:55,097 INFO repeater task nb300_beamplots starting 16.3 seconds late 2015-09-16 08:01:57,078 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 08:01:57,078 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 08:01:57,079 INFO repeater task ashtech_hcorrstats starting 18.3 seconds late 2015-09-16 08:01:58,150 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 08:02:23,180 INFO repeater task nb300_10pings starting 4.4 seconds late 2015-09-16 08:02:23,387 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:02:53,428 INFO repeater task nb300_lastensq starting 4.4 seconds late 2015-09-16 08:02:56,023 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:02:56,023 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:02:39 2015-09-16 08:04:41,154 INFO repeater task nb300_10pings starting 2.4 seconds late 2015-09-16 08:04:41,330 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:04:51,345 INFO repeater task nb300_lastensq starting 2.3 seconds late 2015-09-16 08:04:53,975 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:04:53,975 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:04:39 2015-09-16 08:06:39,106 INFO repeater task ashtech_hcorrstats starting 0.4 seconds late 2015-09-16 08:06:40,253 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 08:06:50,266 INFO repeater task nb300_lastensq starting 1.3 seconds late 2015-09-16 08:06:52,879 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:06:52,879 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:06:39 2015-09-16 08:07:02,892 INFO repeater task nb300_10pings starting 4.1 seconds late 2015-09-16 08:07:03,072 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:08:53,232 INFO repeater task nb300_lastensq starting 4.2 seconds late 2015-09-16 08:08:55,868 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:08:55,868 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:08:39 2015-09-16 08:09:20,906 INFO repeater task nb300_10pings starting 2.2 seconds late 2015-09-16 08:09:21,090 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:10:51,223 INFO repeater task nb300_lastensq starting 2.2 seconds late 2015-09-16 08:10:53,809 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:10:53,809 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:10:39 2015-09-16 08:11:38,874 INFO repeater task nb300_10pings starting 0.1 seconds late 2015-09-16 08:11:39,057 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:11:39,058 INFO repeater task ashtech_hcorrstats starting 0.3 seconds late 2015-09-16 08:11:40,129 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 08:12:50,217 INFO repeater task nb300_lastensq starting 1.2 seconds late 2015-09-16 08:12:52,880 INFO repeater task nb300_lastensq ended after 2.7 seconds 2015-09-16 08:12:52,880 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:12:39 2015-09-16 08:14:02,984 INFO repeater task nb300_10pings starting 4.2 seconds late 2015-09-16 08:14:03,166 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:14:53,236 INFO repeater task nb300_lastensq starting 4.2 seconds late 2015-09-16 08:14:55,817 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:14:55,818 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:14:39 2015-09-16 08:16:20,944 INFO repeater task nb300_10pings starting 2.2 seconds late 2015-09-16 08:16:21,130 INFO repeater task nb300_10pings ended after 0.2 seconds 2015-09-16 08:16:41,160 INFO repeater task nb300_quick starting 2.4 seconds late 2015-09-16 08:16:47,247 INFO repeater task nb300_quick ended after 6.1 seconds 2015-09-16 08:16:47,247 INFO repeater task nb300_3dayplots starting 8.5 seconds late 2015-09-16 08:16:53,027 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-09-16 08:16:53,027 INFO repeater task nb300_3dayplots ended after 5.8 seconds 2015-09-16 08:16:53,028 INFO repeater task nb300_tsstats starting 14.3 seconds late 2015-09-16 08:16:54,066 INFO repeater task nb300_tsstats ended after 1.0 seconds 2015-09-16 08:16:54,067 INFO repeater task nb300_beamplots starting 15.3 seconds late 2015-09-16 08:16:55,915 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 08:16:55,916 INFO repeater task nb300_beamplots ended after 1.8 seconds 2015-09-16 08:16:55,916 INFO repeater task nb300_beamplots starting 17.2 seconds late 2015-09-16 08:16:57,930 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2015-09-16 08:16:57,930 INFO repeater task nb300_beamplots ended after 2.0 seconds 2015-09-16 08:16:57,931 INFO repeater task ashtech_hcorrstats starting 19.2 seconds late 2015-09-16 08:16:59,004 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-09-16 08:17:04,010 INFO repeater task nb300_lastensq starting 15.0 seconds late 2015-09-16 08:17:06,619 INFO repeater task nb300_lastensq ended after 2.6 seconds 2015-09-16 08:17:06,619 INFO repeater task nb300_lastensq: Time from /home/data/SP1518/proc/nb300/load/wrote_ens_time: 2015/09/16 08:16:39 2015-09-16 08:18:41,759 INFO repeater task nb300_10pings starting 3.0 seconds late