2016-07-10 02:27:36,950 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 02:27:36,951 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 02:27:36,951 INFO repeater task ashtech_hcorrstats starting 15.7 seconds late 2016-07-10 02:27:37,947 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:28:33,029 INFO repeater task nb300_lastensq starting 2.0 seconds late 2016-07-10 02:28:35,664 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:28:35,665 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:28:21 2016-07-10 02:29:45,753 INFO repeater task nb300_10pings starting 4.5 seconds late 2016-07-10 02:29:45,933 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:30:31,000 INFO repeater task nb300_lastensq starting 0.0 seconds late 2016-07-10 02:30:33,571 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:30:33,571 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:30:21 2016-07-10 02:32:03,701 INFO repeater task nb300_10pings starting 2.4 seconds late 2016-07-10 02:32:03,879 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:32:23,911 INFO repeater task ashtech_hcorrstats starting 2.6 seconds late 2016-07-10 02:32:24,932 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:32:34,944 INFO repeater task nb300_lastensq starting 3.9 seconds late 2016-07-10 02:32:37,547 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:32:37,547 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:32:21 2016-07-10 02:34:22,670 INFO repeater task nb300_10pings starting 1.4 seconds late 2016-07-10 02:34:22,847 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:34:32,860 INFO repeater task nb300_lastensq starting 1.9 seconds late 2016-07-10 02:34:35,469 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:34:35,470 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:34:21 2016-07-10 02:36:35,622 INFO repeater task nb300_lastensq starting 4.6 seconds late 2016-07-10 02:36:38,236 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:36:38,236 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:36:21 2016-07-10 02:36:43,243 INFO repeater task nb300_10pings starting 2.0 seconds late 2016-07-10 02:36:43,423 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:37:23,478 INFO repeater task ashtech_hcorrstats starting 2.2 seconds late 2016-07-10 02:37:24,487 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:38:34,590 INFO repeater task nb300_lastensq starting 3.6 seconds late 2016-07-10 02:38:37,229 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:38:37,229 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:38:21 2016-07-10 02:39:02,263 INFO repeater task nb300_10pings starting 1.0 seconds late 2016-07-10 02:39:02,444 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:40:32,578 INFO repeater task nb300_lastensq starting 1.6 seconds late 2016-07-10 02:40:35,187 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:40:35,187 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:40:21 2016-07-10 02:41:25,254 INFO repeater task nb300_10pings starting 4.0 seconds late 2016-07-10 02:41:25,434 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:42:25,516 INFO repeater task nb300_quick starting 4.2 seconds late 2016-07-10 02:42:29,275 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 02:42:29,276 INFO repeater task nb300_3dayplots starting 8.0 seconds late 2016-07-10 02:42:33,147 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 02:42:33,147 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 02:42:33,148 INFO repeater task nb300_tsstats starting 11.9 seconds late 2016-07-10 02:42:34,025 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 02:42:34,025 INFO repeater task nb300_beamplots starting 12.7 seconds late 2016-07-10 02:42:35,869 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 02:42:35,869 INFO repeater task nb300_beamplots ended after 1.8 seconds 2016-07-10 02:42:35,870 INFO repeater task nb300_beamplots starting 14.6 seconds late 2016-07-10 02:42:37,812 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 02:42:37,813 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 02:42:37,813 INFO repeater task ashtech_hcorrstats starting 16.5 seconds late 2016-07-10 02:42:38,814 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:42:43,819 INFO repeater task nb300_lastensq starting 12.8 seconds late 2016-07-10 02:42:46,417 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:42:46,418 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:42:21 2016-07-10 02:43:41,500 INFO repeater task nb300_10pings starting 0.2 seconds late 2016-07-10 02:43:41,679 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:44:31,756 INFO repeater task nb300_lastensq starting 0.8 seconds late 2016-07-10 02:44:34,395 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:44:34,396 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:44:21 2016-07-10 02:46:04,531 INFO repeater task nb300_10pings starting 3.3 seconds late 2016-07-10 02:46:04,714 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:46:34,753 INFO repeater task nb300_lastensq starting 3.8 seconds late 2016-07-10 02:46:37,380 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:46:37,381 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:46:21 2016-07-10 02:47:22,443 INFO repeater task ashtech_hcorrstats starting 1.2 seconds late 2016-07-10 02:47:23,453 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:48:23,533 INFO repeater task nb300_10pings starting 2.3 seconds late 2016-07-10 02:48:23,711 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:48:33,725 INFO repeater task nb300_lastensq starting 2.7 seconds late 2016-07-10 02:48:36,336 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:48:36,336 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:48:21 2016-07-10 02:50:31,478 INFO repeater task nb300_lastensq starting 0.5 seconds late 2016-07-10 02:50:34,083 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:50:34,083 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:50:21 2016-07-10 02:50:44,094 INFO repeater task nb300_10pings starting 2.8 seconds late 2016-07-10 02:50:44,274 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:52:24,411 INFO repeater task ashtech_hcorrstats starting 3.1 seconds late 2016-07-10 02:52:25,440 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:52:35,449 INFO repeater task nb300_lastensq starting 4.4 seconds late 2016-07-10 02:52:38,057 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:52:38,057 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:52:21 2016-07-10 02:53:03,091 INFO repeater task nb300_10pings starting 1.8 seconds late 2016-07-10 02:53:03,274 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:54:33,407 INFO repeater task nb300_lastensq starting 2.4 seconds late 2016-07-10 02:54:36,040 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:54:36,040 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:54:21 2016-07-10 02:55:26,114 INFO repeater task nb300_10pings starting 4.8 seconds late 2016-07-10 02:55:26,293 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:56:31,390 INFO repeater task nb300_lastensq starting 0.4 seconds late 2016-07-10 02:56:34,014 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:56:34,014 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:56:21 2016-07-10 02:57:24,091 INFO repeater task nb300_quick starting 2.8 seconds late 2016-07-10 02:57:27,860 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 02:57:27,861 INFO repeater task nb300_3dayplots starting 6.6 seconds late 2016-07-10 02:57:31,749 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 02:57:31,750 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 02:57:31,750 INFO repeater task nb300_tsstats starting 10.5 seconds late 2016-07-10 02:57:32,629 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 02:57:32,630 INFO repeater task nb300_beamplots starting 11.4 seconds late 2016-07-10 02:57:34,484 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 02:57:34,484 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 02:57:34,484 INFO repeater task nb300_beamplots starting 13.2 seconds late 2016-07-10 02:57:36,425 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 02:57:36,425 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 02:57:36,426 INFO repeater task ashtech_hcorrstats starting 15.1 seconds late 2016-07-10 02:57:37,448 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 02:57:42,454 INFO repeater task nb300_10pings starting 1.2 seconds late 2016-07-10 02:57:42,634 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 02:58:32,709 INFO repeater task nb300_lastensq starting 1.7 seconds late 2016-07-10 02:58:35,329 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 02:58:35,330 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 02:58:21 2016-07-10 03:00:05,436 INFO repeater task nb300_10pings starting 4.2 seconds late 2016-07-10 03:00:05,616 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:00:35,649 INFO repeater task nb300_lastensq starting 4.6 seconds late 2016-07-10 03:00:38,280 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:00:38,281 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:00:21 2016-07-10 03:02:23,433 INFO repeater task nb300_10pings starting 2.2 seconds late 2016-07-10 03:02:23,614 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:02:23,615 INFO repeater task ashtech_hcorrstats starting 2.3 seconds late 2016-07-10 03:02:24,625 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:02:34,638 INFO repeater task nb300_lastensq starting 3.6 seconds late 2016-07-10 03:02:37,254 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:02:37,255 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:02:21 2016-07-10 03:04:32,420 INFO repeater task nb300_lastensq starting 1.4 seconds late 2016-07-10 03:04:35,044 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:04:35,044 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:04:21 2016-07-10 03:04:45,056 INFO repeater task nb300_10pings starting 3.8 seconds late 2016-07-10 03:04:45,238 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:06:35,382 INFO repeater task nb300_lastensq starting 4.4 seconds late 2016-07-10 03:06:37,981 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:06:37,982 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:06:21 2016-07-10 03:07:03,018 INFO repeater task nb300_10pings starting 1.7 seconds late 2016-07-10 03:07:03,195 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:07:23,227 INFO repeater task ashtech_hcorrstats starting 1.9 seconds late 2016-07-10 03:07:24,242 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:08:34,318 INFO repeater task nb300_lastensq starting 3.3 seconds late 2016-07-10 03:08:36,947 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:08:36,948 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:08:21 2016-07-10 03:09:22,010 INFO repeater task nb300_10pings starting 0.7 seconds late 2016-07-10 03:09:22,189 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:10:32,290 INFO repeater task nb300_lastensq starting 1.3 seconds late 2016-07-10 03:10:34,900 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:10:34,901 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:10:21 2016-07-10 03:11:45,001 INFO repeater task nb300_10pings starting 3.7 seconds late 2016-07-10 03:11:45,178 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:12:25,235 INFO repeater task nb300_quick starting 4.0 seconds late 2016-07-10 03:12:29,022 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 03:12:29,022 INFO repeater task nb300_3dayplots starting 7.7 seconds late 2016-07-10 03:12:32,927 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 03:12:32,927 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 03:12:32,927 INFO repeater task nb300_tsstats starting 11.6 seconds late 2016-07-10 03:12:33,809 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 03:12:33,810 INFO repeater task nb300_beamplots starting 12.5 seconds late 2016-07-10 03:12:35,655 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:12:35,655 INFO repeater task nb300_beamplots ended after 1.8 seconds 2016-07-10 03:12:35,655 INFO repeater task nb300_beamplots starting 14.4 seconds late 2016-07-10 03:12:37,608 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:12:37,608 INFO repeater task nb300_beamplots ended after 2.0 seconds 2016-07-10 03:12:37,608 INFO repeater task ashtech_hcorrstats starting 16.3 seconds late 2016-07-10 03:12:38,616 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:12:43,622 INFO repeater task nb300_lastensq starting 12.6 seconds late 2016-07-10 03:12:46,248 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:12:46,248 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:12:21 2016-07-10 03:14:01,357 INFO repeater task nb300_10pings starting 0.1 seconds late 2016-07-10 03:14:01,534 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:14:31,571 INFO repeater task nb300_lastensq starting 0.6 seconds late 2016-07-10 03:14:34,160 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:14:34,160 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:14:21 2016-07-10 03:16:24,320 INFO repeater task nb300_10pings starting 3.0 seconds late 2016-07-10 03:16:24,500 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:16:34,514 INFO repeater task nb300_lastensq starting 3.5 seconds late 2016-07-10 03:16:37,155 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:16:37,155 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:16:21 2016-07-10 03:17:22,225 INFO repeater task ashtech_hcorrstats starting 0.9 seconds late 2016-07-10 03:17:23,238 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:18:33,325 INFO repeater task nb300_lastensq starting 2.3 seconds late 2016-07-10 03:18:35,953 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:18:35,953 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:18:21 2016-07-10 03:18:45,968 INFO repeater task nb300_10pings starting 4.7 seconds late 2016-07-10 03:18:46,147 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:20:31,299 INFO repeater task nb300_lastensq starting 0.3 seconds late 2016-07-10 03:20:33,955 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 03:20:33,956 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:20:21 2016-07-10 03:21:03,999 INFO repeater task nb300_10pings starting 2.7 seconds late 2016-07-10 03:21:04,175 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:22:24,270 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2016-07-10 03:22:25,285 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:22:35,299 INFO repeater task nb300_lastensq starting 4.3 seconds late 2016-07-10 03:22:37,905 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:22:37,906 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:22:21 2016-07-10 03:23:22,973 INFO repeater task nb300_10pings starting 1.7 seconds late 2016-07-10 03:23:23,150 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:24:33,253 INFO repeater task nb300_lastensq starting 2.3 seconds late 2016-07-10 03:24:35,862 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:24:35,862 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:24:21 2016-07-10 03:25:45,969 INFO repeater task nb300_10pings starting 4.7 seconds late 2016-07-10 03:25:46,148 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:26:31,214 INFO repeater task nb300_lastensq starting 0.2 seconds late 2016-07-10 03:26:33,822 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:26:33,823 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:26:21 2016-07-10 03:27:23,885 INFO repeater task nb300_quick starting 2.6 seconds late 2016-07-10 03:27:27,695 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 03:27:27,695 INFO repeater task nb300_3dayplots starting 6.4 seconds late 2016-07-10 03:27:31,615 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 03:27:31,616 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 03:27:31,616 INFO repeater task nb300_tsstats starting 10.3 seconds late 2016-07-10 03:27:32,496 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 03:27:32,497 INFO repeater task nb300_beamplots starting 11.2 seconds late 2016-07-10 03:27:34,348 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:27:34,348 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 03:27:34,348 INFO repeater task nb300_beamplots starting 13.1 seconds late 2016-07-10 03:27:36,265 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:27:36,265 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 03:27:36,265 INFO repeater task ashtech_hcorrstats starting 15.0 seconds late 2016-07-10 03:27:37,279 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:28:02,315 INFO repeater task nb300_10pings starting 1.0 seconds late 2016-07-10 03:28:02,494 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:28:32,535 INFO repeater task nb300_lastensq starting 1.5 seconds late 2016-07-10 03:28:35,148 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:28:35,149 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:28:21 2016-07-10 03:30:25,301 INFO repeater task nb300_10pings starting 4.0 seconds late 2016-07-10 03:30:25,478 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:30:35,492 INFO repeater task nb300_lastensq starting 4.5 seconds late 2016-07-10 03:30:38,122 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:30:38,122 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:30:21 2016-07-10 03:32:23,256 INFO repeater task ashtech_hcorrstats starting 2.0 seconds late 2016-07-10 03:32:24,270 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:32:34,284 INFO repeater task nb300_lastensq starting 3.3 seconds late 2016-07-10 03:32:36,932 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:32:36,933 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:32:21 2016-07-10 03:32:41,939 INFO repeater task nb300_10pings starting 0.7 seconds late 2016-07-10 03:32:42,124 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:34:32,280 INFO repeater task nb300_lastensq starting 1.3 seconds late 2016-07-10 03:34:34,907 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:34:34,907 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:34:21 2016-07-10 03:35:04,945 INFO repeater task nb300_10pings starting 3.7 seconds late 2016-07-10 03:35:05,121 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:36:35,242 INFO repeater task nb300_lastensq starting 4.2 seconds late 2016-07-10 03:36:37,909 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 03:36:37,909 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:36:21 2016-07-10 03:37:22,972 INFO repeater task nb300_10pings starting 1.7 seconds late 2016-07-10 03:37:23,154 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:37:23,155 INFO repeater task ashtech_hcorrstats starting 1.9 seconds late 2016-07-10 03:37:24,171 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:38:34,274 INFO repeater task nb300_lastensq starting 3.3 seconds late 2016-07-10 03:38:36,904 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:38:36,905 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:38:21 2016-07-10 03:39:41,995 INFO repeater task nb300_10pings starting 0.7 seconds late 2016-07-10 03:39:42,172 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:40:32,246 INFO repeater task nb300_lastensq starting 1.2 seconds late 2016-07-10 03:40:34,894 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:40:34,894 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:40:21 2016-07-10 03:42:05,012 INFO repeater task nb300_10pings starting 3.7 seconds late 2016-07-10 03:42:05,192 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:42:25,221 INFO repeater task nb300_quick starting 3.9 seconds late 2016-07-10 03:42:29,024 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 03:42:29,025 INFO repeater task nb300_3dayplots starting 7.7 seconds late 2016-07-10 03:42:32,912 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 03:42:32,912 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 03:42:32,913 INFO repeater task nb300_tsstats starting 11.6 seconds late 2016-07-10 03:42:33,814 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 03:42:33,814 INFO repeater task nb300_beamplots starting 12.5 seconds late 2016-07-10 03:42:35,669 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:42:35,670 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 03:42:35,670 INFO repeater task nb300_beamplots starting 14.4 seconds late 2016-07-10 03:42:37,614 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:42:37,615 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 03:42:37,615 INFO repeater task ashtech_hcorrstats starting 16.3 seconds late 2016-07-10 03:42:38,649 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:42:43,652 INFO repeater task nb300_lastensq starting 12.7 seconds late 2016-07-10 03:42:46,282 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:42:46,283 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:42:21 2016-07-10 03:44:21,420 INFO repeater task nb300_10pings starting 0.1 seconds late 2016-07-10 03:44:21,600 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:44:31,612 INFO repeater task nb300_lastensq starting 0.6 seconds late 2016-07-10 03:44:34,268 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 03:44:34,268 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:44:21 2016-07-10 03:46:34,427 INFO repeater task nb300_lastensq starting 3.4 seconds late 2016-07-10 03:46:37,068 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:46:37,068 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:46:21 2016-07-10 03:46:42,075 INFO repeater task nb300_10pings starting 0.8 seconds late 2016-07-10 03:46:42,256 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:47:22,318 INFO repeater task ashtech_hcorrstats starting 1.0 seconds late 2016-07-10 03:47:23,337 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:48:33,418 INFO repeater task nb300_lastensq starting 2.4 seconds late 2016-07-10 03:48:36,073 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 03:48:36,074 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:48:21 2016-07-10 03:49:06,104 INFO repeater task nb300_10pings starting 4.8 seconds late 2016-07-10 03:49:06,284 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:50:31,399 INFO repeater task nb300_lastensq starting 0.4 seconds late 2016-07-10 03:50:34,034 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:50:34,034 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:50:21 2016-07-10 03:51:24,106 INFO repeater task nb300_10pings starting 2.8 seconds late 2016-07-10 03:51:24,286 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:52:24,379 INFO repeater task ashtech_hcorrstats starting 3.1 seconds late 2016-07-10 03:52:25,402 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:52:35,415 INFO repeater task nb300_lastensq starting 4.4 seconds late 2016-07-10 03:52:38,078 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 03:52:38,078 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:52:21 2016-07-10 03:53:43,167 INFO repeater task nb300_10pings starting 1.9 seconds late 2016-07-10 03:53:43,344 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:54:33,415 INFO repeater task nb300_lastensq starting 2.4 seconds late 2016-07-10 03:54:36,042 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:54:36,042 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:54:21 2016-07-10 03:56:06,169 INFO repeater task nb300_10pings starting 4.9 seconds late 2016-07-10 03:56:06,351 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:56:31,380 INFO repeater task nb300_lastensq starting 0.4 seconds late 2016-07-10 03:56:34,007 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 03:56:34,008 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:56:21 2016-07-10 03:57:24,073 INFO repeater task nb300_quick starting 2.8 seconds late 2016-07-10 03:57:27,889 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 03:57:27,890 INFO repeater task nb300_3dayplots starting 6.6 seconds late 2016-07-10 03:57:31,786 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 03:57:31,786 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 03:57:31,787 INFO repeater task nb300_tsstats starting 10.5 seconds late 2016-07-10 03:57:32,686 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 03:57:32,686 INFO repeater task nb300_beamplots starting 11.4 seconds late 2016-07-10 03:57:34,555 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:57:34,555 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 03:57:34,555 INFO repeater task nb300_beamplots starting 13.3 seconds late 2016-07-10 03:57:36,503 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 03:57:36,503 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 03:57:36,503 INFO repeater task ashtech_hcorrstats starting 15.2 seconds late 2016-07-10 03:57:37,545 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 03:58:22,604 INFO repeater task nb300_10pings starting 1.3 seconds late 2016-07-10 03:58:22,787 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 03:58:32,801 INFO repeater task nb300_lastensq starting 1.8 seconds late 2016-07-10 03:58:35,452 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 03:58:35,452 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 03:58:21 2016-07-10 04:00:35,593 INFO repeater task nb300_lastensq starting 4.6 seconds late 2016-07-10 04:00:38,271 INFO repeater task nb300_lastensq ended after 2.7 seconds 2016-07-10 04:00:38,271 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:00:21 2016-07-10 04:00:43,278 INFO repeater task nb300_10pings starting 2.0 seconds late 2016-07-10 04:00:43,459 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:02:23,598 INFO repeater task ashtech_hcorrstats starting 2.3 seconds late 2016-07-10 04:02:24,617 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:02:34,629 INFO repeater task nb300_lastensq starting 3.6 seconds late 2016-07-10 04:02:37,217 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:02:37,217 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:02:21 2016-07-10 04:03:02,254 INFO repeater task nb300_10pings starting 1.0 seconds late 2016-07-10 04:03:02,433 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:04:32,559 INFO repeater task nb300_lastensq starting 1.6 seconds late 2016-07-10 04:04:35,167 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:04:35,168 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:04:21 2016-07-10 04:05:25,225 INFO repeater task nb300_10pings starting 3.9 seconds late 2016-07-10 04:05:25,402 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:06:35,501 INFO repeater task nb300_lastensq starting 4.5 seconds late 2016-07-10 04:06:38,083 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:06:38,084 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:06:21 2016-07-10 04:07:23,153 INFO repeater task ashtech_hcorrstats starting 1.9 seconds late 2016-07-10 04:07:24,181 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:07:44,211 INFO repeater task nb300_10pings starting 2.9 seconds late 2016-07-10 04:07:44,390 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:08:34,464 INFO repeater task nb300_lastensq starting 3.5 seconds late 2016-07-10 04:08:37,051 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:08:37,051 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:08:21 2016-07-10 04:10:02,172 INFO repeater task nb300_10pings starting 0.9 seconds late 2016-07-10 04:10:02,355 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:10:32,394 INFO repeater task nb300_lastensq starting 1.4 seconds late 2016-07-10 04:10:34,970 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:10:34,970 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:10:21 2016-07-10 04:12:25,106 INFO repeater task nb300_10pings starting 3.8 seconds late 2016-07-10 04:12:25,288 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:12:25,288 INFO repeater task nb300_quick starting 4.0 seconds late 2016-07-10 04:12:29,074 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 04:12:29,074 INFO repeater task nb300_3dayplots starting 7.8 seconds late 2016-07-10 04:12:33,014 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 04:12:33,014 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 04:12:33,015 INFO repeater task nb300_tsstats starting 11.7 seconds late 2016-07-10 04:12:33,891 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 04:12:33,892 INFO repeater task nb300_beamplots starting 12.6 seconds late 2016-07-10 04:12:35,772 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:12:35,773 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:12:35,773 INFO repeater task nb300_beamplots starting 14.5 seconds late 2016-07-10 04:12:37,710 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:12:37,711 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:12:37,711 INFO repeater task ashtech_hcorrstats starting 16.4 seconds late 2016-07-10 04:12:38,715 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:12:43,720 INFO repeater task nb300_lastensq starting 12.7 seconds late 2016-07-10 04:12:46,308 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:12:46,308 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:12:21 2016-07-10 04:14:31,451 INFO repeater task nb300_lastensq starting 0.5 seconds late 2016-07-10 04:14:34,029 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:14:34,030 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:14:21 2016-07-10 04:14:44,044 INFO repeater task nb300_10pings starting 2.8 seconds late 2016-07-10 04:14:44,227 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:16:34,392 INFO repeater task nb300_lastensq starting 3.4 seconds late 2016-07-10 04:16:36,975 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:16:36,976 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:16:21 2016-07-10 04:17:02,012 INFO repeater task nb300_10pings starting 0.7 seconds late 2016-07-10 04:17:02,195 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:17:22,227 INFO repeater task ashtech_hcorrstats starting 0.9 seconds late 2016-07-10 04:17:23,252 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:18:33,328 INFO repeater task nb300_lastensq starting 2.3 seconds late 2016-07-10 04:18:35,908 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:18:35,909 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:18:21 2016-07-10 04:19:25,982 INFO repeater task nb300_10pings starting 4.7 seconds late 2016-07-10 04:19:26,162 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:20:31,260 INFO repeater task nb300_lastensq starting 0.3 seconds late 2016-07-10 04:20:33,843 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:20:33,844 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:20:21 2016-07-10 04:21:43,946 INFO repeater task nb300_10pings starting 2.7 seconds late 2016-07-10 04:21:44,123 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:22:24,181 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2016-07-10 04:22:25,193 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:22:35,204 INFO repeater task nb300_lastensq starting 4.2 seconds late 2016-07-10 04:22:37,780 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:22:37,780 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:22:21 2016-07-10 04:24:02,904 INFO repeater task nb300_10pings starting 1.6 seconds late 2016-07-10 04:24:03,086 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:24:33,131 INFO repeater task nb300_lastensq starting 2.1 seconds late 2016-07-10 04:24:35,739 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:24:35,739 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:24:21 2016-07-10 04:26:25,868 INFO repeater task nb300_10pings starting 4.6 seconds late 2016-07-10 04:26:26,049 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:26:31,054 INFO repeater task nb300_lastensq starting 0.1 seconds late 2016-07-10 04:26:33,651 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:26:33,651 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:26:21 2016-07-10 04:27:23,720 INFO repeater task nb300_quick starting 2.4 seconds late 2016-07-10 04:27:27,499 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 04:27:27,499 INFO repeater task nb300_3dayplots starting 6.2 seconds late 2016-07-10 04:27:31,428 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 04:27:31,429 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 04:27:31,429 INFO repeater task nb300_tsstats starting 10.2 seconds late 2016-07-10 04:27:32,313 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 04:27:32,313 INFO repeater task nb300_beamplots starting 11.0 seconds late 2016-07-10 04:27:34,193 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:27:34,194 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:27:34,194 INFO repeater task nb300_beamplots starting 12.9 seconds late 2016-07-10 04:27:36,166 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:27:36,167 INFO repeater task nb300_beamplots ended after 2.0 seconds 2016-07-10 04:27:36,167 INFO repeater task ashtech_hcorrstats starting 14.9 seconds late 2016-07-10 04:27:37,167 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:28:32,250 INFO repeater task nb300_lastensq starting 1.3 seconds late 2016-07-10 04:28:34,853 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:28:34,854 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:28:21 2016-07-10 04:28:44,869 INFO repeater task nb300_10pings starting 3.6 seconds late 2016-07-10 04:28:45,048 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:30:35,202 INFO repeater task nb300_lastensq starting 4.2 seconds late 2016-07-10 04:30:37,824 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:30:37,824 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:30:21 2016-07-10 04:31:02,859 INFO repeater task nb300_10pings starting 1.6 seconds late 2016-07-10 04:31:03,036 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:32:23,149 INFO repeater task ashtech_hcorrstats starting 1.9 seconds late 2016-07-10 04:32:24,157 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:32:34,169 INFO repeater task nb300_lastensq starting 3.2 seconds late 2016-07-10 04:32:36,796 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:32:36,797 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:32:21 2016-07-10 04:33:21,860 INFO repeater task nb300_10pings starting 0.6 seconds late 2016-07-10 04:33:22,042 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:34:32,124 INFO repeater task nb300_lastensq starting 1.1 seconds late 2016-07-10 04:34:34,716 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:34:34,717 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:34:21 2016-07-10 04:35:44,806 INFO repeater task nb300_10pings starting 3.5 seconds late 2016-07-10 04:35:44,985 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:36:35,051 INFO repeater task nb300_lastensq starting 4.1 seconds late 2016-07-10 04:36:37,689 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:36:37,690 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:36:21 2016-07-10 04:37:22,749 INFO repeater task ashtech_hcorrstats starting 1.5 seconds late 2016-07-10 04:37:23,770 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:38:03,829 INFO repeater task nb300_10pings starting 2.6 seconds late 2016-07-10 04:38:04,009 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:38:34,053 INFO repeater task nb300_lastensq starting 3.1 seconds late 2016-07-10 04:38:36,641 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:38:36,641 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:38:21 2016-07-10 04:40:21,788 INFO repeater task nb300_10pings starting 0.5 seconds late 2016-07-10 04:40:21,968 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:40:31,983 INFO repeater task nb300_lastensq starting 1.0 seconds late 2016-07-10 04:40:34,616 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:40:34,617 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:40:21 2016-07-10 04:42:24,763 INFO repeater task nb300_quick starting 3.5 seconds late 2016-07-10 04:42:28,539 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 04:42:28,540 INFO repeater task nb300_3dayplots starting 7.3 seconds late 2016-07-10 04:42:32,491 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 04:42:32,492 INFO repeater task nb300_3dayplots ended after 4.0 seconds 2016-07-10 04:42:32,492 INFO repeater task nb300_tsstats starting 11.2 seconds late 2016-07-10 04:42:33,368 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 04:42:33,369 INFO repeater task nb300_beamplots starting 12.1 seconds late 2016-07-10 04:42:35,280 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:42:35,280 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:42:35,280 INFO repeater task nb300_beamplots starting 14.0 seconds late 2016-07-10 04:42:37,194 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:42:37,195 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:42:37,195 INFO repeater task ashtech_hcorrstats starting 15.9 seconds late 2016-07-10 04:42:38,205 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:42:43,209 INFO repeater task nb300_lastensq starting 12.2 seconds late 2016-07-10 04:42:45,830 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:42:45,830 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:42:21 2016-07-10 04:42:45,831 INFO repeater task nb300_10pings starting 4.6 seconds late 2016-07-10 04:42:46,006 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:44:31,158 INFO repeater task nb300_lastensq starting 0.2 seconds late 2016-07-10 04:44:33,753 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:44:33,753 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:44:21 2016-07-10 04:45:03,793 INFO repeater task nb300_10pings starting 2.5 seconds late 2016-07-10 04:45:03,978 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:46:34,103 INFO repeater task nb300_lastensq starting 3.1 seconds late 2016-07-10 04:46:36,732 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:46:36,732 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:46:21 2016-07-10 04:47:21,798 INFO repeater task nb300_10pings starting 0.5 seconds late 2016-07-10 04:47:21,982 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:47:21,983 INFO repeater task ashtech_hcorrstats starting 0.7 seconds late 2016-07-10 04:47:22,991 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:48:33,085 INFO repeater task nb300_lastensq starting 2.1 seconds late 2016-07-10 04:48:35,687 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:48:35,687 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:48:21 2016-07-10 04:49:45,788 INFO repeater task nb300_10pings starting 4.5 seconds late 2016-07-10 04:49:45,972 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:50:31,019 INFO repeater task nb300_lastensq starting 0.0 seconds late 2016-07-10 04:50:33,635 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:50:33,635 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:50:21 2016-07-10 04:52:03,762 INFO repeater task nb300_10pings starting 2.5 seconds late 2016-07-10 04:52:03,940 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:52:23,972 INFO repeater task ashtech_hcorrstats starting 2.7 seconds late 2016-07-10 04:52:24,988 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:52:34,997 INFO repeater task nb300_lastensq starting 4.0 seconds late 2016-07-10 04:52:37,614 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:52:37,614 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:52:21 2016-07-10 04:54:22,768 INFO repeater task nb300_10pings starting 1.5 seconds late 2016-07-10 04:54:22,944 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:54:32,957 INFO repeater task nb300_lastensq starting 2.0 seconds late 2016-07-10 04:54:35,546 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:54:35,547 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:54:21 2016-07-10 04:56:35,705 INFO repeater task nb300_lastensq starting 4.7 seconds late 2016-07-10 04:56:38,323 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:56:38,323 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:56:21 2016-07-10 04:56:43,330 INFO repeater task nb300_10pings starting 2.1 seconds late 2016-07-10 04:56:43,508 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 04:57:23,562 INFO repeater task nb300_quick starting 2.3 seconds late 2016-07-10 04:57:27,356 INFO repeater task nb300_quick ended after 3.8 seconds 2016-07-10 04:57:27,357 INFO repeater task nb300_3dayplots starting 6.1 seconds late 2016-07-10 04:57:31,299 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2016-07-10 04:57:31,300 INFO repeater task nb300_3dayplots ended after 3.9 seconds 2016-07-10 04:57:31,302 INFO repeater task nb300_tsstats starting 10.0 seconds late 2016-07-10 04:57:32,194 INFO repeater task nb300_tsstats ended after 0.9 seconds 2016-07-10 04:57:32,195 INFO repeater task nb300_beamplots starting 10.9 seconds late 2016-07-10 04:57:34,118 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:57:34,119 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:57:34,119 INFO repeater task nb300_beamplots starting 12.8 seconds late 2016-07-10 04:57:36,015 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2016-07-10 04:57:36,015 INFO repeater task nb300_beamplots ended after 1.9 seconds 2016-07-10 04:57:36,015 INFO repeater task ashtech_hcorrstats starting 14.7 seconds late 2016-07-10 04:57:37,028 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 04:58:32,104 INFO repeater task nb300_lastensq starting 1.1 seconds late 2016-07-10 04:58:34,728 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 04:58:34,729 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 04:58:21 2016-07-10 04:59:04,768 INFO repeater task nb300_10pings starting 3.5 seconds late 2016-07-10 04:59:04,946 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 05:00:35,076 INFO repeater task nb300_lastensq starting 4.1 seconds late 2016-07-10 05:00:37,679 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 05:00:37,679 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 05:00:21 2016-07-10 05:01:22,747 INFO repeater task nb300_10pings starting 1.5 seconds late 2016-07-10 05:01:22,925 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 05:02:23,006 INFO repeater task ashtech_hcorrstats starting 1.7 seconds late 2016-07-10 05:02:24,024 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 05:02:34,038 INFO repeater task nb300_lastensq starting 3.0 seconds late 2016-07-10 05:02:36,640 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 05:02:36,640 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 05:02:21 2016-07-10 05:03:41,724 INFO repeater task nb300_10pings starting 0.4 seconds late 2016-07-10 05:03:41,902 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 05:04:31,974 INFO repeater task nb300_lastensq starting 1.0 seconds late 2016-07-10 05:04:34,584 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 05:04:34,584 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 05:04:21 2016-07-10 05:06:04,688 INFO repeater task nb300_10pings starting 3.4 seconds late 2016-07-10 05:06:04,867 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 05:06:34,910 INFO repeater task nb300_lastensq starting 3.9 seconds late 2016-07-10 05:06:37,539 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 05:06:37,540 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 05:06:21 2016-07-10 05:07:22,609 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2016-07-10 05:07:23,634 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2016-07-10 05:08:23,719 INFO repeater task nb300_10pings starting 2.4 seconds late 2016-07-10 05:08:23,895 INFO repeater task nb300_10pings ended after 0.2 seconds 2016-07-10 05:08:33,910 INFO repeater task nb300_lastensq starting 2.9 seconds late 2016-07-10 05:08:36,532 INFO repeater task nb300_lastensq ended after 2.6 seconds 2016-07-10 05:08:36,533 INFO repeater task nb300_lastensq: Time from /home/data/SP1612/proc/nb300/load/wrote_ens_time: 2016/07/10 05:08:21 2016-07-10 05:09:57,558 INFO Translate translate exiting, adu /home/data/SP1612/rbin/ashtech 2016-07-10 05:09:57,571 INFO Translate translate exiting, hdg /home/data/SP1612/rbin/gpshead 2016-07-10 05:09:57,605 INFO Translate translate exiting, gps /home/data/SP1612/rbin/gpsnav 2016-07-10 05:10:01,652 INFO root cleanup_figs not_logging 2016-07-10 05:10:01,660 INFO root Exiting DAS_while_logging.py