2017-09-19 01:30:41,412 INFO     repeater     task nb300_lastensq starting 0.4 seconds late
2017-09-19 01:30:44,175 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:30:44,175 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:30:31
2017-09-19 01:31:14,218 INFO     repeater     task nb300_10pings starting 2.4 seconds late
2017-09-19 01:31:14,400 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:32:34,513 INFO     repeater     task nb300_quick starting 2.7 seconds late
2017-09-19 01:32:42,142 INFO     repeater     task nb300_quick ended after 7.6 seconds
2017-09-19 01:32:42,143 INFO     repeater     task nb300_3dayplots starting 10.4 seconds late
2017-09-19 01:32:49,999 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 01:32:50,000 INFO     repeater     task nb300_3dayplots ended after 7.9 seconds
2017-09-19 01:32:50,000 INFO     repeater     task nb300_tsstats starting 18.2 seconds late
2017-09-19 01:32:51,164 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 01:32:51,164 INFO     repeater     task nb300_beamplots starting 19.4 seconds late
2017-09-19 01:32:53,129 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 01:32:53,129 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 01:32:53,130 INFO     repeater     task nb300_beamplots starting 21.3 seconds late
2017-09-19 01:32:55,251 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 01:32:55,251 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 01:32:55,252 INFO     repeater     task ashtech_hcorrstats starting 23.5 seconds late
2017-09-19 01:32:56,328 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 01:33:01,333 INFO     repeater     task nb300_lastensq starting 20.3 seconds late
2017-09-19 01:33:04,069 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:33:04,069 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:32:31
2017-09-19 01:33:34,109 INFO     repeater     task nb300_10pings starting 2.3 seconds late
2017-09-19 01:33:34,290 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:34:44,384 INFO     repeater     task nb300_lastensq starting 3.4 seconds late
2017-09-19 01:34:47,179 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:34:47,179 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:34:31
2017-09-19 01:35:52,268 INFO     repeater     task nb300_10pings starting 0.5 seconds late
2017-09-19 01:35:52,452 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:36:42,528 INFO     repeater     task nb300_lastensq starting 1.5 seconds late
2017-09-19 01:36:45,252 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:36:45,253 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:36:31
2017-09-19 01:37:35,330 INFO     repeater     task ashtech_hcorrstats starting 3.5 seconds late
2017-09-19 01:37:36,459 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 01:38:16,513 INFO     repeater     task nb300_10pings starting 4.7 seconds late
2017-09-19 01:38:16,696 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:38:41,733 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2017-09-19 01:38:44,459 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:38:44,459 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:38:31
2017-09-19 01:40:34,607 INFO     repeater     task nb300_10pings starting 2.8 seconds late
2017-09-19 01:40:34,790 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:40:44,805 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2017-09-19 01:40:47,572 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:40:47,572 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:40:31
2017-09-19 01:42:32,727 INFO     repeater     task ashtech_hcorrstats starting 0.9 seconds late
2017-09-19 01:42:33,816 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 01:42:43,829 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2017-09-19 01:42:46,563 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:42:46,563 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:42:31
2017-09-19 01:42:56,575 INFO     repeater     task nb300_10pings starting 4.8 seconds late
2017-09-19 01:42:56,756 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:44:41,887 INFO     repeater     task nb300_lastensq starting 0.9 seconds late
2017-09-19 01:44:44,643 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:44:44,644 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:44:31
2017-09-19 01:45:14,690 INFO     repeater     task nb300_10pings starting 2.9 seconds late
2017-09-19 01:45:14,873 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:46:45,011 INFO     repeater     task nb300_lastensq starting 4.0 seconds late
2017-09-19 01:46:47,739 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:46:47,739 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:46:31
2017-09-19 01:47:32,807 INFO     repeater     task nb300_10pings starting 1.0 seconds late
2017-09-19 01:47:32,993 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:47:32,994 INFO     repeater     task nb300_quick starting 1.2 seconds late
2017-09-19 01:47:40,660 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 01:47:40,660 INFO     repeater     task nb300_3dayplots starting 8.9 seconds late
2017-09-19 01:47:48,421 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 01:47:48,422 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 01:47:48,422 INFO     repeater     task nb300_tsstats starting 16.6 seconds late
2017-09-19 01:47:49,604 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 01:47:49,605 INFO     repeater     task nb300_beamplots starting 17.8 seconds late
2017-09-19 01:47:51,781 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 01:47:51,781 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 01:47:51,781 INFO     repeater     task nb300_beamplots starting 20.0 seconds late
2017-09-19 01:47:53,882 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 01:47:53,882 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 01:47:53,882 INFO     repeater     task ashtech_hcorrstats starting 22.1 seconds late
2017-09-19 01:47:54,950 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 01:48:45,019 INFO     repeater     task nb300_lastensq starting 4.0 seconds late
2017-09-19 01:48:47,791 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:48:47,791 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:48:31
2017-09-19 01:49:52,888 INFO     repeater     task nb300_10pings starting 1.1 seconds late
2017-09-19 01:49:53,069 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:50:43,134 INFO     repeater     task nb300_lastensq starting 2.1 seconds late
2017-09-19 01:50:45,949 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:50:45,949 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:50:31
2017-09-19 01:52:16,088 INFO     repeater     task nb300_10pings starting 4.3 seconds late
2017-09-19 01:52:16,274 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:52:36,302 INFO     repeater     task ashtech_hcorrstats starting 4.5 seconds late
2017-09-19 01:52:37,385 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 01:52:42,391 INFO     repeater     task nb300_lastensq starting 1.4 seconds late
2017-09-19 01:52:45,178 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:52:45,179 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:52:31
2017-09-19 01:54:35,317 INFO     repeater     task nb300_10pings starting 3.5 seconds late
2017-09-19 01:54:35,520 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:54:45,530 INFO     repeater     task nb300_lastensq starting 4.5 seconds late
2017-09-19 01:54:48,268 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:54:48,268 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:54:31
2017-09-19 01:56:43,407 INFO     repeater     task nb300_lastensq starting 2.4 seconds late
2017-09-19 01:56:46,182 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 01:56:46,183 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:56:31
2017-09-19 01:56:56,193 INFO     repeater     task nb300_10pings starting 4.4 seconds late
2017-09-19 01:56:56,402 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 01:57:36,459 INFO     repeater     task ashtech_hcorrstats starting 4.7 seconds late
2017-09-19 01:57:37,536 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 01:58:42,621 INFO     repeater     task nb300_lastensq starting 1.6 seconds late
2017-09-19 01:58:45,358 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 01:58:45,358 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 01:58:31
2017-09-19 01:59:15,395 INFO     repeater     task nb300_10pings starting 3.6 seconds late
2017-09-19 01:59:15,578 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:00:45,682 INFO     repeater     task nb300_lastensq starting 4.7 seconds late
2017-09-19 02:00:48,495 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:00:48,496 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:00:31
2017-09-19 02:01:33,563 INFO     repeater     task nb300_10pings starting 1.8 seconds late
2017-09-19 02:01:33,745 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:02:33,834 INFO     repeater     task nb300_quick starting 2.1 seconds late
2017-09-19 02:02:41,460 INFO     repeater     task nb300_quick ended after 7.6 seconds
2017-09-19 02:02:41,460 INFO     repeater     task nb300_3dayplots starting 9.7 seconds late
2017-09-19 02:02:49,268 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 02:02:49,269 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 02:02:49,269 INFO     repeater     task nb300_tsstats starting 17.5 seconds late
2017-09-19 02:02:50,422 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 02:02:50,422 INFO     repeater     task nb300_beamplots starting 18.6 seconds late
2017-09-19 02:02:52,594 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:02:52,594 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 02:02:52,594 INFO     repeater     task nb300_beamplots starting 20.8 seconds late
2017-09-19 02:02:54,898 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:02:54,898 INFO     repeater     task nb300_beamplots ended after 2.3 seconds
2017-09-19 02:02:54,898 INFO     repeater     task ashtech_hcorrstats starting 23.1 seconds late
2017-09-19 02:02:55,974 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:03:00,979 INFO     repeater     task nb300_lastensq starting 20.0 seconds late
2017-09-19 02:03:03,715 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:03:03,715 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:02:31
2017-09-19 02:03:53,789 INFO     repeater     task nb300_10pings starting 2.0 seconds late
2017-09-19 02:03:53,971 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:04:44,043 INFO     repeater     task nb300_lastensq starting 3.0 seconds late
2017-09-19 02:04:46,717 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:04:46,718 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:04:31
2017-09-19 02:06:11,842 INFO     repeater     task nb300_10pings starting 0.1 seconds late
2017-09-19 02:06:12,026 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:06:42,070 INFO     repeater     task nb300_lastensq starting 1.1 seconds late
2017-09-19 02:06:44,800 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:06:44,801 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:06:31
2017-09-19 02:07:34,876 INFO     repeater     task ashtech_hcorrstats starting 3.1 seconds late
2017-09-19 02:07:35,954 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:08:36,021 INFO     repeater     task nb300_10pings starting 4.2 seconds late
2017-09-19 02:08:36,210 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:08:41,217 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2017-09-19 02:08:43,878 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:08:43,879 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:08:31
2017-09-19 02:10:44,041 INFO     repeater     task nb300_lastensq starting 3.0 seconds late
2017-09-19 02:10:46,776 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:10:46,777 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:10:31
2017-09-19 02:10:56,788 INFO     repeater     task nb300_10pings starting 5.0 seconds late
2017-09-19 02:10:56,970 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:12:32,088 INFO     repeater     task ashtech_hcorrstats starting 0.3 seconds late
2017-09-19 02:12:33,160 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:12:43,174 INFO     repeater     task nb300_lastensq starting 2.2 seconds late
2017-09-19 02:12:45,906 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:12:45,906 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:12:31
2017-09-19 02:13:15,943 INFO     repeater     task nb300_10pings starting 4.2 seconds late
2017-09-19 02:13:16,131 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:14:41,236 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2017-09-19 02:14:43,925 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:14:43,928 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:14:31
2017-09-19 02:15:33,992 INFO     repeater     task nb300_10pings starting 2.2 seconds late
2017-09-19 02:15:34,178 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:16:44,271 INFO     repeater     task nb300_lastensq starting 3.3 seconds late
2017-09-19 02:16:47,010 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:16:47,010 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:16:31
2017-09-19 02:17:32,072 INFO     repeater     task nb300_quick starting 0.3 seconds late
2017-09-19 02:17:39,757 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 02:17:39,758 INFO     repeater     task nb300_3dayplots starting 8.0 seconds late
2017-09-19 02:17:47,579 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 02:17:47,579 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 02:17:47,580 INFO     repeater     task nb300_tsstats starting 15.8 seconds late
2017-09-19 02:17:48,734 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 02:17:48,735 INFO     repeater     task nb300_beamplots starting 17.0 seconds late
2017-09-19 02:17:50,685 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:17:50,685 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 02:17:50,686 INFO     repeater     task nb300_beamplots starting 18.9 seconds late
2017-09-19 02:17:53,030 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:17:53,031 INFO     repeater     task nb300_beamplots ended after 2.3 seconds
2017-09-19 02:17:53,031 INFO     repeater     task ashtech_hcorrstats starting 21.2 seconds late
2017-09-19 02:17:54,097 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:17:59,103 INFO     repeater     task nb300_10pings starting 7.3 seconds late
2017-09-19 02:17:59,286 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:18:44,351 INFO     repeater     task nb300_lastensq starting 3.4 seconds late
2017-09-19 02:18:47,063 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:18:47,063 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:18:31
2017-09-19 02:20:12,184 INFO     repeater     task nb300_10pings starting 0.4 seconds late
2017-09-19 02:20:12,368 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:20:42,412 INFO     repeater     task nb300_lastensq starting 1.4 seconds late
2017-09-19 02:20:45,169 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:20:45,169 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:20:31
2017-09-19 02:22:35,332 INFO     repeater     task nb300_10pings starting 3.5 seconds late
2017-09-19 02:22:35,515 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:22:35,517 INFO     repeater     task ashtech_hcorrstats starting 3.7 seconds late
2017-09-19 02:22:36,581 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:22:41,587 INFO     repeater     task nb300_lastensq starting 0.6 seconds late
2017-09-19 02:22:44,313 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:22:44,314 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:22:31
2017-09-19 02:24:44,494 INFO     repeater     task nb300_lastensq starting 3.5 seconds late
2017-09-19 02:24:47,183 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:24:47,183 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:24:31
2017-09-19 02:24:52,190 INFO     repeater     task nb300_10pings starting 0.4 seconds late
2017-09-19 02:24:52,374 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:26:42,536 INFO     repeater     task nb300_lastensq starting 1.5 seconds late
2017-09-19 02:26:45,288 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:26:45,289 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:26:31
2017-09-19 02:27:15,332 INFO     repeater     task nb300_10pings starting 3.5 seconds late
2017-09-19 02:27:15,515 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:27:35,547 INFO     repeater     task ashtech_hcorrstats starting 3.8 seconds late
2017-09-19 02:27:36,616 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:28:41,688 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2017-09-19 02:28:44,395 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:28:44,395 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:28:31
2017-09-19 02:29:34,470 INFO     repeater     task nb300_10pings starting 2.7 seconds late
2017-09-19 02:29:34,655 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:30:44,755 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2017-09-19 02:30:47,495 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:30:47,496 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:30:31
2017-09-19 02:31:52,585 INFO     repeater     task nb300_10pings starting 0.8 seconds late
2017-09-19 02:31:52,766 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:32:32,814 INFO     repeater     task nb300_quick starting 1.0 seconds late
2017-09-19 02:32:40,490 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 02:32:40,491 INFO     repeater     task nb300_3dayplots starting 8.7 seconds late
2017-09-19 02:32:48,300 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 02:32:48,300 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 02:32:48,300 INFO     repeater     task nb300_tsstats starting 16.5 seconds late
2017-09-19 02:32:49,445 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 02:32:49,446 INFO     repeater     task nb300_beamplots starting 17.7 seconds late
2017-09-19 02:32:51,684 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:32:51,685 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 02:32:51,685 INFO     repeater     task nb300_beamplots starting 19.9 seconds late
2017-09-19 02:32:53,732 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:32:53,732 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 02:32:53,733 INFO     repeater     task ashtech_hcorrstats starting 21.9 seconds late
2017-09-19 02:32:54,797 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:32:59,803 INFO     repeater     task nb300_lastensq starting 18.8 seconds late
2017-09-19 02:33:02,582 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:33:02,582 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:32:31
2017-09-19 02:34:12,665 INFO     repeater     task nb300_10pings starting 0.9 seconds late
2017-09-19 02:34:12,848 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:34:42,893 INFO     repeater     task nb300_lastensq starting 1.9 seconds late
2017-09-19 02:34:45,583 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:34:45,583 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:34:31
2017-09-19 02:36:35,728 INFO     repeater     task nb300_10pings starting 3.9 seconds late
2017-09-19 02:36:35,909 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:36:45,922 INFO     repeater     task nb300_lastensq starting 4.9 seconds late
2017-09-19 02:36:48,675 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:36:48,676 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:36:31
2017-09-19 02:37:33,742 INFO     repeater     task ashtech_hcorrstats starting 2.0 seconds late
2017-09-19 02:37:34,810 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:38:44,908 INFO     repeater     task nb300_lastensq starting 3.9 seconds late
2017-09-19 02:38:47,614 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:38:47,614 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:38:31
2017-09-19 02:38:52,617 INFO     repeater     task nb300_10pings starting 0.8 seconds late
2017-09-19 02:38:52,846 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:40:42,990 INFO     repeater     task nb300_lastensq starting 2.0 seconds late
2017-09-19 02:40:45,744 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:40:45,744 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:40:31
2017-09-19 02:41:15,789 INFO     repeater     task nb300_10pings starting 4.0 seconds late
2017-09-19 02:41:15,974 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:42:36,088 INFO     repeater     task ashtech_hcorrstats starting 4.3 seconds late
2017-09-19 02:42:37,158 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:42:42,164 INFO     repeater     task nb300_lastensq starting 1.2 seconds late
2017-09-19 02:42:44,918 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:42:44,919 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:42:31
2017-09-19 02:43:34,993 INFO     repeater     task nb300_10pings starting 3.2 seconds late
2017-09-19 02:43:35,178 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:44:45,279 INFO     repeater     task nb300_lastensq starting 4.3 seconds late
2017-09-19 02:44:47,980 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:44:47,980 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:44:31
2017-09-19 02:45:53,060 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2017-09-19 02:45:53,248 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:46:43,313 INFO     repeater     task nb300_lastensq starting 2.3 seconds late
2017-09-19 02:46:46,075 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:46:46,075 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:46:31
2017-09-19 02:47:36,140 INFO     repeater     task nb300_quick starting 4.4 seconds late
2017-09-19 02:47:43,800 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 02:47:43,800 INFO     repeater     task nb300_3dayplots starting 12.0 seconds late
2017-09-19 02:47:51,610 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 02:47:51,610 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 02:47:51,610 INFO     repeater     task nb300_tsstats starting 19.8 seconds late
2017-09-19 02:47:52,752 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 02:47:52,753 INFO     repeater     task nb300_beamplots starting 21.0 seconds late
2017-09-19 02:47:54,912 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:47:54,913 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 02:47:54,913 INFO     repeater     task nb300_beamplots starting 23.1 seconds late
2017-09-19 02:47:56,991 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 02:47:56,991 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 02:47:56,992 INFO     repeater     task ashtech_hcorrstats starting 25.2 seconds late
2017-09-19 02:47:58,063 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:48:13,081 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2017-09-19 02:48:13,264 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:48:43,307 INFO     repeater     task nb300_lastensq starting 2.3 seconds late
2017-09-19 02:48:46,055 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:48:46,056 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:48:31
2017-09-19 02:50:36,203 INFO     repeater     task nb300_10pings starting 4.4 seconds late
2017-09-19 02:50:36,387 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:50:41,394 INFO     repeater     task nb300_lastensq starting 0.4 seconds late
2017-09-19 02:50:44,074 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:50:44,074 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:50:31
2017-09-19 02:52:34,219 INFO     repeater     task ashtech_hcorrstats starting 2.4 seconds late
2017-09-19 02:52:35,288 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:52:45,302 INFO     repeater     task nb300_lastensq starting 4.3 seconds late
2017-09-19 02:52:48,075 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 02:52:48,076 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:52:31
2017-09-19 02:52:53,081 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2017-09-19 02:52:53,267 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:54:43,428 INFO     repeater     task nb300_lastensq starting 2.4 seconds late
2017-09-19 02:54:46,122 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:54:46,122 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:54:31
2017-09-19 02:55:16,160 INFO     repeater     task nb300_10pings starting 4.4 seconds late
2017-09-19 02:55:16,344 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:56:41,471 INFO     repeater     task nb300_lastensq starting 0.5 seconds late
2017-09-19 02:56:44,220 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:56:44,220 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:56:31
2017-09-19 02:57:34,295 INFO     repeater     task nb300_10pings starting 2.5 seconds late
2017-09-19 02:57:34,481 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 02:57:34,483 INFO     repeater     task ashtech_hcorrstats starting 2.7 seconds late
2017-09-19 02:57:35,553 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 02:58:45,638 INFO     repeater     task nb300_lastensq starting 4.6 seconds late
2017-09-19 02:58:48,379 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 02:58:48,379 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 02:58:31
2017-09-19 02:59:53,462 INFO     repeater     task nb300_10pings starting 1.7 seconds late
2017-09-19 02:59:53,689 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:00:43,749 INFO     repeater     task nb300_lastensq starting 2.7 seconds late
2017-09-19 03:00:46,498 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:00:46,498 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:00:31
2017-09-19 03:02:16,627 INFO     repeater     task nb300_10pings starting 4.8 seconds late
2017-09-19 03:02:16,810 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:02:31,832 INFO     repeater     task nb300_quick starting 0.1 seconds late
2017-09-19 03:02:39,509 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 03:02:39,509 INFO     repeater     task nb300_3dayplots starting 7.7 seconds late
2017-09-19 03:02:47,292 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 03:02:47,292 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 03:02:47,293 INFO     repeater     task nb300_tsstats starting 15.5 seconds late
2017-09-19 03:02:48,435 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 03:02:48,435 INFO     repeater     task nb300_beamplots starting 16.7 seconds late
2017-09-19 03:02:50,652 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:02:50,652 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 03:02:50,652 INFO     repeater     task nb300_beamplots starting 18.9 seconds late
2017-09-19 03:02:52,906 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:02:52,906 INFO     repeater     task nb300_beamplots ended after 2.3 seconds
2017-09-19 03:02:52,907 INFO     repeater     task ashtech_hcorrstats starting 21.1 seconds late
2017-09-19 03:02:53,974 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:02:58,980 INFO     repeater     task nb300_lastensq starting 18.0 seconds late
2017-09-19 03:03:01,762 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:03:01,763 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:02:31
2017-09-19 03:04:31,894 INFO     repeater     task nb300_10pings starting 0.1 seconds late
2017-09-19 03:04:32,080 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:04:42,086 INFO     repeater     task nb300_lastensq starting 1.1 seconds late
2017-09-19 03:04:44,809 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:04:44,810 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:04:31
2017-09-19 03:06:44,984 INFO     repeater     task nb300_lastensq starting 4.0 seconds late
2017-09-19 03:06:47,751 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:06:47,752 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:06:31
2017-09-19 03:06:52,759 INFO     repeater     task nb300_10pings starting 1.0 seconds late
2017-09-19 03:06:52,944 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:07:33,003 INFO     repeater     task ashtech_hcorrstats starting 1.2 seconds late
2017-09-19 03:07:34,079 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:08:44,173 INFO     repeater     task nb300_lastensq starting 3.2 seconds late
2017-09-19 03:08:46,910 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:08:46,910 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:08:31
2017-09-19 03:09:11,945 INFO     repeater     task nb300_10pings starting 0.2 seconds late
2017-09-19 03:09:12,128 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:10:42,253 INFO     repeater     task nb300_lastensq starting 1.3 seconds late
2017-09-19 03:10:44,957 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:10:44,957 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:10:31
2017-09-19 03:11:35,018 INFO     repeater     task nb300_10pings starting 3.2 seconds late
2017-09-19 03:11:35,202 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:12:35,284 INFO     repeater     task ashtech_hcorrstats starting 3.5 seconds late
2017-09-19 03:12:36,354 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:12:41,356 INFO     repeater     task nb300_lastensq starting 0.4 seconds late
2017-09-19 03:12:44,149 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:12:44,149 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:12:31
2017-09-19 03:13:54,253 INFO     repeater     task nb300_10pings starting 2.5 seconds late
2017-09-19 03:13:54,437 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:14:44,505 INFO     repeater     task nb300_lastensq starting 3.5 seconds late
2017-09-19 03:14:47,235 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:14:47,235 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:14:31
2017-09-19 03:16:12,353 INFO     repeater     task nb300_10pings starting 0.6 seconds late
2017-09-19 03:16:12,536 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:16:42,582 INFO     repeater     task nb300_lastensq starting 1.6 seconds late
2017-09-19 03:16:45,352 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:16:45,353 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:16:31
2017-09-19 03:17:35,423 INFO     repeater     task nb300_quick starting 3.6 seconds late
2017-09-19 03:17:43,026 INFO     repeater     task nb300_quick ended after 7.6 seconds
2017-09-19 03:17:43,027 INFO     repeater     task nb300_3dayplots starting 11.2 seconds late
2017-09-19 03:17:50,807 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 03:17:50,808 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 03:17:50,808 INFO     repeater     task nb300_tsstats starting 19.0 seconds late
2017-09-19 03:17:51,945 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 03:17:51,946 INFO     repeater     task nb300_beamplots starting 20.2 seconds late
2017-09-19 03:17:53,973 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:17:53,974 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 03:17:53,974 INFO     repeater     task nb300_beamplots starting 22.2 seconds late
2017-09-19 03:17:55,999 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:17:55,999 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 03:17:55,999 INFO     repeater     task ashtech_hcorrstats starting 24.2 seconds late
2017-09-19 03:17:57,068 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:18:32,117 INFO     repeater     task nb300_10pings starting 0.3 seconds late
2017-09-19 03:18:32,301 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:18:42,316 INFO     repeater     task nb300_lastensq starting 1.3 seconds late
2017-09-19 03:18:45,093 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:18:45,094 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:18:31
2017-09-19 03:20:45,240 INFO     repeater     task nb300_lastensq starting 4.2 seconds late
2017-09-19 03:20:47,950 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:20:47,950 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:20:31
2017-09-19 03:20:52,957 INFO     repeater     task nb300_10pings starting 1.2 seconds late
2017-09-19 03:20:53,139 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:22:33,267 INFO     repeater     task ashtech_hcorrstats starting 1.5 seconds late
2017-09-19 03:22:34,332 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:22:44,345 INFO     repeater     task nb300_lastensq starting 3.3 seconds late
2017-09-19 03:22:47,137 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:22:47,138 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:22:31
2017-09-19 03:23:12,173 INFO     repeater     task nb300_10pings starting 0.4 seconds late
2017-09-19 03:23:12,352 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:24:42,483 INFO     repeater     task nb300_lastensq starting 1.5 seconds late
2017-09-19 03:24:45,214 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:24:45,215 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:24:31
2017-09-19 03:25:35,269 INFO     repeater     task nb300_10pings starting 3.5 seconds late
2017-09-19 03:25:35,454 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:26:45,546 INFO     repeater     task nb300_lastensq starting 4.5 seconds late
2017-09-19 03:26:48,343 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:26:48,343 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:26:31
2017-09-19 03:27:33,405 INFO     repeater     task ashtech_hcorrstats starting 1.6 seconds late
2017-09-19 03:27:34,478 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:27:54,503 INFO     repeater     task nb300_10pings starting 2.7 seconds late
2017-09-19 03:27:54,688 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:28:44,762 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2017-09-19 03:28:47,550 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:28:47,551 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:28:31
2017-09-19 03:30:12,677 INFO     repeater     task nb300_10pings starting 0.9 seconds late
2017-09-19 03:30:12,861 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:30:42,906 INFO     repeater     task nb300_lastensq starting 1.9 seconds late
2017-09-19 03:30:45,639 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:30:45,639 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:30:31
2017-09-19 03:32:35,797 INFO     repeater     task nb300_10pings starting 4.0 seconds late
2017-09-19 03:32:35,981 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:32:35,981 INFO     repeater     task nb300_quick starting 4.2 seconds late
2017-09-19 03:32:43,718 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 03:32:43,718 INFO     repeater     task nb300_3dayplots starting 11.9 seconds late
2017-09-19 03:32:51,439 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 03:32:51,440 INFO     repeater     task nb300_3dayplots ended after 7.7 seconds
2017-09-19 03:32:51,440 INFO     repeater     task nb300_tsstats starting 19.7 seconds late
2017-09-19 03:32:52,586 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 03:32:52,589 INFO     repeater     task nb300_beamplots starting 20.8 seconds late
2017-09-19 03:32:54,626 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:32:54,626 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 03:32:54,626 INFO     repeater     task nb300_beamplots starting 22.8 seconds late
2017-09-19 03:32:56,694 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:32:56,694 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 03:32:56,694 INFO     repeater     task ashtech_hcorrstats starting 24.9 seconds late
2017-09-19 03:32:57,803 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:33:02,809 INFO     repeater     task nb300_lastensq starting 21.8 seconds late
2017-09-19 03:33:05,576 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:33:05,576 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:32:31
2017-09-19 03:34:45,695 INFO     repeater     task nb300_lastensq starting 4.7 seconds late
2017-09-19 03:34:48,491 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:34:48,491 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:34:31
2017-09-19 03:34:53,493 INFO     repeater     task nb300_10pings starting 1.7 seconds late
2017-09-19 03:34:53,679 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:36:43,817 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2017-09-19 03:36:46,593 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:36:46,594 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:36:31
2017-09-19 03:37:16,635 INFO     repeater     task nb300_10pings starting 4.9 seconds late
2017-09-19 03:37:16,819 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:37:31,844 INFO     repeater     task ashtech_hcorrstats starting 0.1 seconds late
2017-09-19 03:37:32,939 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:38:43,042 INFO     repeater     task nb300_lastensq starting 2.0 seconds late
2017-09-19 03:38:45,830 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:38:45,830 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:38:31
2017-09-19 03:39:35,902 INFO     repeater     task nb300_10pings starting 4.1 seconds late
2017-09-19 03:39:36,085 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:40:41,178 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2017-09-19 03:40:43,923 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:40:43,923 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:40:31
2017-09-19 03:41:54,024 INFO     repeater     task nb300_10pings starting 2.2 seconds late
2017-09-19 03:41:54,209 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:42:34,266 INFO     repeater     task ashtech_hcorrstats starting 2.5 seconds late
2017-09-19 03:42:35,376 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:42:45,389 INFO     repeater     task nb300_lastensq starting 4.4 seconds late
2017-09-19 03:42:48,155 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:42:48,156 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:42:31
2017-09-19 03:44:13,278 INFO     repeater     task nb300_10pings starting 1.5 seconds late
2017-09-19 03:44:13,462 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:44:43,507 INFO     repeater     task nb300_lastensq starting 2.5 seconds late
2017-09-19 03:44:46,299 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:44:46,299 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:44:31
2017-09-19 03:46:36,465 INFO     repeater     task nb300_10pings starting 4.7 seconds late
2017-09-19 03:46:36,696 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:46:41,706 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2017-09-19 03:46:44,462 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:46:44,462 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:46:31
2017-09-19 03:47:34,535 INFO     repeater     task nb300_quick starting 2.8 seconds late
2017-09-19 03:47:42,258 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 03:47:42,258 INFO     repeater     task nb300_3dayplots starting 10.5 seconds late
2017-09-19 03:47:49,998 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 03:47:49,999 INFO     repeater     task nb300_3dayplots ended after 7.7 seconds
2017-09-19 03:47:49,999 INFO     repeater     task nb300_tsstats starting 18.2 seconds late
2017-09-19 03:47:51,143 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 03:47:51,143 INFO     repeater     task nb300_beamplots starting 19.4 seconds late
2017-09-19 03:47:53,378 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:47:53,378 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 03:47:53,379 INFO     repeater     task nb300_beamplots starting 21.6 seconds late
2017-09-19 03:47:55,659 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 03:47:55,659 INFO     repeater     task nb300_beamplots ended after 2.3 seconds
2017-09-19 03:47:55,659 INFO     repeater     task ashtech_hcorrstats starting 23.9 seconds late
2017-09-19 03:47:56,751 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:48:41,811 INFO     repeater     task nb300_lastensq starting 0.8 seconds late
2017-09-19 03:48:44,600 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:48:44,600 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:48:31
2017-09-19 03:48:54,615 INFO     repeater     task nb300_10pings starting 2.8 seconds late
2017-09-19 03:48:54,797 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:50:44,959 INFO     repeater     task nb300_lastensq starting 4.0 seconds late
2017-09-19 03:50:47,724 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:50:47,725 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:50:31
2017-09-19 03:51:12,755 INFO     repeater     task nb300_10pings starting 1.0 seconds late
2017-09-19 03:51:12,982 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:52:33,082 INFO     repeater     task ashtech_hcorrstats starting 1.3 seconds late
2017-09-19 03:52:34,218 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:52:44,231 INFO     repeater     task nb300_lastensq starting 3.2 seconds late
2017-09-19 03:52:46,961 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:52:46,961 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:52:31
2017-09-19 03:53:32,028 INFO     repeater     task nb300_10pings starting 0.2 seconds late
2017-09-19 03:53:32,212 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:54:42,316 INFO     repeater     task nb300_lastensq starting 1.3 seconds late
2017-09-19 03:54:45,107 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:54:45,107 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:54:31
2017-09-19 03:55:55,207 INFO     repeater     task nb300_10pings starting 3.4 seconds late
2017-09-19 03:55:55,392 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:56:45,461 INFO     repeater     task nb300_lastensq starting 4.5 seconds late
2017-09-19 03:56:48,203 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 03:56:48,203 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:56:31
2017-09-19 03:57:33,254 INFO     repeater     task ashtech_hcorrstats starting 1.5 seconds late
2017-09-19 03:57:34,323 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 03:58:14,374 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2017-09-19 03:58:14,557 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 03:58:44,601 INFO     repeater     task nb300_lastensq starting 3.6 seconds late
2017-09-19 03:58:47,409 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 03:58:47,409 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 03:58:31
2017-09-19 04:00:32,555 INFO     repeater     task nb300_10pings starting 0.8 seconds late
2017-09-19 04:00:32,737 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:00:42,751 INFO     repeater     task nb300_lastensq starting 1.8 seconds late
2017-09-19 04:00:45,530 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 04:00:45,531 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:00:31
2017-09-19 04:02:35,679 INFO     repeater     task nb300_quick starting 3.9 seconds late
2017-09-19 04:02:43,418 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 04:02:43,418 INFO     repeater     task nb300_3dayplots starting 11.6 seconds late
2017-09-19 04:02:51,231 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 04:02:51,232 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 04:02:51,232 INFO     repeater     task nb300_tsstats starting 19.4 seconds late
2017-09-19 04:02:52,390 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 04:02:52,391 INFO     repeater     task nb300_beamplots starting 20.6 seconds late
2017-09-19 04:02:54,560 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:02:54,561 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 04:02:54,561 INFO     repeater     task nb300_beamplots starting 22.8 seconds late
2017-09-19 04:02:56,692 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:02:56,692 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 04:02:56,692 INFO     repeater     task ashtech_hcorrstats starting 24.9 seconds late
2017-09-19 04:02:57,759 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:03:02,763 INFO     repeater     task nb300_lastensq starting 21.8 seconds late
2017-09-19 04:03:05,440 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:03:05,441 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:02:31
2017-09-19 04:03:05,441 INFO     repeater     task nb300_10pings starting 13.7 seconds late
2017-09-19 04:03:05,620 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:04:45,744 INFO     repeater     task nb300_lastensq starting 4.7 seconds late
2017-09-19 04:04:48,487 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:04:48,487 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:04:31
2017-09-19 04:05:13,516 INFO     repeater     task nb300_10pings starting 1.7 seconds late
2017-09-19 04:05:13,696 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:06:43,833 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2017-09-19 04:06:46,498 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:06:46,499 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:06:31
2017-09-19 04:07:36,574 INFO     repeater     task nb300_10pings starting 4.8 seconds late
2017-09-19 04:07:36,755 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:07:36,757 INFO     repeater     task ashtech_hcorrstats starting 5.0 seconds late
2017-09-19 04:07:37,870 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:08:42,964 INFO     repeater     task nb300_lastensq starting 2.0 seconds late
2017-09-19 04:08:45,646 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:08:45,647 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:08:31
2017-09-19 04:09:55,753 INFO     repeater     task nb300_10pings starting 4.0 seconds late
2017-09-19 04:09:55,938 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:10:41,001 INFO     repeater     task nb300_lastensq starting 0.0 seconds late
2017-09-19 04:10:43,724 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:10:43,725 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:10:31
2017-09-19 04:12:13,847 INFO     repeater     task nb300_10pings starting 2.1 seconds late
2017-09-19 04:12:14,033 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:12:34,064 INFO     repeater     task ashtech_hcorrstats starting 2.3 seconds late
2017-09-19 04:12:35,131 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:12:45,145 INFO     repeater     task nb300_lastensq starting 4.1 seconds late
2017-09-19 04:12:47,837 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:12:47,837 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:12:31
2017-09-19 04:14:32,981 INFO     repeater     task nb300_10pings starting 1.2 seconds late
2017-09-19 04:14:33,164 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:14:43,178 INFO     repeater     task nb300_lastensq starting 2.2 seconds late
2017-09-19 04:14:45,928 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 04:14:45,929 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:14:31
2017-09-19 04:16:41,081 INFO     repeater     task nb300_lastensq starting 0.1 seconds late
2017-09-19 04:16:43,798 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:16:43,798 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:16:31
2017-09-19 04:16:53,812 INFO     repeater     task nb300_10pings starting 2.0 seconds late
2017-09-19 04:16:53,994 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:17:34,049 INFO     repeater     task nb300_quick starting 2.3 seconds late
2017-09-19 04:17:41,823 INFO     repeater     task nb300_quick ended after 7.8 seconds
2017-09-19 04:17:41,823 INFO     repeater     task nb300_3dayplots starting 10.0 seconds late
2017-09-19 04:17:49,578 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 04:17:49,579 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 04:17:49,579 INFO     repeater     task nb300_tsstats starting 17.8 seconds late
2017-09-19 04:17:50,769 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 04:17:50,770 INFO     repeater     task nb300_beamplots starting 19.0 seconds late
2017-09-19 04:17:52,749 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:17:52,750 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 04:17:52,750 INFO     repeater     task nb300_beamplots starting 21.0 seconds late
2017-09-19 04:17:54,829 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:17:54,829 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 04:17:54,829 INFO     repeater     task ashtech_hcorrstats starting 23.0 seconds late
2017-09-19 04:17:55,932 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:18:46,000 INFO     repeater     task nb300_lastensq starting 5.0 seconds late
2017-09-19 04:18:48,691 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:18:48,691 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:18:31
2017-09-19 04:19:13,724 INFO     repeater     task nb300_10pings starting 1.9 seconds late
2017-09-19 04:19:13,908 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:20:44,034 INFO     repeater     task nb300_lastensq starting 3.0 seconds late
2017-09-19 04:20:46,783 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:20:46,784 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:20:31
2017-09-19 04:21:31,848 INFO     repeater     task nb300_10pings starting 0.1 seconds late
2017-09-19 04:21:32,033 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:22:32,126 INFO     repeater     task ashtech_hcorrstats starting 0.3 seconds late
2017-09-19 04:22:33,180 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:22:43,194 INFO     repeater     task nb300_lastensq starting 2.2 seconds late
2017-09-19 04:22:45,893 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:22:45,893 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:22:31
2017-09-19 04:23:55,995 INFO     repeater     task nb300_10pings starting 4.2 seconds late
2017-09-19 04:23:56,179 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:24:41,248 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2017-09-19 04:24:43,993 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:24:43,993 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:24:31
2017-09-19 04:26:14,125 INFO     repeater     task nb300_10pings starting 2.3 seconds late
2017-09-19 04:26:14,308 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:26:44,342 INFO     repeater     task nb300_lastensq starting 3.3 seconds late
2017-09-19 04:26:47,017 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:26:47,018 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:26:31
2017-09-19 04:27:32,085 INFO     repeater     task ashtech_hcorrstats starting 0.3 seconds late
2017-09-19 04:27:33,184 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:28:33,261 INFO     repeater     task nb300_10pings starting 1.5 seconds late
2017-09-19 04:28:33,468 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:28:43,478 INFO     repeater     task nb300_lastensq starting 2.5 seconds late
2017-09-19 04:28:46,189 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:28:46,190 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:28:31
2017-09-19 04:30:41,349 INFO     repeater     task nb300_lastensq starting 0.3 seconds late
2017-09-19 04:30:44,111 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 04:30:44,112 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:30:31
2017-09-19 04:30:54,125 INFO     repeater     task nb300_10pings starting 2.3 seconds late
2017-09-19 04:30:54,307 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:32:34,441 INFO     repeater     task nb300_quick starting 2.7 seconds late
2017-09-19 04:32:42,401 INFO     repeater     task nb300_quick ended after 8.0 seconds
2017-09-19 04:32:42,402 INFO     repeater     task nb300_3dayplots starting 10.6 seconds late
2017-09-19 04:32:50,220 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 04:32:50,220 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 04:32:50,220 INFO     repeater     task nb300_tsstats starting 18.4 seconds late
2017-09-19 04:32:51,366 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 04:32:51,367 INFO     repeater     task nb300_beamplots starting 19.6 seconds late
2017-09-19 04:32:53,546 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:32:53,546 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 04:32:53,547 INFO     repeater     task nb300_beamplots starting 21.8 seconds late
2017-09-19 04:32:55,850 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:32:55,851 INFO     repeater     task nb300_beamplots ended after 2.3 seconds
2017-09-19 04:32:55,851 INFO     repeater     task ashtech_hcorrstats starting 24.1 seconds late
2017-09-19 04:32:56,900 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 04:33:01,906 INFO     repeater     task nb300_lastensq starting 20.9 seconds late
2017-09-19 04:33:04,667 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 04:33:04,668 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:32:31
2017-09-19 04:33:14,681 INFO     repeater     task nb300_10pings starting 2.9 seconds late
2017-09-19 04:33:14,866 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:34:44,989 INFO     repeater     task nb300_lastensq starting 4.0 seconds late
2017-09-19 04:34:47,697 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:34:47,698 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:34:31
2017-09-19 04:35:32,763 INFO     repeater     task nb300_10pings starting 1.0 seconds late
2017-09-19 04:35:32,945 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:36:43,025 INFO     repeater     task nb300_lastensq starting 2.0 seconds late
2017-09-19 04:36:45,772 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:36:45,772 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:36:31
2017-09-19 04:37:35,845 INFO     repeater     task ashtech_hcorrstats starting 4.1 seconds late
2017-09-19 04:37:36,902 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 04:37:51,924 INFO     repeater     task nb300_10pings starting 0.1 seconds late
2017-09-19 04:37:52,110 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:38:42,173 INFO     repeater     task nb300_lastensq starting 1.2 seconds late
2017-09-19 04:38:44,869 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:38:44,870 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:38:31
2017-09-19 04:40:14,999 INFO     repeater     task nb300_10pings starting 3.2 seconds late
2017-09-19 04:40:15,186 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:40:45,231 INFO     repeater     task nb300_lastensq starting 4.2 seconds late
2017-09-19 04:40:47,980 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:40:47,980 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:40:31
2017-09-19 04:42:33,121 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2017-09-19 04:42:33,305 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:42:33,307 INFO     repeater     task ashtech_hcorrstats starting 1.5 seconds late
2017-09-19 04:42:34,343 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 04:42:44,356 INFO     repeater     task nb300_lastensq starting 3.4 seconds late
2017-09-19 04:42:47,070 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:42:47,070 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:42:31
2017-09-19 04:44:42,233 INFO     repeater     task nb300_lastensq starting 1.2 seconds late
2017-09-19 04:44:44,981 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:44:44,981 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:44:31
2017-09-19 04:44:54,996 INFO     repeater     task nb300_10pings starting 3.2 seconds late
2017-09-19 04:44:55,181 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:46:45,322 INFO     repeater     task nb300_lastensq starting 4.3 seconds late
2017-09-19 04:46:48,030 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:46:48,030 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:46:31
2017-09-19 04:47:13,067 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2017-09-19 04:47:13,256 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:47:33,286 INFO     repeater     task nb300_quick starting 1.5 seconds late
2017-09-19 04:47:41,127 INFO     repeater     task nb300_quick ended after 7.8 seconds
2017-09-19 04:47:41,127 INFO     repeater     task nb300_3dayplots starting 9.3 seconds late
2017-09-19 04:47:48,947 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 04:47:48,947 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 04:47:48,948 INFO     repeater     task nb300_tsstats starting 17.2 seconds late
2017-09-19 04:47:50,097 INFO     repeater     task nb300_tsstats ended after 1.1 seconds
2017-09-19 04:47:50,098 INFO     repeater     task nb300_beamplots starting 18.3 seconds late
2017-09-19 04:47:52,074 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:47:52,074 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 04:47:52,074 INFO     repeater     task nb300_beamplots starting 20.3 seconds late
2017-09-19 04:47:54,201 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 04:47:54,201 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 04:47:54,202 INFO     repeater     task ashtech_hcorrstats starting 22.4 seconds late
2017-09-19 04:47:55,233 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 04:48:45,295 INFO     repeater     task nb300_lastensq starting 4.3 seconds late
2017-09-19 04:48:47,992 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:48:47,993 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:48:31
2017-09-19 04:49:33,059 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2017-09-19 04:49:33,244 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:50:43,343 INFO     repeater     task nb300_lastensq starting 2.3 seconds late
2017-09-19 04:50:46,081 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:50:46,081 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:50:31
2017-09-19 04:51:56,162 INFO     repeater     task nb300_10pings starting 4.4 seconds late
2017-09-19 04:51:56,343 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:52:36,400 INFO     repeater     task ashtech_hcorrstats starting 4.6 seconds late
2017-09-19 04:52:37,447 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 04:52:42,453 INFO     repeater     task nb300_lastensq starting 1.5 seconds late
2017-09-19 04:52:45,224 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 04:52:45,225 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:52:31
2017-09-19 04:54:15,337 INFO     repeater     task nb300_10pings starting 3.6 seconds late
2017-09-19 04:54:15,544 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:54:45,583 INFO     repeater     task nb300_lastensq starting 4.6 seconds late
2017-09-19 04:54:48,299 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:54:48,299 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:54:31
2017-09-19 04:56:33,449 INFO     repeater     task nb300_10pings starting 1.7 seconds late
2017-09-19 04:56:33,633 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 04:56:43,648 INFO     repeater     task nb300_lastensq starting 2.6 seconds late
2017-09-19 04:56:46,419 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 04:56:46,420 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:56:31
2017-09-19 04:57:36,490 INFO     repeater     task ashtech_hcorrstats starting 4.7 seconds late
2017-09-19 04:57:37,528 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 04:58:42,623 INFO     repeater     task nb300_lastensq starting 1.6 seconds late
2017-09-19 04:58:45,356 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 04:58:45,356 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 04:58:31
2017-09-19 04:58:55,371 INFO     repeater     task nb300_10pings starting 3.6 seconds late
2017-09-19 04:58:55,552 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:00:45,701 INFO     repeater     task nb300_lastensq starting 4.7 seconds late
2017-09-19 05:00:48,472 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:00:48,472 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:00:31
2017-09-19 05:01:13,511 INFO     repeater     task nb300_10pings starting 1.7 seconds late
2017-09-19 05:01:13,696 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:02:33,798 INFO     repeater     task nb300_quick starting 2.0 seconds late
2017-09-19 05:02:41,474 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 05:02:41,475 INFO     repeater     task nb300_3dayplots starting 9.7 seconds late
2017-09-19 05:02:49,269 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 05:02:49,270 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 05:02:49,270 INFO     repeater     task nb300_tsstats starting 17.5 seconds late
2017-09-19 05:02:50,422 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 05:02:50,422 INFO     repeater     task nb300_beamplots starting 18.6 seconds late
2017-09-19 05:02:52,613 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 05:02:52,614 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 05:02:52,614 INFO     repeater     task nb300_beamplots starting 20.8 seconds late
2017-09-19 05:02:54,705 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 05:02:54,705 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 05:02:54,706 INFO     repeater     task ashtech_hcorrstats starting 22.9 seconds late
2017-09-19 05:02:55,734 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:03:00,739 INFO     repeater     task nb300_lastensq starting 19.7 seconds late
2017-09-19 05:03:03,505 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:03:03,505 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:02:31
2017-09-19 05:03:33,546 INFO     repeater     task nb300_10pings starting 1.8 seconds late
2017-09-19 05:03:33,726 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:04:43,824 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2017-09-19 05:04:46,513 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:04:46,513 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:04:31
2017-09-19 05:05:56,615 INFO     repeater     task nb300_10pings starting 4.8 seconds late
2017-09-19 05:05:56,797 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:06:41,857 INFO     repeater     task nb300_lastensq starting 0.9 seconds late
2017-09-19 05:06:44,617 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:06:44,618 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:06:31
2017-09-19 05:07:34,691 INFO     repeater     task ashtech_hcorrstats starting 2.9 seconds late
2017-09-19 05:07:35,746 INFO     repeater     task ashtech_hcorrstats ended after 1.1 seconds
2017-09-19 05:08:15,806 INFO     repeater     task nb300_10pings starting 4.0 seconds late
2017-09-19 05:08:15,988 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:08:41,020 INFO     repeater     task nb300_lastensq starting 0.0 seconds late
2017-09-19 05:08:43,762 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:08:43,762 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:08:31
2017-09-19 05:10:33,920 INFO     repeater     task nb300_10pings starting 2.1 seconds late
2017-09-19 05:10:34,104 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:10:44,119 INFO     repeater     task nb300_lastensq starting 3.1 seconds late
2017-09-19 05:10:46,890 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:10:46,891 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:10:31
2017-09-19 05:12:32,038 INFO     repeater     task ashtech_hcorrstats starting 0.3 seconds late
2017-09-19 05:12:33,083 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:12:43,092 INFO     repeater     task nb300_lastensq starting 2.1 seconds late
2017-09-19 05:12:45,873 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:12:45,874 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:12:31
2017-09-19 05:12:55,881 INFO     repeater     task nb300_10pings starting 4.1 seconds late
2017-09-19 05:12:56,067 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:14:41,213 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2017-09-19 05:14:43,929 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:14:43,930 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:14:31
2017-09-19 05:15:13,973 INFO     repeater     task nb300_10pings starting 2.2 seconds late
2017-09-19 05:15:14,160 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:16:44,279 INFO     repeater     task nb300_lastensq starting 3.3 seconds late
2017-09-19 05:16:47,061 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:16:47,061 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:16:31
2017-09-19 05:17:32,128 INFO     repeater     task nb300_10pings starting 0.3 seconds late
2017-09-19 05:17:32,309 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:17:32,310 INFO     repeater     task nb300_quick starting 0.5 seconds late
2017-09-19 05:17:39,993 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 05:17:39,994 INFO     repeater     task nb300_3dayplots starting 8.2 seconds late
2017-09-19 05:17:47,818 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 05:17:47,819 INFO     repeater     task nb300_3dayplots ended after 7.8 seconds
2017-09-19 05:17:47,819 INFO     repeater     task nb300_tsstats starting 16.0 seconds late
2017-09-19 05:17:48,977 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 05:17:48,977 INFO     repeater     task nb300_beamplots starting 17.2 seconds late
2017-09-19 05:17:50,953 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 05:17:50,954 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2017-09-19 05:17:50,954 INFO     repeater     task nb300_beamplots starting 19.2 seconds late
2017-09-19 05:17:53,067 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 05:17:53,067 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 05:17:53,068 INFO     repeater     task ashtech_hcorrstats starting 21.3 seconds late
2017-09-19 05:17:54,112 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:18:44,183 INFO     repeater     task nb300_lastensq starting 3.2 seconds late
2017-09-19 05:18:46,934 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:18:46,934 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:18:31
2017-09-19 05:19:52,019 INFO     repeater     task nb300_10pings starting 0.2 seconds late
2017-09-19 05:19:52,207 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:20:42,276 INFO     repeater     task nb300_lastensq starting 1.3 seconds late
2017-09-19 05:20:45,056 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:20:45,057 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:20:31
2017-09-19 05:22:15,189 INFO     repeater     task nb300_10pings starting 3.4 seconds late
2017-09-19 05:22:15,372 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:22:35,402 INFO     repeater     task ashtech_hcorrstats starting 3.6 seconds late
2017-09-19 05:22:36,448 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:22:41,453 INFO     repeater     task nb300_lastensq starting 0.5 seconds late
2017-09-19 05:22:44,238 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:22:44,239 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:22:31
2017-09-19 05:24:34,389 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2017-09-19 05:24:34,578 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:24:44,592 INFO     repeater     task nb300_lastensq starting 3.6 seconds late
2017-09-19 05:24:47,308 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:24:47,308 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:24:31
2017-09-19 05:26:42,482 INFO     repeater     task nb300_lastensq starting 1.5 seconds late
2017-09-19 05:26:45,261 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:26:45,262 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:26:31
2017-09-19 05:26:55,276 INFO     repeater     task nb300_10pings starting 3.5 seconds late
2017-09-19 05:26:55,459 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:27:35,520 INFO     repeater     task ashtech_hcorrstats starting 3.7 seconds late
2017-09-19 05:27:36,567 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:28:41,657 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2017-09-19 05:28:44,386 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:28:44,386 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:28:31
2017-09-19 05:29:14,424 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2017-09-19 05:29:14,610 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:30:44,738 INFO     repeater     task nb300_lastensq starting 3.7 seconds late
2017-09-19 05:30:47,537 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:30:47,537 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:30:31
2017-09-19 05:31:32,594 INFO     repeater     task nb300_10pings starting 0.8 seconds late
2017-09-19 05:31:32,776 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:32:32,867 INFO     repeater     task nb300_quick starting 1.1 seconds late
2017-09-19 05:32:40,572 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 05:32:40,572 INFO     repeater     task nb300_3dayplots starting 8.8 seconds late
2017-09-19 05:32:48,424 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 05:32:48,424 INFO     repeater     task nb300_3dayplots ended after 7.9 seconds
2017-09-19 05:32:48,425 INFO     repeater     task nb300_tsstats starting 16.6 seconds late
2017-09-19 05:32:49,577 INFO     repeater     task nb300_tsstats ended after 1.2 seconds
2017-09-19 05:32:49,577 INFO     repeater     task nb300_beamplots starting 17.8 seconds late
2017-09-19 05:32:51,820 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 05:32:51,821 INFO     repeater     task nb300_beamplots ended after 2.2 seconds
2017-09-19 05:32:51,821 INFO     repeater     task nb300_beamplots starting 20.0 seconds late
2017-09-19 05:32:53,881 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2017-09-19 05:32:53,881 INFO     repeater     task nb300_beamplots ended after 2.1 seconds
2017-09-19 05:32:53,881 INFO     repeater     task ashtech_hcorrstats starting 22.1 seconds late
2017-09-19 05:32:54,915 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:32:59,920 INFO     repeater     task nb300_lastensq starting 18.9 seconds late
2017-09-19 05:33:02,706 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:33:02,707 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:32:31
2017-09-19 05:33:52,775 INFO     repeater     task nb300_10pings starting 1.0 seconds late
2017-09-19 05:33:52,960 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:34:43,036 INFO     repeater     task nb300_lastensq starting 2.0 seconds late
2017-09-19 05:34:45,775 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:34:45,776 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:34:31
2017-09-19 05:36:15,883 INFO     repeater     task nb300_10pings starting 4.1 seconds late
2017-09-19 05:36:16,068 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:36:41,104 INFO     repeater     task nb300_lastensq starting 0.1 seconds late
2017-09-19 05:36:43,919 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:36:43,919 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:36:31
2017-09-19 05:37:33,976 INFO     repeater     task ashtech_hcorrstats starting 2.2 seconds late
2017-09-19 05:37:35,018 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:38:35,096 INFO     repeater     task nb300_10pings starting 3.3 seconds late
2017-09-19 05:38:35,283 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:38:45,298 INFO     repeater     task nb300_lastensq starting 4.3 seconds late
2017-09-19 05:38:48,064 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:38:48,065 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:38:31
2017-09-19 05:40:43,231 INFO     repeater     task nb300_lastensq starting 2.2 seconds late
2017-09-19 05:40:45,979 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:40:45,980 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:40:31
2017-09-19 05:40:55,994 INFO     repeater     task nb300_10pings starting 4.2 seconds late
2017-09-19 05:40:56,182 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:42:36,326 INFO     repeater     task ashtech_hcorrstats starting 4.5 seconds late
2017-09-19 05:42:37,372 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2017-09-19 05:42:42,374 INFO     repeater     task nb300_lastensq starting 1.4 seconds late
2017-09-19 05:42:45,143 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:42:45,144 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:42:31
2017-09-19 05:43:15,192 INFO     repeater     task nb300_10pings starting 3.4 seconds late
2017-09-19 05:43:15,374 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:44:45,505 INFO     repeater     task nb300_lastensq starting 4.5 seconds late
2017-09-19 05:44:48,240 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2017-09-19 05:44:48,240 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:44:31
2017-09-19 05:45:33,300 INFO     repeater     task nb300_10pings starting 1.5 seconds late
2017-09-19 05:45:33,485 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2017-09-19 05:46:43,583 INFO     repeater     task nb300_lastensq starting 2.6 seconds late
2017-09-19 05:46:46,354 INFO     repeater     task nb300_lastensq ended after 2.8 seconds
2017-09-19 05:46:46,354 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1723b/proc/nb300/load/wrote_ens_time: 2017/09/19 05:46:31
2017-09-19 05:47:36,428 INFO     repeater     task nb300_quick starting 4.6 seconds late
2017-09-19 05:47:44,106 INFO     repeater     task nb300_quick ended after 7.7 seconds
2017-09-19 05:47:44,107 INFO     repeater     task nb300_3dayplots starting 12.3 seconds late
2017-09-19 05:47:52,002 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)

2017-09-19 05:47:52,003 INFO     repeater     task nb300_3dayplots ended after 7.9 seconds
2017-09-19 05:47:52,003 INFO     repeater     task nb300_tsstats starting 20.2 seconds late
2017-09-19 05:47:53,164 INFO     repeater     task nb300_tsstats ended after 1.2 seconds