2018-07-24 07:17:36,099 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 07:17:36,099 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 07:17:36,100 INFO     repeater     task nb300_tsstats starting 9.1 seconds late
2018-07-24 07:17:36,999 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 07:17:37,000 INFO     repeater     task nb300_beamplots starting 10.0 seconds late
2018-07-24 07:17:38,919 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 07:17:38,920 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 07:17:38,920 INFO     repeater     task nb300_beamplots starting 11.9 seconds late
2018-07-24 07:17:40,940 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 07:17:40,940 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 07:17:40,940 INFO     repeater     task ashtech_hcorrstats starting 14.0 seconds late
2018-07-24 07:17:41,900 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 07:18:36,979 INFO     repeater     task nb300_lastensq starting 1.0 seconds late
2018-07-24 07:18:39,586 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:18:39,587 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:18:39,587 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:18:26
2018-07-24 07:18:49,606 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2018-07-24 07:18:49,808 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:20:39,931 INFO     repeater     task nb300_lastensq starting 3.9 seconds late
2018-07-24 07:20:42,615 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:20:42,615 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 07:20:42,615 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:20:26
2018-07-24 07:21:07,653 INFO     repeater     task nb300_10pings starting 0.7 seconds late
2018-07-24 07:21:07,837 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:22:27,945 INFO     repeater     task ashtech_hcorrstats starting 1.0 seconds late
2018-07-24 07:22:28,891 INFO     repeater     task ashtech_hcorrstats ended after 0.9 seconds
2018-07-24 07:22:38,904 INFO     repeater     task nb300_lastensq starting 2.9 seconds late
2018-07-24 07:22:41,548 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:22:41,548 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:22:41,548 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:22:26
2018-07-24 07:23:31,623 INFO     repeater     task nb300_10pings starting 4.6 seconds late
2018-07-24 07:23:31,805 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:24:36,890 INFO     repeater     task nb300_lastensq starting 0.9 seconds late
2018-07-24 07:24:39,513 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:24:39,514 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:24:26
2018-07-24 07:25:49,614 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2018-07-24 07:25:49,797 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:26:39,869 INFO     repeater     task nb300_lastensq starting 3.9 seconds late
2018-07-24 07:26:42,534 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:26:42,535 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 07:26:42,535 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:26:26
2018-07-24 07:27:27,603 INFO     repeater     task ashtech_hcorrstats starting 0.6 seconds late
2018-07-24 07:27:28,547 INFO     repeater     task ashtech_hcorrstats ended after 0.9 seconds
2018-07-24 07:28:08,603 INFO     repeater     task nb300_10pings starting 1.6 seconds late
2018-07-24 07:28:08,782 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:28:38,828 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2018-07-24 07:28:41,467 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:28:41,468 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:28:26
2018-07-24 07:30:31,621 INFO     repeater     task nb300_10pings starting 4.6 seconds late
2018-07-24 07:30:31,807 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:30:36,814 INFO     repeater     task nb300_lastensq starting 0.8 seconds late
2018-07-24 07:30:39,460 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:30:39,461 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:30:39,461 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:30:26
2018-07-24 07:32:29,626 INFO     repeater     task nb300_quick starting 2.6 seconds late
2018-07-24 07:32:32,083 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 07:32:32,084 INFO     repeater     task nb300_quick ended after 2.5 seconds
2018-07-24 07:32:32,084 INFO     repeater     task nb300_3dayplots starting 5.1 seconds late
2018-07-24 07:32:35,180 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 07:32:35,180 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 07:32:35,181 INFO     repeater     task nb300_tsstats starting 8.2 seconds late
2018-07-24 07:32:36,081 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 07:32:36,082 INFO     repeater     task nb300_beamplots starting 9.1 seconds late
2018-07-24 07:32:37,976 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 07:32:37,976 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 07:32:37,977 INFO     repeater     task nb300_beamplots starting 11.0 seconds late
2018-07-24 07:32:39,935 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 07:32:39,935 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 07:32:39,936 INFO     repeater     task ashtech_hcorrstats starting 13.0 seconds late
2018-07-24 07:32:40,876 INFO     repeater     task ashtech_hcorrstats ended after 0.9 seconds
2018-07-24 07:32:45,881 INFO     repeater     task nb300_lastensq starting 9.9 seconds late
2018-07-24 07:32:48,504 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:32:48,504 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:32:48,504 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:32:26
2018-07-24 07:32:48,505 INFO     repeater     task nb300_10pings starting 1.5 seconds late
2018-07-24 07:32:48,681 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:34:38,826 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2018-07-24 07:34:41,422 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:34:41,422 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:34:41,422 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:34:26
2018-07-24 07:35:11,461 INFO     repeater     task nb300_10pings starting 4.5 seconds late
2018-07-24 07:35:11,642 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:36:36,760 INFO     repeater     task nb300_lastensq starting 0.8 seconds late
2018-07-24 07:36:39,393 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:36:39,394 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:36:39,394 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:36:26
2018-07-24 07:37:29,451 INFO     repeater     task nb300_10pings starting 2.5 seconds late
2018-07-24 07:37:29,647 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:37:29,648 INFO     repeater     task ashtech_hcorrstats starting 2.7 seconds late
2018-07-24 07:37:30,603 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 07:38:40,691 INFO     repeater     task nb300_lastensq starting 4.7 seconds late
2018-07-24 07:38:43,313 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:38:43,313 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:38:43,313 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:38:26
2018-07-24 07:39:48,407 INFO     repeater     task nb300_10pings starting 1.4 seconds late
2018-07-24 07:39:48,589 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:40:38,664 INFO     repeater     task nb300_lastensq starting 2.7 seconds late
2018-07-24 07:40:41,315 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:40:41,316 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 07:40:41,316 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:40:26
2018-07-24 07:42:11,442 INFO     repeater     task nb300_10pings starting 4.5 seconds late
2018-07-24 07:42:11,626 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:42:31,655 INFO     repeater     task ashtech_hcorrstats starting 4.7 seconds late
2018-07-24 07:42:32,626 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 07:42:37,632 INFO     repeater     task nb300_lastensq starting 1.6 seconds late
2018-07-24 07:42:40,238 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:42:40,238 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:42:40,238 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:42:26
2018-07-24 07:44:30,398 INFO     repeater     task nb300_10pings starting 3.4 seconds late
2018-07-24 07:44:30,579 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:44:40,593 INFO     repeater     task nb300_lastensq starting 4.6 seconds late
2018-07-24 07:44:43,252 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:44:43,252 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 07:44:43,252 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:44:26
2018-07-24 07:46:38,406 INFO     repeater     task nb300_lastensq starting 2.4 seconds late
2018-07-24 07:46:40,976 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:46:40,976 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:46:40,977 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:46:26
2018-07-24 07:46:50,991 INFO     repeater     task nb300_10pings starting 4.0 seconds late
2018-07-24 07:46:51,176 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:47:31,232 INFO     repeater     task nb300_quick starting 4.3 seconds late
2018-07-24 07:47:33,654 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 07:47:33,654 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 07:47:33,654 INFO     repeater     task nb300_3dayplots starting 6.7 seconds late
2018-07-24 07:47:36,727 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 07:47:36,727 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 07:47:36,727 INFO     repeater     task nb300_tsstats starting 9.7 seconds late
2018-07-24 07:47:37,637 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 07:47:37,637 INFO     repeater     task nb300_beamplots starting 10.7 seconds late
2018-07-24 07:47:39,537 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 07:47:39,537 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 07:47:39,538 INFO     repeater     task nb300_beamplots starting 12.6 seconds late
2018-07-24 07:47:41,512 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 07:47:41,512 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 07:47:41,512 INFO     repeater     task ashtech_hcorrstats starting 14.5 seconds late
2018-07-24 07:47:42,531 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 07:48:37,611 INFO     repeater     task nb300_lastensq starting 1.6 seconds late
2018-07-24 07:48:40,226 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:48:40,226 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:48:40,226 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:48:26
2018-07-24 07:49:10,270 INFO     repeater     task nb300_10pings starting 3.3 seconds late
2018-07-24 07:49:10,451 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:50:40,576 INFO     repeater     task nb300_lastensq starting 4.6 seconds late
2018-07-24 07:50:43,269 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:50:43,270 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 07:50:43,270 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:50:26
2018-07-24 07:51:28,328 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2018-07-24 07:51:28,511 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:52:28,591 INFO     repeater     task ashtech_hcorrstats starting 1.6 seconds late
2018-07-24 07:52:29,555 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 07:52:39,568 INFO     repeater     task nb300_lastensq starting 3.6 seconds late
2018-07-24 07:52:42,154 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:52:42,154 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:52:42,154 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:52:26
2018-07-24 07:53:47,253 INFO     repeater     task nb300_10pings starting 0.3 seconds late
2018-07-24 07:53:47,436 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:54:37,510 INFO     repeater     task nb300_lastensq starting 1.5 seconds late
2018-07-24 07:54:40,128 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:54:40,129 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:54:40,129 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:54:26
2018-07-24 07:56:10,255 INFO     repeater     task nb300_10pings starting 3.3 seconds late
2018-07-24 07:56:10,437 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:56:40,476 INFO     repeater     task nb300_lastensq starting 4.5 seconds late
2018-07-24 07:56:43,065 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:56:43,066 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 07:56:43,066 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:56:26
2018-07-24 07:57:28,136 INFO     repeater     task ashtech_hcorrstats starting 1.2 seconds late
2018-07-24 07:57:29,102 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 07:58:29,184 INFO     repeater     task nb300_10pings starting 2.2 seconds late
2018-07-24 07:58:29,363 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 07:58:39,375 INFO     repeater     task nb300_lastensq starting 3.4 seconds late
2018-07-24 07:58:42,036 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 07:58:42,037 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 07:58:42,037 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 07:58:26
2018-07-24 08:00:37,205 INFO     repeater     task nb300_lastensq starting 1.2 seconds late
2018-07-24 08:00:39,821 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:00:39,822 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:00:39,822 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:00:26
2018-07-24 08:00:49,837 INFO     repeater     task nb300_10pings starting 2.9 seconds late
2018-07-24 08:00:50,019 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:02:30,165 INFO     repeater     task nb300_quick starting 3.2 seconds late
2018-07-24 08:02:32,576 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 08:02:32,576 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 08:02:32,577 INFO     repeater     task nb300_3dayplots starting 5.6 seconds late
2018-07-24 08:02:35,672 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 08:02:35,672 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 08:02:35,672 INFO     repeater     task nb300_tsstats starting 8.7 seconds late
2018-07-24 08:02:36,579 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 08:02:36,580 INFO     repeater     task nb300_beamplots starting 9.6 seconds late
2018-07-24 08:02:38,500 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:02:38,500 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 08:02:38,500 INFO     repeater     task nb300_beamplots starting 11.5 seconds late
2018-07-24 08:02:40,422 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:02:40,422 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 08:02:40,423 INFO     repeater     task ashtech_hcorrstats starting 13.4 seconds late
2018-07-24 08:02:41,435 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:02:46,441 INFO     repeater     task nb300_lastensq starting 10.4 seconds late
2018-07-24 08:02:49,040 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:02:49,040 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:02:49,041 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:02:26
2018-07-24 08:03:09,068 INFO     repeater     task nb300_10pings starting 2.1 seconds late
2018-07-24 08:03:09,250 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:04:39,370 INFO     repeater     task nb300_lastensq starting 3.4 seconds late
2018-07-24 08:04:42,022 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:04:42,022 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:04:42,023 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:04:26
2018-07-24 08:05:27,086 INFO     repeater     task nb300_10pings starting 0.1 seconds late
2018-07-24 08:05:27,267 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:06:37,367 INFO     repeater     task nb300_lastensq starting 1.4 seconds late
2018-07-24 08:06:39,938 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:06:39,938 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:06:39,939 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:06:26
2018-07-24 08:07:30,014 INFO     repeater     task ashtech_hcorrstats starting 3.0 seconds late
2018-07-24 08:07:30,976 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:07:51,004 INFO     repeater     task nb300_10pings starting 4.0 seconds late
2018-07-24 08:07:51,191 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:08:36,244 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2018-07-24 08:08:38,896 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:08:38,896 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:08:38,897 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:08:26
2018-07-24 08:10:09,024 INFO     repeater     task nb300_10pings starting 2.0 seconds late
2018-07-24 08:10:09,210 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:10:39,255 INFO     repeater     task nb300_lastensq starting 3.3 seconds late
2018-07-24 08:10:41,851 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:10:41,851 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:10:41,851 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:10:26
2018-07-24 08:12:27,009 INFO     repeater     task nb300_10pings starting 0.0 seconds late
2018-07-24 08:12:27,192 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:12:27,193 INFO     repeater     task ashtech_hcorrstats starting 0.2 seconds late
2018-07-24 08:12:28,161 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:12:38,172 INFO     repeater     task nb300_lastensq starting 2.2 seconds late
2018-07-24 08:12:40,814 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:12:40,814 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:12:40,815 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:12:26
2018-07-24 08:14:40,983 INFO     repeater     task nb300_lastensq starting 5.0 seconds late
2018-07-24 08:14:43,582 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:14:43,583 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:14:43,583 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:14:26
2018-07-24 08:14:48,590 INFO     repeater     task nb300_10pings starting 1.6 seconds late
2018-07-24 08:14:48,777 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:16:38,929 INFO     repeater     task nb300_lastensq starting 2.9 seconds late
2018-07-24 08:16:41,569 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:16:41,569 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:16:41,569 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:16:26
2018-07-24 08:17:11,615 INFO     repeater     task nb300_10pings starting 4.6 seconds late
2018-07-24 08:17:11,797 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:17:31,825 INFO     repeater     task nb300_quick starting 4.8 seconds late
2018-07-24 08:17:34,195 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 08:17:34,195 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 08:17:34,195 INFO     repeater     task nb300_3dayplots starting 7.2 seconds late
2018-07-24 08:17:37,301 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 08:17:37,301 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 08:17:37,301 INFO     repeater     task nb300_tsstats starting 10.3 seconds late
2018-07-24 08:17:38,158 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 08:17:38,158 INFO     repeater     task nb300_beamplots starting 11.2 seconds late
2018-07-24 08:17:40,113 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:17:40,114 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 08:17:40,114 INFO     repeater     task nb300_beamplots starting 13.1 seconds late
2018-07-24 08:17:42,124 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:17:42,124 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 08:17:42,125 INFO     repeater     task ashtech_hcorrstats starting 15.1 seconds late
2018-07-24 08:17:43,092 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:18:38,162 INFO     repeater     task nb300_lastensq starting 2.2 seconds late
2018-07-24 08:18:40,811 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:18:40,811 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:18:40,811 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:18:26
2018-07-24 08:19:30,878 INFO     repeater     task nb300_10pings starting 3.9 seconds late
2018-07-24 08:19:31,063 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:20:36,157 INFO     repeater     task nb300_lastensq starting 0.2 seconds late
2018-07-24 08:20:38,740 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:20:38,740 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:20:38,741 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:20:26
2018-07-24 08:21:48,845 INFO     repeater     task nb300_10pings starting 1.9 seconds late
2018-07-24 08:21:49,030 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:22:29,085 INFO     repeater     task ashtech_hcorrstats starting 2.1 seconds late
2018-07-24 08:22:30,055 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:22:40,069 INFO     repeater     task nb300_lastensq starting 4.1 seconds late
2018-07-24 08:22:42,699 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:22:42,699 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:22:42,699 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:22:26
2018-07-24 08:24:07,818 INFO     repeater     task nb300_10pings starting 0.8 seconds late
2018-07-24 08:24:08,001 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:24:38,046 INFO     repeater     task nb300_lastensq starting 2.0 seconds late
2018-07-24 08:24:40,639 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:24:40,639 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:24:40,639 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:24:26
2018-07-24 08:26:30,794 INFO     repeater     task nb300_10pings starting 3.8 seconds late
2018-07-24 08:26:30,975 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:26:40,989 INFO     repeater     task nb300_lastensq starting 5.0 seconds late
2018-07-24 08:26:43,647 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:26:43,648 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:26:43,648 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:26:26
2018-07-24 08:27:28,711 INFO     repeater     task ashtech_hcorrstats starting 1.7 seconds late
2018-07-24 08:27:29,684 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:28:39,787 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2018-07-24 08:28:42,379 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:28:42,380 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:28:42,380 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:28:26
2018-07-24 08:28:47,382 INFO     repeater     task nb300_10pings starting 0.4 seconds late
2018-07-24 08:28:47,564 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:30:37,713 INFO     repeater     task nb300_lastensq starting 1.7 seconds late
2018-07-24 08:30:40,372 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:30:40,372 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:30:40,373 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:30:26
2018-07-24 08:31:10,408 INFO     repeater     task nb300_10pings starting 3.4 seconds late
2018-07-24 08:31:10,591 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:32:30,712 INFO     repeater     task nb300_quick starting 3.7 seconds late
2018-07-24 08:32:33,107 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 08:32:33,107 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 08:32:33,107 INFO     repeater     task nb300_3dayplots starting 6.1 seconds late
2018-07-24 08:32:36,213 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 08:32:36,213 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 08:32:36,213 INFO     repeater     task nb300_tsstats starting 9.2 seconds late
2018-07-24 08:32:37,083 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 08:32:37,084 INFO     repeater     task nb300_beamplots starting 10.1 seconds late
2018-07-24 08:32:38,998 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:32:38,998 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 08:32:38,999 INFO     repeater     task nb300_beamplots starting 12.0 seconds late
2018-07-24 08:32:40,969 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:32:40,969 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 08:32:40,970 INFO     repeater     task ashtech_hcorrstats starting 14.0 seconds late
2018-07-24 08:32:41,942 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:32:46,947 INFO     repeater     task nb300_lastensq starting 10.9 seconds late
2018-07-24 08:32:49,597 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:32:49,597 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:32:49,597 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:32:26
2018-07-24 08:33:29,655 INFO     repeater     task nb300_10pings starting 2.7 seconds late
2018-07-24 08:33:29,842 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:34:39,943 INFO     repeater     task nb300_lastensq starting 3.9 seconds late
2018-07-24 08:34:42,539 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:34:42,540 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:34:42,540 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:34:26
2018-07-24 08:35:47,637 INFO     repeater     task nb300_10pings starting 0.7 seconds late
2018-07-24 08:35:47,818 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:36:37,881 INFO     repeater     task nb300_lastensq starting 1.9 seconds late
2018-07-24 08:36:40,548 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:36:40,549 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:36:40,549 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:36:26
2018-07-24 08:37:30,622 INFO     repeater     task ashtech_hcorrstats starting 3.6 seconds late
2018-07-24 08:37:31,597 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:38:11,657 INFO     repeater     task nb300_10pings starting 4.7 seconds late
2018-07-24 08:38:11,841 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:38:36,874 INFO     repeater     task nb300_lastensq starting 0.9 seconds late
2018-07-24 08:38:39,466 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:38:39,466 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:38:39,466 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:38:26
2018-07-24 08:40:29,624 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2018-07-24 08:40:29,806 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:40:39,815 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2018-07-24 08:40:42,487 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:40:42,487 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:40:42,488 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:40:26
2018-07-24 08:42:27,643 INFO     repeater     task ashtech_hcorrstats starting 0.7 seconds late
2018-07-24 08:42:28,618 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:42:38,631 INFO     repeater     task nb300_lastensq starting 2.6 seconds late
2018-07-24 08:42:41,250 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:42:41,250 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:42:41,250 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:42:26
2018-07-24 08:42:51,263 INFO     repeater     task nb300_10pings starting 4.3 seconds late
2018-07-24 08:42:51,473 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:44:36,614 INFO     repeater     task nb300_lastensq starting 0.6 seconds late
2018-07-24 08:44:39,272 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:44:39,273 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:44:39,273 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:44:26
2018-07-24 08:45:09,314 INFO     repeater     task nb300_10pings starting 2.3 seconds late
2018-07-24 08:45:09,498 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:46:39,623 INFO     repeater     task nb300_lastensq starting 3.6 seconds late
2018-07-24 08:46:42,229 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:46:42,229 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:46:42,229 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:46:26
2018-07-24 08:47:27,282 INFO     repeater     task nb300_10pings starting 0.3 seconds late
2018-07-24 08:47:27,466 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:47:27,467 INFO     repeater     task nb300_quick starting 0.5 seconds late
2018-07-24 08:47:29,913 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 08:47:29,913 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 08:47:29,914 INFO     repeater     task nb300_3dayplots starting 2.9 seconds late
2018-07-24 08:47:33,038 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 08:47:33,038 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 08:47:33,038 INFO     repeater     task nb300_tsstats starting 6.1 seconds late
2018-07-24 08:47:33,910 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 08:47:33,911 INFO     repeater     task nb300_beamplots starting 6.9 seconds late
2018-07-24 08:47:35,827 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:47:35,827 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 08:47:35,828 INFO     repeater     task nb300_beamplots starting 8.8 seconds late
2018-07-24 08:47:37,777 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 08:47:37,777 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 08:47:37,778 INFO     repeater     task ashtech_hcorrstats starting 10.8 seconds late
2018-07-24 08:47:38,777 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:48:38,849 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2018-07-24 08:48:41,449 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:48:41,450 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:48:41,450 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:48:26
2018-07-24 08:49:51,549 INFO     repeater     task nb300_10pings starting 4.6 seconds late
2018-07-24 08:49:51,730 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:50:36,790 INFO     repeater     task nb300_lastensq starting 0.8 seconds late
2018-07-24 08:50:39,446 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:50:39,446 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:50:39,446 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:50:26
2018-07-24 08:52:09,574 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2018-07-24 08:52:09,753 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:52:29,786 INFO     repeater     task ashtech_hcorrstats starting 2.8 seconds late
2018-07-24 08:52:30,781 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:52:40,794 INFO     repeater     task nb300_lastensq starting 4.8 seconds late
2018-07-24 08:52:43,380 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:52:43,381 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:52:43,381 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:52:26
2018-07-24 08:54:28,525 INFO     repeater     task nb300_10pings starting 1.5 seconds late
2018-07-24 08:54:28,705 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:54:38,720 INFO     repeater     task nb300_lastensq starting 2.7 seconds late
2018-07-24 08:54:41,377 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 08:54:41,377 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:54:41,378 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:54:26
2018-07-24 08:56:36,548 INFO     repeater     task nb300_lastensq starting 0.5 seconds late
2018-07-24 08:56:39,184 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 08:56:39,184 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:56:26
2018-07-24 08:56:49,198 INFO     repeater     task nb300_10pings starting 2.2 seconds late
2018-07-24 08:56:49,379 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 08:57:29,437 INFO     repeater     task ashtech_hcorrstats starting 2.5 seconds late
2018-07-24 08:57:30,419 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 08:58:40,499 INFO     repeater     task nb300_lastensq starting 4.5 seconds late
2018-07-24 08:58:43,185 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 08:58:43,186 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 08:58:26
2018-07-24 08:59:08,223 INFO     repeater     task nb300_10pings starting 1.2 seconds late
2018-07-24 08:59:08,406 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:00:38,539 INFO     repeater     task nb300_lastensq starting 2.5 seconds late
2018-07-24 09:00:41,164 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:00:41,165 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:00:41,165 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:00:26
2018-07-24 09:01:31,265 INFO     repeater     task nb300_10pings starting 4.3 seconds late
2018-07-24 09:01:31,472 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:02:31,539 INFO     repeater     task nb300_quick starting 4.6 seconds late
2018-07-24 09:02:33,994 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 09:02:33,994 INFO     repeater     task nb300_quick ended after 2.5 seconds
2018-07-24 09:02:33,995 INFO     repeater     task nb300_3dayplots starting 7.0 seconds late
2018-07-24 09:02:37,114 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 09:02:37,114 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 09:02:37,115 INFO     repeater     task nb300_tsstats starting 10.1 seconds late
2018-07-24 09:02:37,982 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 09:02:37,983 INFO     repeater     task nb300_beamplots starting 11.0 seconds late
2018-07-24 09:02:39,838 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:02:39,838 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 09:02:39,838 INFO     repeater     task nb300_beamplots starting 12.9 seconds late
2018-07-24 09:02:41,864 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:02:41,864 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 09:02:41,864 INFO     repeater     task ashtech_hcorrstats starting 14.9 seconds late
2018-07-24 09:02:42,835 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:02:47,841 INFO     repeater     task nb300_lastensq starting 11.8 seconds late
2018-07-24 09:02:50,477 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:02:50,477 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:02:26
2018-07-24 09:03:50,564 INFO     repeater     task nb300_10pings starting 3.6 seconds late
2018-07-24 09:03:50,749 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:04:40,825 INFO     repeater     task nb300_lastensq starting 4.8 seconds late
2018-07-24 09:04:43,484 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:04:43,485 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:04:43,485 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:04:26
2018-07-24 09:06:08,609 INFO     repeater     task nb300_10pings starting 1.6 seconds late
2018-07-24 09:06:08,793 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:06:38,835 INFO     repeater     task nb300_lastensq starting 2.8 seconds late
2018-07-24 09:06:41,441 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:06:41,442 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:06:41,442 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:06:26
2018-07-24 09:07:31,519 INFO     repeater     task ashtech_hcorrstats starting 4.5 seconds late
2018-07-24 09:07:32,493 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:08:27,558 INFO     repeater     task nb300_10pings starting 0.6 seconds late
2018-07-24 09:08:27,742 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:08:37,756 INFO     repeater     task nb300_lastensq starting 1.8 seconds late
2018-07-24 09:08:40,451 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:08:40,451 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:08:26
2018-07-24 09:10:40,624 INFO     repeater     task nb300_lastensq starting 4.6 seconds late
2018-07-24 09:10:43,254 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:10:43,254 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:10:43,255 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:10:26
2018-07-24 09:10:48,261 INFO     repeater     task nb300_10pings starting 1.3 seconds late
2018-07-24 09:10:48,442 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:12:28,580 INFO     repeater     task ashtech_hcorrstats starting 1.6 seconds late
2018-07-24 09:12:29,598 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:12:39,610 INFO     repeater     task nb300_lastensq starting 3.6 seconds late
2018-07-24 09:12:42,278 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:12:42,278 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:12:26
2018-07-24 09:13:07,315 INFO     repeater     task nb300_10pings starting 0.3 seconds late
2018-07-24 09:13:07,496 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:14:37,614 INFO     repeater     task nb300_lastensq starting 1.6 seconds late
2018-07-24 09:14:40,245 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:14:40,246 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:14:40,246 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:14:26
2018-07-24 09:15:30,314 INFO     repeater     task nb300_10pings starting 3.3 seconds late
2018-07-24 09:15:30,496 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:16:40,598 INFO     repeater     task nb300_lastensq starting 4.6 seconds late
2018-07-24 09:16:43,248 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:16:43,248 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:16:26
2018-07-24 09:17:28,301 INFO     repeater     task nb300_quick starting 1.3 seconds late
2018-07-24 09:17:30,744 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 09:17:30,745 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 09:17:30,745 INFO     repeater     task nb300_3dayplots starting 3.8 seconds late
2018-07-24 09:17:33,859 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 09:17:33,859 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 09:17:33,859 INFO     repeater     task nb300_tsstats starting 6.9 seconds late
2018-07-24 09:17:34,731 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 09:17:34,731 INFO     repeater     task nb300_beamplots starting 7.8 seconds late
2018-07-24 09:17:36,693 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:17:36,693 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 09:17:36,694 INFO     repeater     task nb300_beamplots starting 9.7 seconds late
2018-07-24 09:17:38,606 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:17:38,607 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 09:17:38,607 INFO     repeater     task ashtech_hcorrstats starting 11.6 seconds late
2018-07-24 09:17:39,569 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:17:49,583 INFO     repeater     task nb300_10pings starting 2.6 seconds late
2018-07-24 09:17:49,764 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:18:39,839 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2018-07-24 09:18:42,490 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:18:42,490 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:18:42,490 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:18:26
2018-07-24 09:20:07,614 INFO     repeater     task nb300_10pings starting 0.6 seconds late
2018-07-24 09:20:07,798 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:20:37,842 INFO     repeater     task nb300_lastensq starting 1.8 seconds late
2018-07-24 09:20:40,499 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:20:40,499 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:20:26
2018-07-24 09:22:30,658 INFO     repeater     task nb300_10pings starting 3.7 seconds late
2018-07-24 09:22:30,844 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:22:30,846 INFO     repeater     task ashtech_hcorrstats starting 3.9 seconds late
2018-07-24 09:22:31,826 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:22:36,831 INFO     repeater     task nb300_lastensq starting 0.8 seconds late
2018-07-24 09:22:39,476 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:22:39,477 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:22:39,477 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:22:26
2018-07-24 09:24:39,655 INFO     repeater     task nb300_lastensq starting 3.7 seconds late
2018-07-24 09:24:42,313 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:24:42,313 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:24:26
2018-07-24 09:24:47,320 INFO     repeater     task nb300_10pings starting 0.3 seconds late
2018-07-24 09:24:47,561 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:26:37,709 INFO     repeater     task nb300_lastensq starting 1.7 seconds late
2018-07-24 09:26:40,385 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:26:40,386 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:26:40,386 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:26:26
2018-07-24 09:27:10,426 INFO     repeater     task nb300_10pings starting 3.4 seconds late
2018-07-24 09:27:10,607 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:27:30,639 INFO     repeater     task ashtech_hcorrstats starting 3.7 seconds late
2018-07-24 09:27:31,611 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:28:36,701 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2018-07-24 09:28:39,349 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:28:39,349 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:28:39,350 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:28:26
2018-07-24 09:29:29,419 INFO     repeater     task nb300_10pings starting 2.4 seconds late
2018-07-24 09:29:29,602 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:30:39,702 INFO     repeater     task nb300_lastensq starting 3.7 seconds late
2018-07-24 09:30:42,329 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:30:42,329 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:30:26
2018-07-24 09:31:47,423 INFO     repeater     task nb300_10pings starting 0.4 seconds late
2018-07-24 09:31:47,605 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:32:27,664 INFO     repeater     task nb300_quick starting 0.7 seconds late
2018-07-24 09:32:30,148 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 09:32:30,149 INFO     repeater     task nb300_quick ended after 2.5 seconds
2018-07-24 09:32:30,149 INFO     repeater     task nb300_3dayplots starting 3.2 seconds late
2018-07-24 09:32:33,243 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 09:32:33,243 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 09:32:33,244 INFO     repeater     task nb300_tsstats starting 6.3 seconds late
2018-07-24 09:32:34,111 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 09:32:34,112 INFO     repeater     task nb300_beamplots starting 7.1 seconds late
2018-07-24 09:32:36,048 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:32:36,049 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 09:32:36,049 INFO     repeater     task nb300_beamplots starting 9.1 seconds late
2018-07-24 09:32:37,955 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:32:37,956 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 09:32:37,956 INFO     repeater     task ashtech_hcorrstats starting 11.0 seconds late
2018-07-24 09:32:38,921 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:32:43,927 INFO     repeater     task nb300_lastensq starting 7.9 seconds late
2018-07-24 09:32:46,647 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:32:46,647 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:32:26
2018-07-24 09:34:11,766 INFO     repeater     task nb300_10pings starting 4.8 seconds late
2018-07-24 09:34:11,950 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:34:36,987 INFO     repeater     task nb300_lastensq starting 1.0 seconds late
2018-07-24 09:34:39,576 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:34:39,576 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:34:39,576 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:34:26
2018-07-24 09:36:29,727 INFO     repeater     task nb300_10pings starting 2.7 seconds late
2018-07-24 09:36:29,917 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:36:39,931 INFO     repeater     task nb300_lastensq starting 3.9 seconds late
2018-07-24 09:36:42,593 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:36:42,594 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:36:42,594 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:36:26
2018-07-24 09:37:27,665 INFO     repeater     task ashtech_hcorrstats starting 0.7 seconds late
2018-07-24 09:37:28,623 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:38:38,724 INFO     repeater     task nb300_lastensq starting 2.7 seconds late
2018-07-24 09:38:41,392 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:38:41,392 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:38:41,392 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:38:26
2018-07-24 09:38:51,403 INFO     repeater     task nb300_10pings starting 4.4 seconds late
2018-07-24 09:38:51,596 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:40:36,729 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2018-07-24 09:40:39,342 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:40:39,342 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:40:39,342 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:40:26
2018-07-24 09:41:09,387 INFO     repeater     task nb300_10pings starting 2.4 seconds late
2018-07-24 09:41:09,573 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:42:29,696 INFO     repeater     task ashtech_hcorrstats starting 2.7 seconds late
2018-07-24 09:42:30,669 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:42:40,682 INFO     repeater     task nb300_lastensq starting 4.7 seconds late
2018-07-24 09:42:43,363 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:42:43,363 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:42:43,363 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:42:26
2018-07-24 09:43:28,424 INFO     repeater     task nb300_10pings starting 1.4 seconds late
2018-07-24 09:43:28,606 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:44:38,707 INFO     repeater     task nb300_lastensq starting 2.7 seconds late
2018-07-24 09:44:41,323 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:44:41,324 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:44:41,324 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:44:26
2018-07-24 09:45:51,424 INFO     repeater     task nb300_10pings starting 4.4 seconds late
2018-07-24 09:45:51,613 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:46:36,675 INFO     repeater     task nb300_lastensq starting 0.7 seconds late
2018-07-24 09:46:39,367 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:46:39,367 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:46:26
2018-07-24 09:47:29,433 INFO     repeater     task nb300_quick starting 2.5 seconds late
2018-07-24 09:47:31,839 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 09:47:31,840 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 09:47:31,840 INFO     repeater     task nb300_3dayplots starting 4.9 seconds late
2018-07-24 09:47:35,039 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 09:47:35,039 INFO     repeater     task nb300_3dayplots ended after 3.2 seconds
2018-07-24 09:47:35,039 INFO     repeater     task nb300_tsstats starting 8.1 seconds late
2018-07-24 09:47:35,927 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 09:47:35,928 INFO     repeater     task nb300_beamplots starting 8.9 seconds late
2018-07-24 09:47:37,794 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:47:37,794 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 09:47:37,795 INFO     repeater     task nb300_beamplots starting 10.8 seconds late
2018-07-24 09:47:39,818 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 09:47:39,819 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 09:47:39,819 INFO     repeater     task ashtech_hcorrstats starting 12.8 seconds late
2018-07-24 09:47:40,776 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:48:10,816 INFO     repeater     task nb300_10pings starting 3.8 seconds late
2018-07-24 09:48:10,999 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:48:36,036 INFO     repeater     task nb300_lastensq starting 0.0 seconds late
2018-07-24 09:48:38,666 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:48:38,666 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:48:38,666 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:48:26
2018-07-24 09:50:28,833 INFO     repeater     task nb300_10pings starting 1.9 seconds late
2018-07-24 09:50:29,017 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:50:39,032 INFO     repeater     task nb300_lastensq starting 3.0 seconds late
2018-07-24 09:50:41,747 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:50:41,748 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:50:26
2018-07-24 09:52:31,911 INFO     repeater     task ashtech_hcorrstats starting 4.9 seconds late
2018-07-24 09:52:32,885 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:52:37,891 INFO     repeater     task nb300_lastensq starting 1.9 seconds late
2018-07-24 09:52:40,595 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:52:40,595 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:52:26
2018-07-24 09:52:50,610 INFO     repeater     task nb300_10pings starting 3.6 seconds late
2018-07-24 09:52:50,793 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:54:40,956 INFO     repeater     task nb300_lastensq starting 5.0 seconds late
2018-07-24 09:54:43,559 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 09:54:43,559 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 09:54:43,560 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:54:26
2018-07-24 09:55:08,594 INFO     repeater     task nb300_10pings starting 1.6 seconds late
2018-07-24 09:55:08,780 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:56:38,912 INFO     repeater     task nb300_lastensq starting 2.9 seconds late
2018-07-24 09:56:41,619 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:56:41,620 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:56:26
2018-07-24 09:57:31,692 INFO     repeater     task nb300_10pings starting 4.7 seconds late
2018-07-24 09:57:31,873 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 09:57:31,875 INFO     repeater     task ashtech_hcorrstats starting 4.9 seconds late
2018-07-24 09:57:32,840 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 09:58:37,930 INFO     repeater     task nb300_lastensq starting 1.9 seconds late
2018-07-24 09:58:40,587 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 09:58:40,588 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 09:58:26
2018-07-24 09:59:50,690 INFO     repeater     task nb300_10pings starting 3.7 seconds late
2018-07-24 09:59:50,869 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:00:40,941 INFO     repeater     task nb300_lastensq starting 4.9 seconds late
2018-07-24 10:00:43,623 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 10:00:43,624 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 10:00:43,624 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:00:26
2018-07-24 10:02:08,745 INFO     repeater     task nb300_10pings starting 1.8 seconds late
2018-07-24 10:02:08,927 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:02:28,957 INFO     repeater     task nb300_quick starting 2.0 seconds late
2018-07-24 10:02:31,389 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 10:02:31,389 INFO     repeater     task nb300_quick ended after 2.4 seconds
2018-07-24 10:02:31,390 INFO     repeater     task nb300_3dayplots starting 4.4 seconds late
2018-07-24 10:02:34,543 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 10:02:34,544 INFO     repeater     task nb300_3dayplots ended after 3.2 seconds
2018-07-24 10:02:34,544 INFO     repeater     task nb300_tsstats starting 7.6 seconds late
2018-07-24 10:02:35,425 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 10:02:35,426 INFO     repeater     task nb300_beamplots starting 8.4 seconds late
2018-07-24 10:02:37,366 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 10:02:37,366 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 10:02:37,366 INFO     repeater     task nb300_beamplots starting 10.4 seconds late
2018-07-24 10:02:39,341 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 10:02:39,341 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 10:02:39,341 INFO     repeater     task ashtech_hcorrstats starting 12.4 seconds late
2018-07-24 10:02:40,303 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 10:02:45,309 INFO     repeater     task nb300_lastensq starting 9.3 seconds late
2018-07-24 10:02:47,947 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 10:02:47,947 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 10:02:47,947 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:02:26
2018-07-24 10:04:28,076 INFO     repeater     task nb300_10pings starting 1.1 seconds late
2018-07-24 10:04:28,261 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:04:38,275 INFO     repeater     task nb300_lastensq starting 2.3 seconds late
2018-07-24 10:04:40,873 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 10:04:40,873 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 10:04:40,873 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:04:26
2018-07-24 10:06:36,039 INFO     repeater     task nb300_lastensq starting 0.0 seconds late
2018-07-24 10:06:38,624 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 10:06:38,624 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:06:26
2018-07-24 10:06:48,635 INFO     repeater     task nb300_10pings starting 1.7 seconds late
2018-07-24 10:06:48,836 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:07:28,886 INFO     repeater     task ashtech_hcorrstats starting 1.9 seconds late
2018-07-24 10:07:29,836 INFO     repeater     task ashtech_hcorrstats ended after 0.9 seconds
2018-07-24 10:08:39,939 INFO     repeater     task nb300_lastensq starting 3.9 seconds late
2018-07-24 10:08:42,508 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 10:08:42,509 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:08:26
2018-07-24 10:09:07,546 INFO     repeater     task nb300_10pings starting 0.6 seconds late
2018-07-24 10:09:07,728 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:10:37,853 INFO     repeater     task nb300_lastensq starting 1.9 seconds late
2018-07-24 10:10:40,507 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 10:10:40,508 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:10:26
2018-07-24 10:11:30,579 INFO     repeater     task nb300_10pings starting 3.6 seconds late
2018-07-24 10:11:30,761 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:12:30,851 INFO     repeater     task ashtech_hcorrstats starting 3.9 seconds late
2018-07-24 10:12:31,814 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 10:12:36,820 INFO     repeater     task nb300_lastensq starting 0.8 seconds late
2018-07-24 10:12:39,421 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 10:12:39,421 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 10:12:39,422 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:12:26
2018-07-24 10:13:49,504 INFO     repeater     task nb300_10pings starting 2.5 seconds late
2018-07-24 10:13:49,688 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:14:39,755 INFO     repeater     task nb300_lastensq starting 3.8 seconds late
2018-07-24 10:14:42,410 INFO     repeater     task nb300_lastensq output:
/usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan.
  warnings.warn("Warning: converting a masked element to nan.")

2018-07-24 10:14:42,410 INFO     repeater     task nb300_lastensq ended after 2.7 seconds
2018-07-24 10:14:42,411 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:14:26
2018-07-24 10:16:07,540 INFO     repeater     task nb300_10pings starting 0.6 seconds late
2018-07-24 10:16:07,723 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:16:37,763 INFO     repeater     task nb300_lastensq starting 1.8 seconds late
2018-07-24 10:16:40,344 INFO     repeater     task nb300_lastensq ended after 2.6 seconds
2018-07-24 10:16:40,344 INFO     repeater     task nb300_lastensq: Time from /home/data/SP1821/proc/nb300/load/wrote_ens_time: 2018/07/24 10:16:26
2018-07-24 10:17:30,408 INFO     repeater     task nb300_quick starting 3.4 seconds late
2018-07-24 10:17:32,899 INFO     repeater     task nb300_quick output:
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)

2018-07-24 10:17:32,899 INFO     repeater     task nb300_quick ended after 2.5 seconds
2018-07-24 10:17:32,900 INFO     repeater     task nb300_3dayplots starting 5.9 seconds late
2018-07-24 10:17:36,012 INFO     repeater     task nb300_3dayplots output:
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered
  warnings.warn("All-NaN slice encountered", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered
  warnings.warn("All-NaN axis encountered", RuntimeWarning)
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked
zgrid failed
zgrid failed: returning all values masked

2018-07-24 10:17:36,012 INFO     repeater     task nb300_3dayplots ended after 3.1 seconds
2018-07-24 10:17:36,012 INFO     repeater     task nb300_tsstats starting 9.0 seconds late
2018-07-24 10:17:36,960 INFO     repeater     task nb300_tsstats ended after 0.9 seconds
2018-07-24 10:17:36,960 INFO     repeater     task nb300_beamplots starting 10.0 seconds late
2018-07-24 10:17:38,861 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 10:17:38,861 INFO     repeater     task nb300_beamplots ended after 1.9 seconds
2018-07-24 10:17:38,861 INFO     repeater     task nb300_beamplots starting 11.9 seconds late
2018-07-24 10:17:40,858 INFO     repeater     task nb300_beamplots output:
Starting procsetup
scriptpath is /usr/local/currents/bin

2018-07-24 10:17:40,859 INFO     repeater     task nb300_beamplots ended after 2.0 seconds
2018-07-24 10:17:40,859 INFO     repeater     task ashtech_hcorrstats starting 13.9 seconds late
2018-07-24 10:17:41,873 INFO     repeater     task ashtech_hcorrstats ended after 1.0 seconds
2018-07-24 10:18:31,941 INFO     repeater     task nb300_10pings starting 5.0 seconds late
2018-07-24 10:18:32,123 INFO     repeater     task nb300_10pings ended after 0.2 seconds
2018-07-24 10:18:37,130 INFO     repeater     task nb300_lastensq starting 1.1 seconds late