2017-02-19 10:29:55,612 INFO repeater task nb300_10pings starting 2.3 seconds late 2017-02-19 10:29:55,804 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:30:15,833 INFO repeater task nb300_quick starting 2.5 seconds late 2017-02-19 10:30:21,515 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 10:30:21,515 INFO repeater task nb300_3dayplots starting 8.2 seconds late 2017-02-19 10:30:27,118 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 10:30:27,119 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 10:30:27,119 INFO repeater task nb300_tsstats starting 13.8 seconds late 2017-02-19 10:30:28,156 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 10:30:28,156 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 10:30:28,157 INFO repeater task nb300_beamplots starting 14.9 seconds late 2017-02-19 10:30:29,968 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 10:30:29,968 INFO repeater task nb300_beamplots ended after 1.8 seconds 2017-02-19 10:30:29,968 INFO repeater task nb300_beamplots starting 16.7 seconds late 2017-02-19 10:30:31,921 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 10:30:31,921 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 10:30:31,921 INFO repeater task ashtech_hcorrstats starting 18.6 seconds late 2017-02-19 10:30:32,983 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 10:30:37,989 INFO repeater task nb300_lastensq starting 15.0 seconds late 2017-02-19 10:30:40,582 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:30:40,582 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:30:13 2017-02-19 10:32:15,726 INFO repeater task nb300_10pings starting 2.4 seconds late 2017-02-19 10:32:15,911 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:32:25,923 INFO repeater task nb300_lastensq starting 2.9 seconds late 2017-02-19 10:32:28,548 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:32:28,549 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:32:13 2017-02-19 10:34:23,699 INFO repeater task nb300_lastensq starting 0.7 seconds late 2017-02-19 10:34:26,273 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:34:26,274 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:34:13 2017-02-19 10:34:36,288 INFO repeater task nb300_10pings starting 3.0 seconds late 2017-02-19 10:34:36,470 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:35:16,526 INFO repeater task ashtech_hcorrstats starting 3.2 seconds late 2017-02-19 10:35:17,574 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 10:36:27,669 INFO repeater task nb300_lastensq starting 4.7 seconds late 2017-02-19 10:36:30,294 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:36:30,295 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:36:13 2017-02-19 10:36:55,326 INFO repeater task nb300_10pings starting 2.0 seconds late 2017-02-19 10:36:55,508 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:38:25,624 INFO repeater task nb300_lastensq starting 2.6 seconds late 2017-02-19 10:38:28,198 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:38:28,199 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:38:13 2017-02-19 10:39:18,262 INFO repeater task nb300_10pings starting 5.0 seconds late 2017-02-19 10:39:18,445 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:40:13,526 INFO repeater task ashtech_hcorrstats starting 0.2 seconds late 2017-02-19 10:40:14,571 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 10:40:24,584 INFO repeater task nb300_lastensq starting 1.6 seconds late 2017-02-19 10:40:27,182 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:40:27,183 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:40:13 2017-02-19 10:41:37,289 INFO repeater task nb300_10pings starting 4.0 seconds late 2017-02-19 10:41:37,470 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:42:27,534 INFO repeater task nb300_lastensq starting 4.5 seconds late 2017-02-19 10:42:30,124 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:42:30,124 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:42:13 2017-02-19 10:43:55,247 INFO repeater task nb300_10pings starting 2.0 seconds late 2017-02-19 10:43:55,429 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:44:25,471 INFO repeater task nb300_lastensq starting 2.5 seconds late 2017-02-19 10:44:28,064 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:44:28,064 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:44:13 2017-02-19 10:45:18,136 INFO repeater task nb300_quick starting 4.8 seconds late 2017-02-19 10:45:23,835 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 10:45:23,835 INFO repeater task nb300_3dayplots starting 10.5 seconds late 2017-02-19 10:45:29,452 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 10:45:29,452 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 10:45:29,453 INFO repeater task nb300_tsstats starting 16.2 seconds late 2017-02-19 10:45:30,464 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 10:45:30,465 INFO repeater task nb300_beamplots starting 17.2 seconds late 2017-02-19 10:45:32,349 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 10:45:32,349 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 10:45:32,349 INFO repeater task nb300_beamplots starting 19.1 seconds late 2017-02-19 10:45:34,296 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 10:45:34,296 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 10:45:34,297 INFO repeater task ashtech_hcorrstats starting 21.0 seconds late 2017-02-19 10:45:35,336 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 10:46:15,390 INFO repeater task nb300_10pings starting 2.1 seconds late 2017-02-19 10:46:15,570 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:46:25,584 INFO repeater task nb300_lastensq starting 2.6 seconds late 2017-02-19 10:46:28,189 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:46:28,189 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:46:13 2017-02-19 10:48:23,351 INFO repeater task nb300_lastensq starting 0.4 seconds late 2017-02-19 10:48:25,934 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:48:25,934 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:48:13 2017-02-19 10:48:35,949 INFO repeater task nb300_10pings starting 2.7 seconds late 2017-02-19 10:48:36,132 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:50:16,279 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2017-02-19 10:50:17,318 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 10:50:27,331 INFO repeater task nb300_lastensq starting 4.3 seconds late 2017-02-19 10:50:29,923 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:50:29,923 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:50:13 2017-02-19 10:50:54,950 INFO repeater task nb300_10pings starting 1.7 seconds late 2017-02-19 10:50:55,128 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:52:25,253 INFO repeater task nb300_lastensq starting 2.3 seconds late 2017-02-19 10:52:27,840 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:52:27,840 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:52:13 2017-02-19 10:53:17,912 INFO repeater task nb300_10pings starting 4.6 seconds late 2017-02-19 10:53:18,091 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:54:23,185 INFO repeater task nb300_lastensq starting 0.2 seconds late 2017-02-19 10:54:25,790 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:54:25,791 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:54:13 2017-02-19 10:55:15,847 INFO repeater task ashtech_hcorrstats starting 2.5 seconds late 2017-02-19 10:55:16,887 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 10:55:36,915 INFO repeater task nb300_10pings starting 3.6 seconds late 2017-02-19 10:55:37,091 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:56:27,168 INFO repeater task nb300_lastensq starting 4.2 seconds late 2017-02-19 10:56:29,815 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:56:29,815 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:56:13 2017-02-19 10:57:54,924 INFO repeater task nb300_10pings starting 1.6 seconds late 2017-02-19 10:57:55,106 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 10:58:25,145 INFO repeater task nb300_lastensq starting 2.1 seconds late 2017-02-19 10:58:27,725 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 10:58:27,726 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 10:58:13 2017-02-19 11:00:17,883 INFO repeater task nb300_10pings starting 4.6 seconds late 2017-02-19 11:00:18,062 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:00:18,062 INFO repeater task nb300_quick starting 4.8 seconds late 2017-02-19 11:00:23,703 INFO repeater task nb300_quick ended after 5.6 seconds 2017-02-19 11:00:23,704 INFO repeater task nb300_3dayplots starting 10.4 seconds late 2017-02-19 11:00:29,351 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 11:00:29,351 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 11:00:29,352 INFO repeater task nb300_tsstats starting 16.1 seconds late 2017-02-19 11:00:30,376 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 11:00:30,376 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 11:00:30,376 INFO repeater task nb300_beamplots starting 17.1 seconds late 2017-02-19 11:00:32,237 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:00:32,237 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 11:00:32,238 INFO repeater task nb300_beamplots starting 18.9 seconds late 2017-02-19 11:00:34,190 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:00:34,191 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 11:00:34,191 INFO repeater task ashtech_hcorrstats starting 20.9 seconds late 2017-02-19 11:00:35,237 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:00:40,243 INFO repeater task nb300_lastensq starting 17.2 seconds late 2017-02-19 11:00:42,845 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:00:42,846 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:00:13 2017-02-19 11:02:28,002 INFO repeater task nb300_lastensq starting 5.0 seconds late 2017-02-19 11:02:30,584 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:02:30,584 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:02:13 2017-02-19 11:02:35,591 INFO repeater task nb300_10pings starting 2.3 seconds late 2017-02-19 11:02:35,772 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:04:25,931 INFO repeater task nb300_lastensq starting 2.9 seconds late 2017-02-19 11:04:28,560 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:04:28,561 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:04:13 2017-02-19 11:04:53,589 INFO repeater task nb300_10pings starting 0.3 seconds late 2017-02-19 11:04:53,771 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:05:13,799 INFO repeater task ashtech_hcorrstats starting 0.5 seconds late 2017-02-19 11:05:14,843 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:06:24,937 INFO repeater task nb300_lastensq starting 1.9 seconds late 2017-02-19 11:06:27,541 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:06:27,541 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:06:13 2017-02-19 11:07:17,615 INFO repeater task nb300_10pings starting 4.3 seconds late 2017-02-19 11:07:17,796 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:08:27,903 INFO repeater task nb300_lastensq starting 4.9 seconds late 2017-02-19 11:08:30,501 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:08:30,501 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:08:13 2017-02-19 11:09:35,594 INFO repeater task nb300_10pings starting 2.3 seconds late 2017-02-19 11:09:35,773 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:10:15,834 INFO repeater task ashtech_hcorrstats starting 2.5 seconds late 2017-02-19 11:10:16,873 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:10:26,887 INFO repeater task nb300_lastensq starting 3.9 seconds late 2017-02-19 11:10:29,485 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:10:29,485 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:10:13 2017-02-19 11:11:54,611 INFO repeater task nb300_10pings starting 1.3 seconds late 2017-02-19 11:11:54,791 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:12:24,835 INFO repeater task nb300_lastensq starting 1.8 seconds late 2017-02-19 11:12:27,439 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:12:27,440 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:12:13 2017-02-19 11:14:17,603 INFO repeater task nb300_10pings starting 4.3 seconds late 2017-02-19 11:14:17,781 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:14:27,796 INFO repeater task nb300_lastensq starting 4.8 seconds late 2017-02-19 11:14:30,408 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:14:30,408 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:14:13 2017-02-19 11:15:15,477 INFO repeater task nb300_quick starting 2.2 seconds late 2017-02-19 11:15:21,138 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 11:15:21,139 INFO repeater task nb300_3dayplots starting 7.8 seconds late 2017-02-19 11:15:26,751 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 11:15:26,751 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 11:15:26,752 INFO repeater task nb300_tsstats starting 13.5 seconds late 2017-02-19 11:15:27,771 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 11:15:27,771 INFO repeater task nb300_beamplots starting 14.5 seconds late 2017-02-19 11:15:29,661 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:15:29,661 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 11:15:29,661 INFO repeater task nb300_beamplots starting 16.4 seconds late 2017-02-19 11:15:31,621 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:15:31,621 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 11:15:31,622 INFO repeater task ashtech_hcorrstats starting 18.3 seconds late 2017-02-19 11:15:32,662 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:16:27,747 INFO repeater task nb300_lastensq starting 4.7 seconds late 2017-02-19 11:16:30,368 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:16:30,369 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:16:13 2017-02-19 11:16:35,375 INFO repeater task nb300_10pings starting 2.1 seconds late 2017-02-19 11:16:35,556 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:18:25,717 INFO repeater task nb300_lastensq starting 2.7 seconds late 2017-02-19 11:18:28,325 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:18:28,326 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:18:13 2017-02-19 11:18:53,359 INFO repeater task nb300_10pings starting 0.1 seconds late 2017-02-19 11:18:53,537 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:20:13,641 INFO repeater task ashtech_hcorrstats starting 0.3 seconds late 2017-02-19 11:20:14,693 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 11:20:24,704 INFO repeater task nb300_lastensq starting 1.7 seconds late 2017-02-19 11:20:27,329 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:20:27,329 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:20:13 2017-02-19 11:21:17,398 INFO repeater task nb300_10pings starting 4.1 seconds late 2017-02-19 11:21:17,580 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:22:27,681 INFO repeater task nb300_lastensq starting 4.7 seconds late 2017-02-19 11:22:30,282 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:22:30,282 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:22:13 2017-02-19 11:23:35,367 INFO repeater task nb300_10pings starting 2.1 seconds late 2017-02-19 11:23:35,546 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:24:25,615 INFO repeater task nb300_lastensq starting 2.6 seconds late 2017-02-19 11:24:28,226 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:24:28,226 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:24:13 2017-02-19 11:25:18,285 INFO repeater task ashtech_hcorrstats starting 5.0 seconds late 2017-02-19 11:25:19,328 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:25:54,380 INFO repeater task nb300_10pings starting 1.1 seconds late 2017-02-19 11:25:54,559 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:26:24,599 INFO repeater task nb300_lastensq starting 1.6 seconds late 2017-02-19 11:26:27,229 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:26:27,229 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:26:13 2017-02-19 11:28:17,385 INFO repeater task nb300_10pings starting 4.1 seconds late 2017-02-19 11:28:17,566 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:28:27,581 INFO repeater task nb300_lastensq starting 4.6 seconds late 2017-02-19 11:28:30,224 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:28:30,224 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:28:13 2017-02-19 11:30:15,362 INFO repeater task nb300_quick starting 2.1 seconds late 2017-02-19 11:30:21,026 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 11:30:21,026 INFO repeater task nb300_3dayplots starting 7.7 seconds late 2017-02-19 11:30:26,633 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 11:30:26,633 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 11:30:26,634 INFO repeater task nb300_tsstats starting 13.3 seconds late 2017-02-19 11:30:27,648 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 11:30:27,649 INFO repeater task nb300_beamplots starting 14.4 seconds late 2017-02-19 11:30:29,532 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:30:29,532 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 11:30:29,532 INFO repeater task nb300_beamplots starting 16.2 seconds late 2017-02-19 11:30:31,477 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:30:31,477 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 11:30:31,478 INFO repeater task ashtech_hcorrstats starting 18.2 seconds late 2017-02-19 11:30:32,511 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:30:37,514 INFO repeater task nb300_lastensq starting 14.5 seconds late 2017-02-19 11:30:40,140 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:30:40,140 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:30:13 2017-02-19 11:30:40,141 INFO repeater task nb300_10pings starting 6.9 seconds late 2017-02-19 11:30:40,320 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:32:25,467 INFO repeater task nb300_lastensq starting 2.5 seconds late 2017-02-19 11:32:28,065 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:32:28,066 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:32:13 2017-02-19 11:32:58,106 INFO repeater task nb300_10pings starting 4.8 seconds late 2017-02-19 11:32:58,287 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:34:23,399 INFO repeater task nb300_lastensq starting 0.4 seconds late 2017-02-19 11:34:26,021 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:34:26,021 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:34:13 2017-02-19 11:35:16,093 INFO repeater task nb300_10pings starting 2.8 seconds late 2017-02-19 11:35:16,273 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:35:16,274 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2017-02-19 11:35:17,325 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 11:36:27,426 INFO repeater task nb300_lastensq starting 4.4 seconds late 2017-02-19 11:36:30,025 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:36:30,026 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:36:13 2017-02-19 11:37:35,115 INFO repeater task nb300_10pings starting 1.8 seconds late 2017-02-19 11:37:35,297 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:38:25,368 INFO repeater task nb300_lastensq starting 2.4 seconds late 2017-02-19 11:38:27,992 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:38:27,992 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:38:13 2017-02-19 11:39:58,122 INFO repeater task nb300_10pings starting 4.8 seconds late 2017-02-19 11:39:58,306 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:40:13,327 INFO repeater task ashtech_hcorrstats starting 0.0 seconds late 2017-02-19 11:40:14,385 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 11:40:24,399 INFO repeater task nb300_lastensq starting 1.4 seconds late 2017-02-19 11:40:27,022 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:40:27,022 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:40:13 2017-02-19 11:42:17,187 INFO repeater task nb300_10pings starting 3.9 seconds late 2017-02-19 11:42:17,368 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:42:27,382 INFO repeater task nb300_lastensq starting 4.4 seconds late 2017-02-19 11:42:29,999 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:42:30,000 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:42:13 2017-02-19 11:44:25,163 INFO repeater task nb300_lastensq starting 2.2 seconds late 2017-02-19 11:44:27,773 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:44:27,773 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:44:13 2017-02-19 11:44:37,788 INFO repeater task nb300_10pings starting 4.5 seconds late 2017-02-19 11:44:37,968 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:45:18,025 INFO repeater task nb300_quick starting 4.7 seconds late 2017-02-19 11:45:23,693 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 11:45:23,693 INFO repeater task nb300_3dayplots starting 10.4 seconds late 2017-02-19 11:45:29,281 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 11:45:29,281 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 11:45:29,281 INFO repeater task nb300_tsstats starting 16.0 seconds late 2017-02-19 11:45:30,298 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 11:45:30,299 INFO repeater task nb300_beamplots starting 17.0 seconds late 2017-02-19 11:45:32,199 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:45:32,199 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 11:45:32,200 INFO repeater task nb300_beamplots starting 18.9 seconds late 2017-02-19 11:45:34,161 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 11:45:34,161 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 11:45:34,161 INFO repeater task ashtech_hcorrstats starting 20.9 seconds late 2017-02-19 11:45:35,210 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:46:25,284 INFO repeater task nb300_lastensq starting 2.3 seconds late 2017-02-19 11:46:27,910 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:46:27,911 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:46:13 2017-02-19 11:46:57,953 INFO repeater task nb300_10pings starting 4.7 seconds late 2017-02-19 11:46:58,132 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:48:23,249 INFO repeater task nb300_lastensq starting 0.2 seconds late 2017-02-19 11:48:25,861 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:48:25,862 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:48:13 2017-02-19 11:49:15,936 INFO repeater task nb300_10pings starting 2.6 seconds late 2017-02-19 11:49:16,116 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:50:16,206 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2017-02-19 11:50:17,267 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 11:50:27,280 INFO repeater task nb300_lastensq starting 4.3 seconds late 2017-02-19 11:50:29,914 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:50:29,915 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:50:13 2017-02-19 11:51:35,014 INFO repeater task nb300_10pings starting 1.7 seconds late 2017-02-19 11:51:35,196 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:52:25,260 INFO repeater task nb300_lastensq starting 2.3 seconds late 2017-02-19 11:52:27,856 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:52:27,856 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:52:13 2017-02-19 11:53:57,987 INFO repeater task nb300_10pings starting 4.7 seconds late 2017-02-19 11:53:58,164 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:54:23,200 INFO repeater task nb300_lastensq starting 0.2 seconds late 2017-02-19 11:54:25,817 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:54:25,817 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:54:13 2017-02-19 11:55:15,892 INFO repeater task ashtech_hcorrstats starting 2.6 seconds late 2017-02-19 11:55:16,937 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 11:56:17,022 INFO repeater task nb300_10pings starting 3.7 seconds late 2017-02-19 11:56:17,200 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 11:56:27,214 INFO repeater task nb300_lastensq starting 4.2 seconds late 2017-02-19 11:56:29,823 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:56:29,824 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:56:13 2017-02-19 11:58:24,997 INFO repeater task nb300_lastensq starting 2.0 seconds late 2017-02-19 11:58:27,634 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 11:58:27,634 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 11:58:13 2017-02-19 11:58:37,643 INFO repeater task nb300_10pings starting 4.4 seconds late 2017-02-19 11:58:37,824 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:00:17,956 INFO repeater task nb300_quick starting 4.7 seconds late 2017-02-19 12:00:23,638 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 12:00:23,639 INFO repeater task nb300_3dayplots starting 10.3 seconds late 2017-02-19 12:00:29,266 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 12:00:29,267 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 12:00:29,267 INFO repeater task nb300_tsstats starting 16.0 seconds late 2017-02-19 12:00:30,280 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 12:00:30,281 INFO repeater task nb300_beamplots starting 17.0 seconds late 2017-02-19 12:00:32,162 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:00:32,162 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 12:00:32,162 INFO repeater task nb300_beamplots starting 18.9 seconds late 2017-02-19 12:00:34,069 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:00:34,069 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 12:00:34,070 INFO repeater task ashtech_hcorrstats starting 20.8 seconds late 2017-02-19 12:00:35,111 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:00:40,117 INFO repeater task nb300_lastensq starting 17.1 seconds late 2017-02-19 12:00:42,777 INFO repeater task nb300_lastensq ended after 2.7 seconds 2017-02-19 12:00:42,778 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:00:13 2017-02-19 12:00:57,798 INFO repeater task nb300_10pings starting 4.5 seconds late 2017-02-19 12:00:57,981 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:02:23,098 INFO repeater task nb300_lastensq starting 0.1 seconds late 2017-02-19 12:02:25,674 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:02:25,674 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:02:13 2017-02-19 12:03:15,750 INFO repeater task nb300_10pings starting 2.5 seconds late 2017-02-19 12:03:15,928 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:04:26,035 INFO repeater task nb300_lastensq starting 3.0 seconds late 2017-02-19 12:04:28,619 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:04:28,620 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:04:13 2017-02-19 12:05:13,690 INFO repeater task ashtech_hcorrstats starting 0.4 seconds late 2017-02-19 12:05:14,735 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:05:34,759 INFO repeater task nb300_10pings starting 1.5 seconds late 2017-02-19 12:05:34,939 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:06:25,009 INFO repeater task nb300_lastensq starting 2.0 seconds late 2017-02-19 12:06:27,591 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:06:27,591 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:06:13 2017-02-19 12:07:57,719 INFO repeater task nb300_10pings starting 4.4 seconds late 2017-02-19 12:07:57,900 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:08:27,944 INFO repeater task nb300_lastensq starting 4.9 seconds late 2017-02-19 12:08:30,543 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:08:30,544 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:08:13 2017-02-19 12:10:15,700 INFO repeater task nb300_10pings starting 2.4 seconds late 2017-02-19 12:10:15,877 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:10:15,878 INFO repeater task ashtech_hcorrstats starting 2.6 seconds late 2017-02-19 12:10:16,917 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:10:26,930 INFO repeater task nb300_lastensq starting 3.9 seconds late 2017-02-19 12:10:29,510 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:10:29,510 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:10:13 2017-02-19 12:12:24,676 INFO repeater task nb300_lastensq starting 1.7 seconds late 2017-02-19 12:12:27,274 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:12:27,274 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:12:13 2017-02-19 12:12:37,289 INFO repeater task nb300_10pings starting 4.0 seconds late 2017-02-19 12:12:37,468 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:14:27,615 INFO repeater task nb300_lastensq starting 4.6 seconds late 2017-02-19 12:14:30,195 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:14:30,195 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:14:13 2017-02-19 12:14:55,229 INFO repeater task nb300_10pings starting 1.9 seconds late 2017-02-19 12:14:55,410 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:15:15,439 INFO repeater task nb300_quick starting 2.1 seconds late 2017-02-19 12:15:21,127 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 12:15:21,128 INFO repeater task nb300_3dayplots starting 7.8 seconds late 2017-02-19 12:15:26,715 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 12:15:26,715 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 12:15:26,716 INFO repeater task nb300_tsstats starting 13.4 seconds late 2017-02-19 12:15:27,720 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 12:15:27,720 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 12:15:27,721 INFO repeater task nb300_beamplots starting 14.4 seconds late 2017-02-19 12:15:29,585 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:15:29,586 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 12:15:29,586 INFO repeater task nb300_beamplots starting 16.3 seconds late 2017-02-19 12:15:31,546 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:15:31,546 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 12:15:31,547 INFO repeater task ashtech_hcorrstats starting 18.2 seconds late 2017-02-19 12:15:32,593 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:16:27,652 INFO repeater task nb300_lastensq starting 4.7 seconds late 2017-02-19 12:16:30,242 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:16:30,243 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:16:13 2017-02-19 12:17:15,304 INFO repeater task nb300_10pings starting 2.0 seconds late 2017-02-19 12:17:15,501 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:18:25,585 INFO repeater task nb300_lastensq starting 2.6 seconds late 2017-02-19 12:18:28,177 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:18:28,177 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:18:13 2017-02-19 12:19:38,267 INFO repeater task nb300_10pings starting 5.0 seconds late 2017-02-19 12:19:38,448 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:20:13,503 INFO repeater task ashtech_hcorrstats starting 0.2 seconds late 2017-02-19 12:20:14,549 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:20:24,563 INFO repeater task nb300_lastensq starting 1.6 seconds late 2017-02-19 12:20:27,154 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:20:27,154 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:20:13 2017-02-19 12:21:57,279 INFO repeater task nb300_10pings starting 4.0 seconds late 2017-02-19 12:21:57,457 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:22:27,500 INFO repeater task nb300_lastensq starting 4.5 seconds late 2017-02-19 12:22:30,064 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:22:30,064 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:22:13 2017-02-19 12:24:15,205 INFO repeater task nb300_10pings starting 1.9 seconds late 2017-02-19 12:24:15,384 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:24:25,391 INFO repeater task nb300_lastensq starting 2.4 seconds late 2017-02-19 12:24:27,993 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:24:27,994 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:24:13 2017-02-19 12:25:18,053 INFO repeater task ashtech_hcorrstats starting 4.8 seconds late 2017-02-19 12:25:19,098 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:26:24,193 INFO repeater task nb300_lastensq starting 1.2 seconds late 2017-02-19 12:26:26,762 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:26:26,762 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:26:13 2017-02-19 12:26:36,777 INFO repeater task nb300_10pings starting 3.5 seconds late 2017-02-19 12:26:36,958 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:28:27,126 INFO repeater task nb300_lastensq starting 4.1 seconds late 2017-02-19 12:28:29,724 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:28:29,725 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:28:13 2017-02-19 12:28:54,762 INFO repeater task nb300_10pings starting 1.5 seconds late 2017-02-19 12:28:54,941 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:30:15,054 INFO repeater task nb300_quick starting 1.8 seconds late 2017-02-19 12:30:20,751 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 12:30:20,751 INFO repeater task nb300_3dayplots starting 7.5 seconds late 2017-02-19 12:30:26,348 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 12:30:26,348 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 12:30:26,349 INFO repeater task nb300_tsstats starting 13.1 seconds late 2017-02-19 12:30:27,365 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 12:30:27,365 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 12:30:27,366 INFO repeater task nb300_beamplots starting 14.1 seconds late 2017-02-19 12:30:29,245 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:30:29,245 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 12:30:29,246 INFO repeater task nb300_beamplots starting 16.0 seconds late 2017-02-19 12:30:31,155 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:30:31,155 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 12:30:31,155 INFO repeater task ashtech_hcorrstats starting 17.9 seconds late 2017-02-19 12:30:32,206 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 12:30:37,211 INFO repeater task nb300_lastensq starting 14.2 seconds late 2017-02-19 12:30:39,795 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:30:39,795 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:30:13 2017-02-19 12:31:14,848 INFO repeater task nb300_10pings starting 1.6 seconds late 2017-02-19 12:31:15,028 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:32:25,125 INFO repeater task nb300_lastensq starting 2.1 seconds late 2017-02-19 12:32:27,709 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:32:27,710 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:32:13 2017-02-19 12:33:37,809 INFO repeater task nb300_10pings starting 4.5 seconds late 2017-02-19 12:33:37,990 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:34:23,057 INFO repeater task nb300_lastensq starting 0.1 seconds late 2017-02-19 12:34:25,645 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:34:25,645 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:34:13 2017-02-19 12:35:15,720 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2017-02-19 12:35:16,768 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:35:56,827 INFO repeater task nb300_10pings starting 3.5 seconds late 2017-02-19 12:35:57,008 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:36:27,053 INFO repeater task nb300_lastensq starting 4.1 seconds late 2017-02-19 12:36:29,624 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:36:29,625 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:36:13 2017-02-19 12:38:14,772 INFO repeater task nb300_10pings starting 1.5 seconds late 2017-02-19 12:38:14,952 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:38:24,966 INFO repeater task nb300_lastensq starting 2.0 seconds late 2017-02-19 12:38:27,558 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:38:27,558 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:38:13 2017-02-19 12:40:17,722 INFO repeater task ashtech_hcorrstats starting 4.4 seconds late 2017-02-19 12:40:18,770 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:40:23,774 INFO repeater task nb300_lastensq starting 0.8 seconds late 2017-02-19 12:40:26,387 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:40:26,387 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:40:13 2017-02-19 12:40:36,402 INFO repeater task nb300_10pings starting 3.1 seconds late 2017-02-19 12:40:36,590 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:42:26,713 INFO repeater task nb300_lastensq starting 3.7 seconds late 2017-02-19 12:42:29,331 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:42:29,332 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:42:13 2017-02-19 12:42:54,365 INFO repeater task nb300_10pings starting 1.1 seconds late 2017-02-19 12:42:54,543 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:44:24,645 INFO repeater task nb300_lastensq starting 1.6 seconds late 2017-02-19 12:44:27,223 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:44:27,223 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:44:13 2017-02-19 12:45:17,299 INFO repeater task nb300_10pings starting 4.0 seconds late 2017-02-19 12:45:17,476 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:45:17,476 INFO repeater task nb300_quick starting 4.2 seconds late 2017-02-19 12:45:23,196 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 12:45:23,196 INFO repeater task nb300_3dayplots starting 9.9 seconds late 2017-02-19 12:45:28,792 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 12:45:28,792 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 12:45:28,792 INFO repeater task nb300_tsstats starting 15.5 seconds late 2017-02-19 12:45:29,794 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 12:45:29,794 INFO repeater task nb300_beamplots starting 16.5 seconds late 2017-02-19 12:45:31,635 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:45:31,635 INFO repeater task nb300_beamplots ended after 1.8 seconds 2017-02-19 12:45:31,635 INFO repeater task nb300_beamplots starting 18.3 seconds late 2017-02-19 12:45:33,584 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 12:45:33,584 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 12:45:33,584 INFO repeater task ashtech_hcorrstats starting 20.3 seconds late 2017-02-19 12:45:34,628 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:46:24,682 INFO repeater task nb300_lastensq starting 1.7 seconds late 2017-02-19 12:46:27,246 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:46:27,246 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:46:13 2017-02-19 12:47:37,342 INFO repeater task nb300_10pings starting 4.1 seconds late 2017-02-19 12:47:37,525 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:48:27,597 INFO repeater task nb300_lastensq starting 4.6 seconds late 2017-02-19 12:48:30,193 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:48:30,194 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:48:13 2017-02-19 12:49:55,314 INFO repeater task nb300_10pings starting 2.0 seconds late 2017-02-19 12:49:55,494 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:50:15,525 INFO repeater task ashtech_hcorrstats starting 2.2 seconds late 2017-02-19 12:50:16,555 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 12:50:26,569 INFO repeater task nb300_lastensq starting 3.6 seconds late 2017-02-19 12:50:29,145 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:50:29,145 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:50:13 2017-02-19 12:52:14,291 INFO repeater task nb300_10pings starting 1.0 seconds late 2017-02-19 12:52:14,485 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:52:24,495 INFO repeater task nb300_lastensq starting 1.5 seconds late 2017-02-19 12:52:27,063 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:52:27,063 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:52:13 2017-02-19 12:54:27,226 INFO repeater task nb300_lastensq starting 4.2 seconds late 2017-02-19 12:54:29,794 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:54:29,794 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:54:13 2017-02-19 12:54:34,802 INFO repeater task nb300_10pings starting 1.5 seconds late 2017-02-19 12:54:34,982 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:55:15,042 INFO repeater task ashtech_hcorrstats starting 1.7 seconds late 2017-02-19 12:55:16,105 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2017-02-19 12:56:26,207 INFO repeater task nb300_lastensq starting 3.2 seconds late 2017-02-19 12:56:28,791 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:56:28,791 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:56:13 2017-02-19 12:56:53,826 INFO repeater task nb300_10pings starting 0.5 seconds late 2017-02-19 12:56:54,004 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 12:58:24,133 INFO repeater task nb300_lastensq starting 1.1 seconds late 2017-02-19 12:58:26,746 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 12:58:26,746 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 12:58:13 2017-02-19 12:59:16,817 INFO repeater task nb300_10pings starting 3.5 seconds late 2017-02-19 12:59:16,998 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:00:17,087 INFO repeater task nb300_quick starting 3.8 seconds late 2017-02-19 13:00:22,787 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 13:00:22,788 INFO repeater task nb300_3dayplots starting 9.5 seconds late 2017-02-19 13:00:28,414 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 13:00:28,414 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 13:00:28,414 INFO repeater task nb300_tsstats starting 15.1 seconds late 2017-02-19 13:00:29,427 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 13:00:29,427 INFO repeater task nb300_beamplots starting 16.1 seconds late 2017-02-19 13:00:31,310 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:00:31,310 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 13:00:31,310 INFO repeater task nb300_beamplots starting 18.0 seconds late 2017-02-19 13:00:33,305 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:00:33,305 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 13:00:33,306 INFO repeater task ashtech_hcorrstats starting 20.0 seconds late 2017-02-19 13:00:34,349 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:00:39,355 INFO repeater task nb300_lastensq starting 16.4 seconds late 2017-02-19 13:00:41,975 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:00:41,975 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:00:41,976 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:00:13 2017-02-19 13:01:37,056 INFO repeater task nb300_10pings starting 3.8 seconds late 2017-02-19 13:01:37,237 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:02:27,306 INFO repeater task nb300_lastensq starting 4.3 seconds late 2017-02-19 13:02:29,929 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:02:29,930 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:02:13 2017-02-19 13:03:55,053 INFO repeater task nb300_10pings starting 1.8 seconds late 2017-02-19 13:03:55,230 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:04:25,274 INFO repeater task nb300_lastensq starting 2.3 seconds late 2017-02-19 13:04:27,899 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:04:27,900 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:04:13 2017-02-19 13:05:17,963 INFO repeater task ashtech_hcorrstats starting 4.7 seconds late 2017-02-19 13:05:19,000 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:06:14,083 INFO repeater task nb300_10pings starting 0.8 seconds late 2017-02-19 13:06:14,261 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:06:24,275 INFO repeater task nb300_lastensq starting 1.3 seconds late 2017-02-19 13:06:26,871 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:06:26,872 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:06:13 2017-02-19 13:08:27,053 INFO repeater task nb300_lastensq starting 4.1 seconds late 2017-02-19 13:08:29,695 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:08:29,696 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:08:29,696 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:08:13 2017-02-19 13:08:34,703 INFO repeater task nb300_10pings starting 1.4 seconds late 2017-02-19 13:08:34,885 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:10:14,999 INFO repeater task ashtech_hcorrstats starting 1.7 seconds late 2017-02-19 13:10:16,038 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:10:26,051 INFO repeater task nb300_lastensq starting 3.1 seconds late 2017-02-19 13:10:28,671 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:10:28,671 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:10:13 2017-02-19 13:10:53,704 INFO repeater task nb300_10pings starting 0.4 seconds late 2017-02-19 13:10:53,886 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:12:24,013 INFO repeater task nb300_lastensq starting 1.0 seconds late 2017-02-19 13:12:26,649 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:12:26,650 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:12:26,650 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:12:13 2017-02-19 13:13:16,722 INFO repeater task nb300_10pings starting 3.4 seconds late 2017-02-19 13:13:16,904 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:14:26,980 INFO repeater task nb300_lastensq starting 4.0 seconds late 2017-02-19 13:14:29,592 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:14:29,592 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:14:29,593 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:14:13 2017-02-19 13:15:14,654 INFO repeater task nb300_quick starting 1.4 seconds late 2017-02-19 13:15:20,391 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 13:15:20,391 INFO repeater task nb300_3dayplots starting 7.1 seconds late 2017-02-19 13:15:26,053 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 13:15:26,053 INFO repeater task nb300_3dayplots ended after 5.7 seconds 2017-02-19 13:15:26,053 INFO repeater task nb300_tsstats starting 12.8 seconds late 2017-02-19 13:15:27,066 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 13:15:27,067 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 13:15:27,067 INFO repeater task nb300_beamplots starting 13.8 seconds late 2017-02-19 13:15:28,938 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:15:28,938 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 13:15:28,939 INFO repeater task nb300_beamplots starting 15.6 seconds late 2017-02-19 13:15:30,905 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:15:30,905 INFO repeater task nb300_beamplots ended after 2.0 seconds 2017-02-19 13:15:30,906 INFO repeater task ashtech_hcorrstats starting 17.6 seconds late 2017-02-19 13:15:31,932 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:15:36,934 INFO repeater task nb300_10pings starting 3.6 seconds late 2017-02-19 13:15:37,113 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:16:27,189 INFO repeater task nb300_lastensq starting 4.2 seconds late 2017-02-19 13:16:29,817 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:16:29,817 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:16:29,817 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:16:13 2017-02-19 13:17:54,916 INFO repeater task nb300_10pings starting 1.6 seconds late 2017-02-19 13:17:55,095 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:18:25,139 INFO repeater task nb300_lastensq starting 2.1 seconds late 2017-02-19 13:18:27,772 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:18:27,772 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:18:13 2017-02-19 13:20:17,928 INFO repeater task nb300_10pings starting 4.6 seconds late 2017-02-19 13:20:18,109 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:20:18,110 INFO repeater task ashtech_hcorrstats starting 4.8 seconds late 2017-02-19 13:20:19,152 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:20:24,158 INFO repeater task nb300_lastensq starting 1.2 seconds late 2017-02-19 13:20:26,807 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:20:26,807 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:20:13 2017-02-19 13:22:26,963 INFO repeater task nb300_lastensq starting 4.0 seconds late 2017-02-19 13:22:29,597 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:22:29,598 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:22:13 2017-02-19 13:22:34,605 INFO repeater task nb300_10pings starting 1.3 seconds late 2017-02-19 13:22:34,786 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:24:24,945 INFO repeater task nb300_lastensq starting 1.9 seconds late 2017-02-19 13:24:27,551 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:24:27,551 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:24:13 2017-02-19 13:24:57,591 INFO repeater task nb300_10pings starting 4.3 seconds late 2017-02-19 13:24:57,768 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:25:17,800 INFO repeater task ashtech_hcorrstats starting 4.5 seconds late 2017-02-19 13:25:18,846 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:26:23,943 INFO repeater task nb300_lastensq starting 0.9 seconds late 2017-02-19 13:26:26,543 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:26:26,543 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:26:26,544 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:26:13 2017-02-19 13:27:16,617 INFO repeater task nb300_10pings starting 3.3 seconds late 2017-02-19 13:27:16,800 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:28:26,904 INFO repeater task nb300_lastensq starting 3.9 seconds late 2017-02-19 13:28:29,499 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:28:29,500 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:28:13 2017-02-19 13:29:34,598 INFO repeater task nb300_10pings starting 1.3 seconds late 2017-02-19 13:29:34,776 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:30:14,831 INFO repeater task nb300_quick starting 1.5 seconds late 2017-02-19 13:30:20,552 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 13:30:20,553 INFO repeater task nb300_3dayplots starting 7.3 seconds late 2017-02-19 13:30:26,186 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 13:30:26,186 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 13:30:26,187 INFO repeater task nb300_tsstats starting 12.9 seconds late 2017-02-19 13:30:27,197 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 13:30:27,198 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 13:30:27,198 INFO repeater task nb300_beamplots starting 13.9 seconds late 2017-02-19 13:30:29,028 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:30:29,028 INFO repeater task nb300_beamplots ended after 1.8 seconds 2017-02-19 13:30:29,028 INFO repeater task nb300_beamplots starting 15.7 seconds late 2017-02-19 13:30:30,976 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:30:30,976 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 13:30:30,977 INFO repeater task ashtech_hcorrstats starting 17.7 seconds late 2017-02-19 13:30:32,006 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:30:37,011 INFO repeater task nb300_lastensq starting 14.0 seconds late 2017-02-19 13:30:39,648 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-19 13:30:39,648 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:30:39,648 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:30:13 2017-02-19 13:31:54,739 INFO repeater task nb300_10pings starting 1.4 seconds late 2017-02-19 13:31:54,922 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:32:24,968 INFO repeater task nb300_lastensq starting 2.0 seconds late 2017-02-19 13:32:27,560 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:32:27,560 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:32:13 2017-02-19 13:34:17,721 INFO repeater task nb300_10pings starting 4.4 seconds late 2017-02-19 13:34:17,902 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:34:27,916 INFO repeater task nb300_lastensq starting 4.9 seconds late 2017-02-19 13:34:30,554 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:34:30,554 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:34:13 2017-02-19 13:35:15,616 INFO repeater task ashtech_hcorrstats starting 2.3 seconds late 2017-02-19 13:35:16,651 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:36:26,752 INFO repeater task nb300_lastensq starting 3.8 seconds late 2017-02-19 13:36:29,391 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:36:29,392 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:36:13 2017-02-19 13:36:34,399 INFO repeater task nb300_10pings starting 1.1 seconds late 2017-02-19 13:36:34,581 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:38:24,716 INFO repeater task nb300_lastensq starting 1.7 seconds late 2017-02-19 13:38:27,363 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:38:27,364 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:38:13 2017-02-19 13:38:57,401 INFO repeater task nb300_10pings starting 4.1 seconds late 2017-02-19 13:38:57,584 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:40:17,703 INFO repeater task ashtech_hcorrstats starting 4.4 seconds late 2017-02-19 13:40:18,738 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:40:23,744 INFO repeater task nb300_lastensq starting 0.7 seconds late 2017-02-19 13:40:26,384 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:40:26,385 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:40:13 2017-02-19 13:41:16,459 INFO repeater task nb300_10pings starting 3.2 seconds late 2017-02-19 13:41:16,640 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:42:26,734 INFO repeater task nb300_lastensq starting 3.7 seconds late 2017-02-19 13:42:29,367 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:42:29,368 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:42:13 2017-02-19 13:43:34,454 INFO repeater task nb300_10pings starting 1.2 seconds late 2017-02-19 13:43:34,634 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:44:24,710 INFO repeater task nb300_lastensq starting 1.7 seconds late 2017-02-19 13:44:27,338 INFO repeater task nb300_lastensq ended after 2.6 seconds 2017-02-19 13:44:27,338 INFO repeater task nb300_lastensq: Time from /home/data/SP1703/proc/nb300/load/wrote_ens_time: 2017/02/19 13:44:13 2017-02-19 13:45:17,411 INFO repeater task nb300_quick starting 4.1 seconds late 2017-02-19 13:45:23,117 INFO repeater task nb300_quick ended after 5.7 seconds 2017-02-19 13:45:23,117 INFO repeater task nb300_3dayplots starting 9.8 seconds late 2017-02-19 13:45:28,746 INFO repeater task nb300_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-02-19 13:45:28,747 INFO repeater task nb300_3dayplots ended after 5.6 seconds 2017-02-19 13:45:28,747 INFO repeater task nb300_tsstats starting 15.5 seconds late 2017-02-19 13:45:29,759 INFO repeater task nb300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-02-19 13:45:29,759 INFO repeater task nb300_tsstats ended after 1.0 seconds 2017-02-19 13:45:29,760 INFO repeater task nb300_beamplots starting 16.5 seconds late 2017-02-19 13:45:31,629 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:45:31,629 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 13:45:31,630 INFO repeater task nb300_beamplots starting 18.3 seconds late 2017-02-19 13:45:33,524 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-19 13:45:33,524 INFO repeater task nb300_beamplots ended after 1.9 seconds 2017-02-19 13:45:33,525 INFO repeater task ashtech_hcorrstats starting 20.2 seconds late 2017-02-19 13:45:34,550 INFO repeater task ashtech_hcorrstats ended after 1.0 seconds 2017-02-19 13:45:54,580 INFO repeater task nb300_10pings starting 1.3 seconds late 2017-02-19 13:45:54,758 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-19 13:46:07,762 INFO Translate translate exiting, adu /home/data/SP1703/rbin/ashtech 2017-02-19 13:46:07,804 INFO Translate translate exiting, hdg /home/data/SP1703/rbin/gpshead 2017-02-19 13:46:07,805 INFO Translate translate exiting, gps /home/data/SP1703/rbin/gpsnav 2017-02-19 13:46:09,778 INFO root cleanup_figs not_logging 2017-02-19 13:46:09,787 INFO root Exiting DAS_while_logging.py 2017-02-25 15:59:51,535 INFO root Starting DAS_while_logging.py 2017-02-25 15:59:51,615 INFO root Cruiseid: SP1704 2017-02-25 15:59:51,666 INFO root starting run method 2017-02-25 15:59:51,667 INFO Translate gpsnav gps 1 2017-02-25 15:59:51,667 INFO Translate 2017-02-25 15:59:51,667 INFO Translate gpshead hdg 1 2017-02-25 15:59:51,667 INFO Translate 2017-02-25 15:59:51,667 INFO Translate ashtech adu 1 2017-02-25 15:59:51,667 INFO Translate 2017-02-25 15:59:51,670 INFO root active_procdirnames: ['nb300'] 2017-02-25 15:59:51,670 INFO repeater PeriodicSystem: nb300_lastensq interval is 130 seconds cd /home/adcp/uhdas_tmp; run_lastensq.py -d nb300 --update_pygbin --averages --plotens_mpl --kt_vecprof --pingwarnN 30 --ktsdir 2017-02-25 15:59:51,670 INFO repeater PeriodicSystem: nb300_10pings interval is 140 seconds cd /home/adcp/uhdas_tmp; run_grab_lastpings.py -d nb300 2017-02-25 15:59:51,670 INFO repeater PeriodicSystem: nb300_quick interval is 900 seconds cd /home/adcp/uhdas_tmp; run_quick.py -d nb300 2017-02-25 15:59:51,671 INFO repeater PeriodicSystem: nb300_3dayplots interval is 900 seconds cd /home/adcp/uhdas_tmp; run_3dayplots_mpl.py -d nb300 2017-02-25 15:59:51,671 INFO repeater PeriodicSystem: nb300_tsstats interval is 900 seconds cd /home/adcp/uhdas_tmp; run_plot_tsstats.py nb300 2017-02-25 15:59:51,671 INFO repeater PeriodicSystem: nb300_beamplots interval is 900 seconds cd /home/adcp/uhdas_tmp; plot_beamdiagnostics.py -d nb300 --savefigs --webcopy -p scattering 2017-02-25 15:59:51,671 INFO repeater PeriodicSystem: nb300_beamplots interval is 900 seconds cd /home/adcp/uhdas_tmp; plot_beamdiagnostics.py -d nb300 --savefigs --webcopy -p velocity 2017-02-25 15:59:51,672 INFO uhdas_webgen generating /home/adcp/www/figures/thumbnails.html 2017-02-25 15:59:51,672 INFO uhdas_webgen generating /home/adcp/www/figures/beam_diagnostics.html 2017-02-25 15:59:51,672 INFO uhdas_webgen ptype=scattering 2017-02-25 15:59:51,672 INFO uhdas_webgen title=UNCALIBRATED scattering, dB 2017-02-25 15:59:51,672 INFO uhdas_webgen inst=nb300 2017-02-25 15:59:51,672 INFO uhdas_webgen ptype=velocity 2017-02-25 15:59:51,673 INFO uhdas_webgen title=velocity, m/s 2017-02-25 15:59:51,673 INFO uhdas_webgen inst=nb300 2017-02-25 15:59:51,673 INFO uhdas_webgen generating /home/adcp/www/figures/tsstats.html 2017-02-25 15:59:51,673 INFO uhdas_webgen ptype=tsstats 2017-02-25 15:59:51,673 INFO uhdas_webgen title=(some) ocean velocity errors 2017-02-25 15:59:51,673 INFO uhdas_webgen inst=nb300 2017-02-25 15:59:51,673 INFO uhdas_webgen generating /home/adcp/www/figures/sci.html 2017-02-25 15:59:51,673 INFO uhdas_webgen ptype=ktvecprof 2017-02-25 15:59:51,673 INFO uhdas_webgen title=speed+direction profile 2017-02-25 15:59:51,673 INFO uhdas_webgen inst=nb300 2017-02-25 15:59:51,673 INFO uhdas_webgen adding ashtech to simplefiglinks 2017-02-25 15:59:51,674 INFO uhdas_webgen generating /home/adcp/www/simple_figlinks.html 2017-02-25 15:59:51,674 INFO repeater PeriodicSystem: ashtech_hcorrstats interval is 300 seconds cd /home/adcp/uhdas_tmp; run_hbinhcorrstats.py --hcorr_inst ashtech --plotdh --printstats 2017-02-25 15:59:51,674 INFO root cleanup_figs no_figure 2017-02-25 16:02:01,897 INFO repeater task nb300_lastensq starting 0.2 seconds late 2017-02-25 16:02:05,957 INFO repeater task nb300_lastensq ended after 4.1 seconds 2017-02-25 16:02:05,964 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:01:51 2017-02-25 16:02:15,979 INFO repeater task nb300_10pings starting 4.3 seconds late 2017-02-25 16:02:16,164 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:04:01,311 INFO repeater task nb300_lastensq starting 0.3 seconds late 2017-02-25 16:04:03,787 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:04:03,788 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:03:51 2017-02-25 16:04:33,831 INFO repeater task nb300_10pings starting 2.2 seconds late 2017-02-25 16:04:34,008 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:04:54,040 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2017-02-25 16:04:54,783 INFO repeater task ashtech_hcorrstats output: could not get ashtech stats Traceback (most recent call last): File "/usr/local/currents/bin/run_hbinhcorrstats.py", line 148, in H.hcorr_stats(winsecs=winsecs) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/hbin_diagnostics.py", line 309, in hcorr_stats self.enspercent = np.round(100.0 * self.ensnumgood/self.ensnumtotal) ZeroDivisionError: float division by zero 2017-02-25 16:04:54,784 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2017-02-25 16:06:04,884 INFO repeater task nb300_lastensq starting 3.9 seconds late 2017-02-25 16:06:07,365 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:06:07,366 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:05:51 2017-02-25 16:06:52,431 INFO repeater task nb300_10pings starting 0.8 seconds late 2017-02-25 16:06:52,607 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:08:02,710 INFO repeater task nb300_lastensq starting 1.7 seconds late 2017-02-25 16:08:05,211 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:08:05,212 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:07:51 2017-02-25 16:09:15,310 INFO repeater task nb300_10pings starting 3.6 seconds late 2017-02-25 16:09:15,492 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:09:55,553 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2017-02-25 16:09:56,268 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2017-02-25 16:10:01,273 INFO repeater task nb300_lastensq starting 0.3 seconds late 2017-02-25 16:10:03,744 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:10:03,744 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:09:51 2017-02-25 16:11:33,867 INFO repeater task nb300_10pings starting 2.2 seconds late 2017-02-25 16:11:34,044 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:12:04,083 INFO repeater task nb300_lastensq starting 3.1 seconds late 2017-02-25 16:12:06,565 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:12:06,565 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:11:51 2017-02-25 16:13:51,716 INFO repeater task nb300_10pings starting 0.0 seconds late 2017-02-25 16:13:51,893 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:14:01,904 INFO repeater task nb300_lastensq starting 0.9 seconds late 2017-02-25 16:14:04,383 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:14:04,384 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:13:51 2017-02-25 16:14:54,455 INFO repeater task nb300_quick starting 2.8 seconds late 2017-02-25 16:14:56,976 INFO repeater task nb300_quick output: /usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice. warnings.warn("Mean of empty slice.", RuntimeWarning) 2017-02-25 16:14:56,976 INFO repeater task nb300_quick ended after 2.5 seconds 2017-02-25 16:14:56,977 INFO repeater task nb300_3dayplots starting 5.3 seconds late 2017-02-25 16:14:59,854 INFO repeater task nb300_3dayplots ended after 2.9 seconds 2017-02-25 16:14:59,854 INFO repeater task nb300_tsstats starting 8.2 seconds late 2017-02-25 16:15:00,755 INFO repeater task nb300_tsstats ended after 0.9 seconds 2017-02-25 16:15:00,756 INFO repeater task nb300_beamplots starting 9.1 seconds late 2017-02-25 16:15:02,272 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-25 16:15:02,272 INFO repeater task nb300_beamplots ended after 1.5 seconds 2017-02-25 16:15:02,272 INFO repeater task nb300_beamplots starting 10.6 seconds late 2017-02-25 16:15:03,751 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-25 16:15:03,751 INFO repeater task nb300_beamplots ended after 1.5 seconds 2017-02-25 16:15:03,751 INFO repeater task ashtech_hcorrstats starting 12.1 seconds late 2017-02-25 16:15:04,482 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2017-02-25 16:16:04,563 INFO repeater task nb300_lastensq starting 3.6 seconds late 2017-02-25 16:16:07,042 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:16:07,042 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:15:51 2017-02-25 16:16:12,049 INFO repeater task nb300_10pings starting 0.4 seconds late 2017-02-25 16:16:12,227 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:18:02,383 INFO repeater task nb300_lastensq starting 1.4 seconds late 2017-02-25 16:18:04,889 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:18:04,890 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:17:51 2017-02-25 16:18:34,923 INFO repeater task nb300_10pings starting 3.3 seconds late 2017-02-25 16:18:35,098 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:19:55,207 INFO repeater task ashtech_hcorrstats starting 3.5 seconds late 2017-02-25 16:19:55,939 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2017-02-25 16:20:05,952 INFO repeater task nb300_lastensq starting 5.0 seconds late 2017-02-25 16:20:08,466 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:20:08,467 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:20:08,467 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:19:51 2017-02-25 16:20:53,532 INFO repeater task nb300_10pings starting 1.9 seconds late 2017-02-25 16:20:53,712 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:22:03,812 INFO repeater task nb300_lastensq starting 2.8 seconds late 2017-02-25 16:22:06,310 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:22:06,310 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:21:51 2017-02-25 16:23:16,411 INFO repeater task nb300_10pings starting 4.7 seconds late 2017-02-25 16:23:16,586 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:24:01,653 INFO repeater task nb300_lastensq starting 0.7 seconds late 2017-02-25 16:24:04,130 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:24:04,131 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:24:04,131 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:23:51 2017-02-25 16:24:54,205 INFO repeater task ashtech_hcorrstats starting 2.5 seconds late 2017-02-25 16:24:54,965 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2017-02-25 16:25:35,022 INFO repeater task nb300_10pings starting 3.4 seconds late 2017-02-25 16:25:35,199 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:26:05,245 INFO repeater task nb300_lastensq starting 4.2 seconds late 2017-02-25 16:26:07,748 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:26:07,748 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:26:07,748 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:25:51 2017-02-25 16:27:52,886 INFO repeater task nb300_10pings starting 1.2 seconds late 2017-02-25 16:27:53,064 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:28:03,078 INFO repeater task nb300_lastensq starting 2.1 seconds late 2017-02-25 16:28:05,574 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:28:05,575 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:28:05,575 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:27:51 2017-02-25 16:29:55,726 INFO repeater task nb300_quick starting 4.1 seconds late 2017-02-25 16:29:57,972 INFO repeater task nb300_quick output: /usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice. warnings.warn("Mean of empty slice.", RuntimeWarning) 2017-02-25 16:29:57,972 INFO repeater task nb300_quick ended after 2.2 seconds 2017-02-25 16:29:57,972 INFO repeater task nb300_3dayplots starting 6.3 seconds late 2017-02-25 16:30:01,006 INFO repeater task nb300_3dayplots ended after 3.0 seconds 2017-02-25 16:30:01,007 INFO repeater task nb300_tsstats starting 9.3 seconds late 2017-02-25 16:30:01,839 INFO repeater task nb300_tsstats ended after 0.8 seconds 2017-02-25 16:30:01,840 INFO repeater task nb300_beamplots starting 10.2 seconds late 2017-02-25 16:30:03,403 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-25 16:30:03,404 INFO repeater task nb300_beamplots ended after 1.6 seconds 2017-02-25 16:30:03,404 INFO repeater task nb300_beamplots starting 11.7 seconds late 2017-02-25 16:30:04,962 INFO repeater task nb300_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-02-25 16:30:04,962 INFO repeater task nb300_beamplots ended after 1.6 seconds 2017-02-25 16:30:04,962 INFO repeater task ashtech_hcorrstats starting 13.3 seconds late 2017-02-25 16:30:05,692 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2017-02-25 16:30:10,698 INFO repeater task nb300_lastensq starting 9.7 seconds late 2017-02-25 16:30:13,200 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:30:13,200 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:30:13,201 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:29:51 2017-02-25 16:30:13,201 INFO repeater task nb300_10pings starting 1.5 seconds late 2017-02-25 16:30:13,374 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:32:03,533 INFO repeater task nb300_lastensq starting 2.5 seconds late 2017-02-25 16:32:06,011 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:32:06,011 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:32:06,011 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:31:51 2017-02-25 16:32:36,055 INFO repeater task nb300_10pings starting 4.4 seconds late 2017-02-25 16:32:36,231 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:34:01,352 INFO repeater task nb300_lastensq starting 0.4 seconds late 2017-02-25 16:34:03,837 INFO repeater task nb300_lastensq output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2017-02-25 16:34:03,837 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:34:03,837 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:33:51 2017-02-25 16:34:53,905 INFO repeater task nb300_10pings starting 2.2 seconds late 2017-02-25 16:34:54,084 INFO repeater task nb300_10pings ended after 0.2 seconds 2017-02-25 16:34:54,085 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2017-02-25 16:34:54,848 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2017-02-25 16:36:04,941 INFO repeater task nb300_lastensq starting 3.9 seconds late 2017-02-25 16:36:07,422 INFO repeater task nb300_lastensq ended after 2.5 seconds 2017-02-25 16:36:07,422 INFO repeater task nb300_lastensq: Time from /home/data/SP1704/proc/nb300/load/wrote_ens_time: 2017/02/25 16:35:51 2017-02-25 16:37:12,517 INFO repeater task nb300_10pings starting 0.8 seconds late 2017-02-25 16:37:12,696 INFO repeater task nb300_10pings ended after 0.2 seconds