2017-11-08 10:15:49,229 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 10:15:49,243 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:15:34 2017-11-08 10:18:29,541 INFO repeater task os75nb_10pings starting 1.4 seconds late 2017-11-08 10:18:29,642 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:20:09,813 INFO repeater task posmv_hcorrstats starting 1.7 seconds late 2017-11-08 10:20:10,463 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 10:20:10,463 INFO repeater task posmv_plotposmv starting 2.3 seconds late 2017-11-08 10:20:11,494 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 10:20:46,553 INFO repeater task os75nb_lastensq starting 2.6 seconds late 2017-11-08 10:20:48,194 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 10:20:48,194 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:20:34 2017-11-08 10:23:48,514 INFO repeater task os75nb_10pings starting 0.4 seconds late 2017-11-08 10:23:48,617 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:25:08,763 INFO repeater task posmv_hcorrstats starting 0.6 seconds late 2017-11-08 10:25:09,416 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 10:25:09,416 INFO repeater task posmv_plotposmv starting 1.3 seconds late 2017-11-08 10:25:10,481 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 10:25:45,544 INFO repeater task os75nb_lastensq starting 1.5 seconds late 2017-11-08 10:25:47,160 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 10:25:47,160 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:25:34 2017-11-08 10:29:12,514 INFO repeater task os75nb_10pings starting 4.4 seconds late 2017-11-08 10:29:12,622 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:30:12,723 INFO repeater task os75nb_quick starting 4.6 seconds late 2017-11-08 10:30:15,795 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 10:30:15,795 INFO repeater task os75nb_3dayplots starting 7.7 seconds late 2017-11-08 10:30:20,986 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 10:30:20,986 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 10:30:20,986 INFO repeater task os75nb_tsstats starting 12.9 seconds late 2017-11-08 10:30:21,553 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 10:30:21,554 INFO repeater task os75nb_beamplots starting 13.4 seconds late 2017-11-08 10:30:22,510 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 10:30:22,510 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 10:30:22,511 INFO repeater task os75nb_beamplots starting 14.4 seconds late 2017-11-08 10:30:23,503 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 10:30:23,503 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 10:30:23,503 INFO repeater task posmv_hcorrstats starting 15.4 seconds late 2017-11-08 10:30:24,149 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 10:30:24,149 INFO repeater task posmv_plotposmv starting 16.0 seconds late 2017-11-08 10:30:25,172 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 10:30:45,207 INFO repeater task os75nb_lastensq starting 1.2 seconds late 2017-11-08 10:30:46,847 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 10:30:46,847 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:30:34 2017-11-08 10:34:32,244 INFO repeater task os75nb_10pings starting 4.1 seconds late 2017-11-08 10:34:32,353 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:35:12,425 INFO repeater task posmv_hcorrstats starting 4.3 seconds late 2017-11-08 10:35:13,063 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 10:35:13,063 INFO repeater task posmv_plotposmv starting 4.9 seconds late 2017-11-08 10:35:14,078 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 10:35:44,130 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2017-11-08 10:35:45,843 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 10:35:45,843 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:35:34 2017-11-08 10:39:51,290 INFO repeater task os75nb_10pings starting 3.2 seconds late 2017-11-08 10:39:51,393 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:40:11,428 INFO repeater task posmv_hcorrstats starting 3.3 seconds late 2017-11-08 10:40:12,075 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 10:40:12,075 INFO repeater task posmv_plotposmv starting 3.9 seconds late 2017-11-08 10:40:13,157 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 10:40:48,219 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2017-11-08 10:40:49,887 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 10:40:49,887 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:40:34 2017-11-08 10:45:10,345 INFO repeater task os75nb_10pings starting 2.2 seconds late 2017-11-08 10:45:10,457 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:45:10,457 INFO repeater task os75nb_quick starting 2.3 seconds late 2017-11-08 10:45:13,564 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 10:45:13,565 INFO repeater task os75nb_3dayplots starting 5.4 seconds late 2017-11-08 10:45:18,821 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 10:45:18,821 INFO repeater task os75nb_3dayplots ended after 5.3 seconds 2017-11-08 10:45:18,821 INFO repeater task os75nb_tsstats starting 10.7 seconds late 2017-11-08 10:45:19,440 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 10:45:19,440 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 10:45:19,440 INFO repeater task os75nb_beamplots starting 11.3 seconds late 2017-11-08 10:45:20,399 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 10:45:20,400 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 10:45:20,400 INFO repeater task os75nb_beamplots starting 12.3 seconds late 2017-11-08 10:45:21,381 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 10:45:21,381 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 10:45:21,381 INFO repeater task posmv_hcorrstats starting 13.3 seconds late 2017-11-08 10:45:22,075 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 10:45:22,076 INFO repeater task posmv_plotposmv starting 13.9 seconds late 2017-11-08 10:45:23,095 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 10:45:48,130 INFO repeater task os75nb_lastensq starting 4.1 seconds late 2017-11-08 10:45:49,784 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 10:45:49,784 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:45:34 2017-11-08 10:50:10,254 INFO repeater task posmv_hcorrstats starting 2.1 seconds late 2017-11-08 10:50:10,965 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 10:50:10,966 INFO repeater task posmv_plotposmv starting 2.8 seconds late 2017-11-08 10:50:11,984 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 10:50:32,020 INFO repeater task os75nb_10pings starting 3.9 seconds late 2017-11-08 10:50:32,125 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 10:50:47,150 INFO repeater task os75nb_lastensq starting 3.2 seconds late 2017-11-08 10:50:48,817 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 10:50:48,817 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:50:34 2017-11-08 10:55:09,276 INFO repeater task posmv_hcorrstats starting 1.1 seconds late 2017-11-08 10:55:09,907 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 10:55:09,907 INFO repeater task posmv_plotposmv starting 1.8 seconds late 2017-11-08 10:55:10,960 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 10:55:46,020 INFO repeater task os75nb_lastensq starting 2.0 seconds late 2017-11-08 10:55:47,668 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 10:55:47,669 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 10:55:34 2017-11-08 10:55:52,674 INFO repeater task os75nb_10pings starting 4.5 seconds late 2017-11-08 10:55:52,784 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:00:08,238 INFO repeater task os75nb_quick starting 0.1 seconds late 2017-11-08 11:00:11,406 INFO repeater task os75nb_quick ended after 3.2 seconds 2017-11-08 11:00:11,407 INFO repeater task os75nb_3dayplots starting 3.3 seconds late 2017-11-08 11:00:16,630 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 11:00:16,630 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 11:00:16,630 INFO repeater task os75nb_tsstats starting 8.5 seconds late 2017-11-08 11:00:17,204 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 11:00:17,204 INFO repeater task os75nb_beamplots starting 9.1 seconds late 2017-11-08 11:00:18,171 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:00:18,172 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 11:00:18,172 INFO repeater task os75nb_beamplots starting 10.0 seconds late 2017-11-08 11:00:19,140 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:00:19,140 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 11:00:19,141 INFO repeater task posmv_hcorrstats starting 11.0 seconds late 2017-11-08 11:00:19,785 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:00:19,785 INFO repeater task posmv_plotposmv starting 11.7 seconds late 2017-11-08 11:00:20,849 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:00:45,891 INFO repeater task os75nb_lastensq starting 1.9 seconds late 2017-11-08 11:00:47,543 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:00:47,543 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:00:34 2017-11-08 11:01:12,580 INFO repeater task os75nb_10pings starting 4.5 seconds late 2017-11-08 11:01:12,691 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:05:13,127 INFO repeater task posmv_hcorrstats starting 5.0 seconds late 2017-11-08 11:05:13,826 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 11:05:13,827 INFO repeater task posmv_plotposmv starting 5.7 seconds late 2017-11-08 11:05:14,887 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:05:44,937 INFO repeater task os75nb_lastensq starting 0.9 seconds late 2017-11-08 11:05:46,547 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 11:05:46,547 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:05:34 2017-11-08 11:06:31,627 INFO repeater task os75nb_10pings starting 3.5 seconds late 2017-11-08 11:06:31,738 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:10:12,144 INFO repeater task posmv_hcorrstats starting 4.0 seconds late 2017-11-08 11:10:12,795 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 11:10:12,795 INFO repeater task posmv_plotposmv starting 4.7 seconds late 2017-11-08 11:10:13,841 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 11:10:48,898 INFO repeater task os75nb_lastensq starting 4.9 seconds late 2017-11-08 11:10:50,551 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:10:50,551 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:10:34 2017-11-08 11:11:50,660 INFO repeater task os75nb_10pings starting 2.5 seconds late 2017-11-08 11:11:50,769 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:15:11,128 INFO repeater task os75nb_quick starting 3.0 seconds late 2017-11-08 11:15:14,265 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 11:15:14,265 INFO repeater task os75nb_3dayplots starting 6.1 seconds late 2017-11-08 11:15:19,436 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 11:15:19,436 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 11:15:19,436 INFO repeater task os75nb_tsstats starting 11.3 seconds late 2017-11-08 11:15:20,016 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 11:15:20,016 INFO repeater task os75nb_beamplots starting 11.9 seconds late 2017-11-08 11:15:20,997 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:15:20,997 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 11:15:20,997 INFO repeater task os75nb_beamplots starting 12.9 seconds late 2017-11-08 11:15:21,995 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:15:21,995 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 11:15:21,995 INFO repeater task posmv_hcorrstats starting 13.9 seconds late 2017-11-08 11:15:22,640 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:15:22,641 INFO repeater task posmv_plotposmv starting 14.5 seconds late 2017-11-08 11:15:23,704 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:15:48,744 INFO repeater task os75nb_lastensq starting 4.7 seconds late 2017-11-08 11:15:50,389 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 11:15:50,390 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:15:34 2017-11-08 11:17:10,531 INFO repeater task os75nb_10pings starting 2.4 seconds late 2017-11-08 11:17:10,641 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:20:10,966 INFO repeater task posmv_hcorrstats starting 2.8 seconds late 2017-11-08 11:20:11,604 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:20:11,604 INFO repeater task posmv_plotposmv starting 3.5 seconds late 2017-11-08 11:20:12,659 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:20:47,717 INFO repeater task os75nb_lastensq starting 3.7 seconds late 2017-11-08 11:20:49,367 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:20:49,368 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:20:34 2017-11-08 11:22:29,550 INFO repeater task os75nb_10pings starting 1.4 seconds late 2017-11-08 11:22:29,667 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:25:09,946 INFO repeater task posmv_hcorrstats starting 1.8 seconds late 2017-11-08 11:25:10,615 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 11:25:10,616 INFO repeater task posmv_plotposmv starting 2.5 seconds late 2017-11-08 11:25:11,655 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 11:25:46,712 INFO repeater task os75nb_lastensq starting 2.7 seconds late 2017-11-08 11:25:48,385 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:25:48,386 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:25:34 2017-11-08 11:27:48,591 INFO repeater task os75nb_10pings starting 0.5 seconds late 2017-11-08 11:27:48,698 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:30:08,933 INFO repeater task os75nb_quick starting 0.8 seconds late 2017-11-08 11:30:12,026 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 11:30:12,027 INFO repeater task os75nb_3dayplots starting 3.9 seconds late 2017-11-08 11:30:17,161 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 11:30:17,162 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2017-11-08 11:30:17,162 INFO repeater task os75nb_tsstats starting 9.0 seconds late 2017-11-08 11:30:17,753 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 11:30:17,754 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 11:30:17,754 INFO repeater task os75nb_beamplots starting 9.6 seconds late 2017-11-08 11:30:18,696 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:30:18,696 INFO repeater task os75nb_beamplots ended after 0.9 seconds 2017-11-08 11:30:18,696 INFO repeater task os75nb_beamplots starting 10.6 seconds late 2017-11-08 11:30:19,726 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:30:19,727 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 11:30:19,727 INFO repeater task posmv_hcorrstats starting 11.6 seconds late 2017-11-08 11:30:20,374 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:30:20,374 INFO repeater task posmv_plotposmv starting 12.2 seconds late 2017-11-08 11:30:21,401 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 11:30:46,442 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2017-11-08 11:30:48,081 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 11:30:48,082 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:30:34 2017-11-08 11:33:08,320 INFO repeater task os75nb_10pings starting 0.2 seconds late 2017-11-08 11:33:08,437 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:35:08,657 INFO repeater task posmv_hcorrstats starting 0.5 seconds late 2017-11-08 11:35:09,307 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 11:35:09,308 INFO repeater task posmv_plotposmv starting 1.2 seconds late 2017-11-08 11:35:10,329 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 11:35:45,391 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2017-11-08 11:35:47,071 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:35:47,071 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:35:34 2017-11-08 11:38:32,360 INFO repeater task os75nb_10pings starting 4.2 seconds late 2017-11-08 11:38:32,469 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:40:12,650 INFO repeater task posmv_hcorrstats starting 4.5 seconds late 2017-11-08 11:40:13,290 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:40:13,290 INFO repeater task posmv_plotposmv starting 5.2 seconds late 2017-11-08 11:40:14,382 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:40:44,431 INFO repeater task os75nb_lastensq starting 0.4 seconds late 2017-11-08 11:40:46,061 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 11:40:46,061 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:40:34 2017-11-08 11:43:51,397 INFO repeater task os75nb_10pings starting 3.3 seconds late 2017-11-08 11:43:51,516 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:45:11,656 INFO repeater task os75nb_quick starting 3.5 seconds late 2017-11-08 11:45:14,828 INFO repeater task os75nb_quick ended after 3.2 seconds 2017-11-08 11:45:14,828 INFO repeater task os75nb_3dayplots starting 6.7 seconds late 2017-11-08 11:45:19,981 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 11:45:19,981 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 11:45:19,981 INFO repeater task os75nb_tsstats starting 11.9 seconds late 2017-11-08 11:45:20,553 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 11:45:20,554 INFO repeater task os75nb_beamplots starting 12.4 seconds late 2017-11-08 11:45:21,498 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:45:21,498 INFO repeater task os75nb_beamplots ended after 0.9 seconds 2017-11-08 11:45:21,498 INFO repeater task os75nb_beamplots starting 13.4 seconds late 2017-11-08 11:45:22,463 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 11:45:22,463 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 11:45:22,463 INFO repeater task posmv_hcorrstats starting 14.3 seconds late 2017-11-08 11:45:23,178 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 11:45:23,178 INFO repeater task posmv_plotposmv starting 15.0 seconds late 2017-11-08 11:45:24,203 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 11:45:44,237 INFO repeater task os75nb_lastensq starting 0.2 seconds late 2017-11-08 11:45:45,904 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:45:45,904 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:45:34 2017-11-08 11:49:11,262 INFO repeater task os75nb_10pings starting 3.1 seconds late 2017-11-08 11:49:11,376 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:50:11,478 INFO repeater task posmv_hcorrstats starting 3.3 seconds late 2017-11-08 11:50:12,116 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:50:12,117 INFO repeater task posmv_plotposmv starting 4.0 seconds late 2017-11-08 11:50:13,173 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:50:48,235 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2017-11-08 11:50:49,927 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:50:49,928 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:50:34 2017-11-08 11:54:30,311 INFO repeater task os75nb_10pings starting 2.2 seconds late 2017-11-08 11:54:30,422 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 11:55:10,498 INFO repeater task posmv_hcorrstats starting 2.4 seconds late 2017-11-08 11:55:11,133 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 11:55:11,133 INFO repeater task posmv_plotposmv starting 3.0 seconds late 2017-11-08 11:55:12,206 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 11:55:47,269 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2017-11-08 11:55:48,976 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 11:55:48,976 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 11:55:34 2017-11-08 11:59:49,402 INFO repeater task os75nb_10pings starting 1.3 seconds late 2017-11-08 11:59:49,509 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:00:09,537 INFO repeater task os75nb_quick starting 1.4 seconds late 2017-11-08 12:00:12,673 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 12:00:12,673 INFO repeater task os75nb_3dayplots starting 4.5 seconds late 2017-11-08 12:00:17,856 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 12:00:17,856 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 12:00:17,857 INFO repeater task os75nb_tsstats starting 9.7 seconds late 2017-11-08 12:00:18,420 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 12:00:18,420 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 12:00:18,421 INFO repeater task os75nb_beamplots starting 10.3 seconds late 2017-11-08 12:00:19,395 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:00:19,395 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:00:19,395 INFO repeater task os75nb_beamplots starting 11.3 seconds late 2017-11-08 12:00:20,351 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:00:20,351 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:00:20,352 INFO repeater task posmv_hcorrstats starting 12.2 seconds late 2017-11-08 12:00:20,999 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 12:00:20,999 INFO repeater task posmv_plotposmv starting 12.9 seconds late 2017-11-08 12:00:22,001 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:00:47,041 INFO repeater task os75nb_lastensq starting 3.0 seconds late 2017-11-08 12:00:48,713 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:00:48,713 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:00:34 2017-11-08 12:05:09,177 INFO repeater task os75nb_10pings starting 1.0 seconds late 2017-11-08 12:05:09,283 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:05:09,284 INFO repeater task posmv_hcorrstats starting 1.2 seconds late 2017-11-08 12:05:09,928 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 12:05:09,928 INFO repeater task posmv_plotposmv starting 1.8 seconds late 2017-11-08 12:05:10,985 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 12:05:46,042 INFO repeater task os75nb_lastensq starting 2.0 seconds late 2017-11-08 12:05:47,792 INFO repeater task os75nb_lastensq ended after 1.8 seconds 2017-11-08 12:05:47,792 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:05:34 2017-11-08 12:10:08,262 INFO repeater task posmv_hcorrstats starting 0.1 seconds late 2017-11-08 12:10:08,957 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:10:08,958 INFO repeater task posmv_plotposmv starting 0.8 seconds late 2017-11-08 12:10:09,983 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:10:30,006 INFO repeater task os75nb_10pings starting 1.9 seconds late 2017-11-08 12:10:30,109 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:10:45,133 INFO repeater task os75nb_lastensq starting 1.1 seconds late 2017-11-08 12:10:46,788 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:10:46,788 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:10:34 2017-11-08 12:15:12,244 INFO repeater task os75nb_quick starting 4.1 seconds late 2017-11-08 12:15:15,357 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 12:15:15,358 INFO repeater task os75nb_3dayplots starting 7.2 seconds late 2017-11-08 12:15:20,560 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 12:15:20,560 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 12:15:20,560 INFO repeater task os75nb_tsstats starting 12.4 seconds late 2017-11-08 12:15:21,158 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 12:15:21,158 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 12:15:21,158 INFO repeater task os75nb_beamplots starting 13.0 seconds late 2017-11-08 12:15:22,173 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:15:22,173 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:15:22,174 INFO repeater task os75nb_beamplots starting 14.0 seconds late 2017-11-08 12:15:23,140 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:15:23,140 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:15:23,140 INFO repeater task posmv_hcorrstats starting 15.0 seconds late 2017-11-08 12:15:23,806 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:15:23,806 INFO repeater task posmv_plotposmv starting 15.7 seconds late 2017-11-08 12:15:24,818 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:15:44,853 INFO repeater task os75nb_lastensq starting 0.9 seconds late 2017-11-08 12:15:46,530 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:15:46,530 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:15:34 2017-11-08 12:15:51,533 INFO repeater task os75nb_10pings starting 3.4 seconds late 2017-11-08 12:15:51,647 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:20:12,105 INFO repeater task posmv_hcorrstats starting 4.0 seconds late 2017-11-08 12:20:12,751 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 12:20:12,752 INFO repeater task posmv_plotposmv starting 4.6 seconds late 2017-11-08 12:20:13,795 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:20:48,852 INFO repeater task os75nb_lastensq starting 4.9 seconds late 2017-11-08 12:20:50,492 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 12:20:50,492 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:20:34 2017-11-08 12:21:10,523 INFO repeater task os75nb_10pings starting 2.4 seconds late 2017-11-08 12:21:10,625 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:25:11,054 INFO repeater task posmv_hcorrstats starting 2.9 seconds late 2017-11-08 12:25:11,725 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:25:11,726 INFO repeater task posmv_plotposmv starting 3.6 seconds late 2017-11-08 12:25:12,784 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 12:25:47,827 INFO repeater task os75nb_lastensq starting 3.8 seconds late 2017-11-08 12:25:49,528 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:25:49,528 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:25:34 2017-11-08 12:26:29,595 INFO repeater task os75nb_10pings starting 1.5 seconds late 2017-11-08 12:26:29,709 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:30:10,095 INFO repeater task os75nb_quick starting 2.0 seconds late 2017-11-08 12:30:13,249 INFO repeater task os75nb_quick ended after 3.2 seconds 2017-11-08 12:30:13,249 INFO repeater task os75nb_3dayplots starting 5.1 seconds late 2017-11-08 12:30:18,411 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 12:30:18,411 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 12:30:18,411 INFO repeater task os75nb_tsstats starting 10.3 seconds late 2017-11-08 12:30:18,993 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 12:30:18,993 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 12:30:18,993 INFO repeater task os75nb_beamplots starting 10.9 seconds late 2017-11-08 12:30:19,923 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:30:19,924 INFO repeater task os75nb_beamplots ended after 0.9 seconds 2017-11-08 12:30:19,924 INFO repeater task os75nb_beamplots starting 11.8 seconds late 2017-11-08 12:30:20,921 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:30:20,921 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:30:20,921 INFO repeater task posmv_hcorrstats starting 12.8 seconds late 2017-11-08 12:30:21,572 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:30:21,572 INFO repeater task posmv_plotposmv starting 13.4 seconds late 2017-11-08 12:30:22,617 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:30:47,662 INFO repeater task os75nb_lastensq starting 3.7 seconds late 2017-11-08 12:30:49,393 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:30:49,393 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:30:34 2017-11-08 12:31:49,498 INFO repeater task os75nb_10pings starting 1.4 seconds late 2017-11-08 12:31:49,606 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:35:09,940 INFO repeater task posmv_hcorrstats starting 1.8 seconds late 2017-11-08 12:35:10,604 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:35:10,604 INFO repeater task posmv_plotposmv starting 2.5 seconds late 2017-11-08 12:35:11,618 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:35:46,679 INFO repeater task os75nb_lastensq starting 2.7 seconds late 2017-11-08 12:35:48,354 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:35:48,354 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:35:34 2017-11-08 12:37:08,498 INFO repeater task os75nb_10pings starting 0.4 seconds late 2017-11-08 12:37:08,603 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:40:08,932 INFO repeater task posmv_hcorrstats starting 0.8 seconds late 2017-11-08 12:40:09,618 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:40:09,618 INFO repeater task posmv_plotposmv starting 1.5 seconds late 2017-11-08 12:40:10,640 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:40:45,680 INFO repeater task os75nb_lastensq starting 1.7 seconds late 2017-11-08 12:40:47,379 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:40:47,379 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:40:34 2017-11-08 12:42:32,554 INFO repeater task os75nb_10pings starting 4.4 seconds late 2017-11-08 12:42:32,661 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:45:12,950 INFO repeater task os75nb_quick starting 4.8 seconds late 2017-11-08 12:45:16,087 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 12:45:16,087 INFO repeater task os75nb_3dayplots starting 8.0 seconds late 2017-11-08 12:45:21,364 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 12:45:21,364 INFO repeater task os75nb_3dayplots ended after 5.3 seconds 2017-11-08 12:45:21,364 INFO repeater task os75nb_tsstats starting 13.2 seconds late 2017-11-08 12:45:21,953 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 12:45:21,953 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 12:45:21,953 INFO repeater task os75nb_beamplots starting 13.8 seconds late 2017-11-08 12:45:22,921 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:45:22,921 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:45:22,922 INFO repeater task os75nb_beamplots starting 14.8 seconds late 2017-11-08 12:45:23,944 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 12:45:23,944 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 12:45:23,944 INFO repeater task posmv_hcorrstats starting 15.8 seconds late 2017-11-08 12:45:24,591 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 12:45:24,591 INFO repeater task posmv_plotposmv starting 16.5 seconds late 2017-11-08 12:45:25,662 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 12:45:45,696 INFO repeater task os75nb_lastensq starting 1.7 seconds late 2017-11-08 12:45:47,354 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 12:45:47,354 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:45:34 2017-11-08 12:47:52,568 INFO repeater task os75nb_10pings starting 4.4 seconds late 2017-11-08 12:47:52,674 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:50:12,929 INFO repeater task posmv_hcorrstats starting 4.8 seconds late 2017-11-08 12:50:13,598 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:50:13,598 INFO repeater task posmv_plotposmv starting 5.5 seconds late 2017-11-08 12:50:14,621 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:50:44,673 INFO repeater task os75nb_lastensq starting 0.7 seconds late 2017-11-08 12:50:46,288 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 12:50:46,288 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:50:34 2017-11-08 12:53:11,553 INFO repeater task os75nb_10pings starting 3.4 seconds late 2017-11-08 12:53:11,665 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 12:55:11,887 INFO repeater task posmv_hcorrstats starting 3.8 seconds late 2017-11-08 12:55:12,542 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 12:55:12,542 INFO repeater task posmv_plotposmv starting 4.4 seconds late 2017-11-08 12:55:13,579 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 12:55:48,617 INFO repeater task os75nb_lastensq starting 4.6 seconds late 2017-11-08 12:55:50,246 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 12:55:50,247 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 12:55:34 2017-11-08 12:58:30,546 INFO repeater task os75nb_10pings starting 2.4 seconds late 2017-11-08 12:58:30,657 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:00:10,833 INFO repeater task os75nb_quick starting 2.7 seconds late 2017-11-08 13:00:13,976 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 13:00:13,976 INFO repeater task os75nb_3dayplots starting 5.8 seconds late 2017-11-08 13:00:19,096 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 13:00:19,096 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2017-11-08 13:00:19,097 INFO repeater task os75nb_tsstats starting 11.0 seconds late 2017-11-08 13:00:19,682 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 13:00:19,682 INFO repeater task os75nb_beamplots starting 11.6 seconds late 2017-11-08 13:00:20,621 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:00:20,621 INFO repeater task os75nb_beamplots ended after 0.9 seconds 2017-11-08 13:00:20,622 INFO repeater task os75nb_beamplots starting 12.5 seconds late 2017-11-08 13:00:21,586 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:00:21,586 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:00:21,586 INFO repeater task posmv_hcorrstats starting 13.5 seconds late 2017-11-08 13:00:22,224 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:00:22,225 INFO repeater task posmv_plotposmv starting 14.1 seconds late 2017-11-08 13:00:23,284 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 13:00:48,328 INFO repeater task os75nb_lastensq starting 4.3 seconds late 2017-11-08 13:00:49,987 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:00:49,988 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:00:34 2017-11-08 13:03:50,304 INFO repeater task os75nb_10pings starting 2.2 seconds late 2017-11-08 13:03:50,412 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:05:10,554 INFO repeater task posmv_hcorrstats starting 2.4 seconds late 2017-11-08 13:05:11,196 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:05:11,196 INFO repeater task posmv_plotposmv starting 3.1 seconds late 2017-11-08 13:05:12,227 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:05:47,282 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2017-11-08 13:05:48,933 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:05:48,933 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:05:34 2017-11-08 13:09:09,291 INFO repeater task os75nb_10pings starting 1.2 seconds late 2017-11-08 13:09:09,406 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:10:09,521 INFO repeater task posmv_hcorrstats starting 1.4 seconds late 2017-11-08 13:10:10,182 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 13:10:10,183 INFO repeater task posmv_plotposmv starting 2.1 seconds late 2017-11-08 13:10:11,234 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 13:10:46,297 INFO repeater task os75nb_lastensq starting 2.3 seconds late 2017-11-08 13:10:47,950 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:10:47,950 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:10:34 2017-11-08 13:14:28,354 INFO repeater task os75nb_10pings starting 0.2 seconds late 2017-11-08 13:14:28,467 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:15:08,530 INFO repeater task os75nb_quick starting 0.4 seconds late 2017-11-08 13:15:11,556 INFO repeater task os75nb_quick ended after 3.0 seconds 2017-11-08 13:15:11,556 INFO repeater task os75nb_3dayplots starting 3.4 seconds late 2017-11-08 13:15:16,788 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 13:15:16,788 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 13:15:16,788 INFO repeater task os75nb_tsstats starting 8.7 seconds late 2017-11-08 13:15:17,356 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 13:15:17,356 INFO repeater task os75nb_beamplots starting 9.2 seconds late 2017-11-08 13:15:18,390 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:15:18,391 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:15:18,391 INFO repeater task os75nb_beamplots starting 10.3 seconds late 2017-11-08 13:15:19,370 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:15:19,370 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:15:19,370 INFO repeater task posmv_hcorrstats starting 11.2 seconds late 2017-11-08 13:15:20,028 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 13:15:20,029 INFO repeater task posmv_plotposmv starting 11.9 seconds late 2017-11-08 13:15:21,071 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:15:46,115 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2017-11-08 13:15:47,781 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:15:47,781 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:15:34 2017-11-08 13:19:48,168 INFO repeater task os75nb_10pings starting 0.0 seconds late 2017-11-08 13:19:48,265 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:20:08,287 INFO repeater task posmv_hcorrstats starting 0.2 seconds late 2017-11-08 13:20:08,921 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:20:08,921 INFO repeater task posmv_plotposmv starting 0.8 seconds late 2017-11-08 13:20:09,965 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:20:45,020 INFO repeater task os75nb_lastensq starting 1.0 seconds late 2017-11-08 13:20:46,667 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 13:20:46,667 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:20:34 2017-11-08 13:25:12,121 INFO repeater task os75nb_10pings starting 4.0 seconds late 2017-11-08 13:25:12,221 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:25:12,222 INFO repeater task posmv_hcorrstats starting 4.1 seconds late 2017-11-08 13:25:12,879 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 13:25:12,879 INFO repeater task posmv_plotposmv starting 4.7 seconds late 2017-11-08 13:25:13,905 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:25:48,963 INFO repeater task os75nb_lastensq starting 5.0 seconds late 2017-11-08 13:25:50,596 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 13:25:50,597 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:25:34 2017-11-08 13:30:11,061 INFO repeater task os75nb_quick starting 2.9 seconds late 2017-11-08 13:30:14,216 INFO repeater task os75nb_quick ended after 3.2 seconds 2017-11-08 13:30:14,217 INFO repeater task os75nb_3dayplots starting 6.1 seconds late 2017-11-08 13:30:19,352 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 13:30:19,353 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2017-11-08 13:30:19,353 INFO repeater task os75nb_tsstats starting 11.2 seconds late 2017-11-08 13:30:19,921 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 13:30:19,921 INFO repeater task os75nb_beamplots starting 11.8 seconds late 2017-11-08 13:30:20,877 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:30:20,877 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:30:20,877 INFO repeater task os75nb_beamplots starting 12.7 seconds late 2017-11-08 13:30:21,884 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:30:21,884 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:30:21,885 INFO repeater task posmv_hcorrstats starting 13.8 seconds late 2017-11-08 13:30:22,521 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:30:22,521 INFO repeater task posmv_plotposmv starting 14.4 seconds late 2017-11-08 13:30:23,547 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:30:28,554 INFO repeater task os75nb_10pings starting 0.4 seconds late 2017-11-08 13:30:28,659 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:30:48,721 INFO repeater task os75nb_lastensq starting 4.7 seconds late 2017-11-08 13:30:50,340 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 13:30:50,340 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:30:34 2017-11-08 13:35:10,755 INFO repeater task posmv_hcorrstats starting 2.6 seconds late 2017-11-08 13:35:11,410 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 13:35:11,410 INFO repeater task posmv_plotposmv starting 3.3 seconds late 2017-11-08 13:35:12,442 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:35:47,503 INFO repeater task os75nb_lastensq starting 3.5 seconds late 2017-11-08 13:35:49,178 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:35:49,178 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:35:34 2017-11-08 13:35:49,178 INFO repeater task os75nb_10pings starting 1.0 seconds late 2017-11-08 13:35:49,274 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:40:09,734 INFO repeater task posmv_hcorrstats starting 1.6 seconds late 2017-11-08 13:40:10,355 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:40:10,356 INFO repeater task posmv_plotposmv starting 2.2 seconds late 2017-11-08 13:40:11,393 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:40:46,447 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2017-11-08 13:40:48,099 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:40:48,100 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:40:34 2017-11-08 13:41:08,135 INFO repeater task os75nb_10pings starting 0.0 seconds late 2017-11-08 13:41:08,242 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:45:08,663 INFO repeater task os75nb_quick starting 0.5 seconds late 2017-11-08 13:45:11,760 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 13:45:11,761 INFO repeater task os75nb_3dayplots starting 3.6 seconds late 2017-11-08 13:45:16,858 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 13:45:16,858 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2017-11-08 13:45:16,858 INFO repeater task os75nb_tsstats starting 8.7 seconds late 2017-11-08 13:45:17,426 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 13:45:17,426 INFO repeater task os75nb_beamplots starting 9.3 seconds late 2017-11-08 13:45:18,410 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:45:18,410 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:45:18,410 INFO repeater task os75nb_beamplots starting 10.3 seconds late 2017-11-08 13:45:19,383 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 13:45:19,384 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 13:45:19,384 INFO repeater task posmv_hcorrstats starting 11.3 seconds late 2017-11-08 13:45:20,030 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:45:20,030 INFO repeater task posmv_plotposmv starting 11.9 seconds late 2017-11-08 13:45:21,083 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 13:45:46,122 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2017-11-08 13:45:47,768 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 13:45:47,768 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:45:34 2017-11-08 13:46:32,844 INFO repeater task os75nb_10pings starting 4.7 seconds late 2017-11-08 13:46:32,955 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:50:08,333 INFO repeater task posmv_hcorrstats starting 0.2 seconds late 2017-11-08 13:50:08,979 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:50:08,980 INFO repeater task posmv_plotposmv starting 0.8 seconds late 2017-11-08 13:50:10,039 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 13:50:45,098 INFO repeater task os75nb_lastensq starting 1.1 seconds late 2017-11-08 13:50:46,692 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 13:50:46,692 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:50:34 2017-11-08 13:51:51,808 INFO repeater task os75nb_10pings starting 3.7 seconds late 2017-11-08 13:51:51,923 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 13:55:12,282 INFO repeater task posmv_hcorrstats starting 4.2 seconds late 2017-11-08 13:55:12,898 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 13:55:12,898 INFO repeater task posmv_plotposmv starting 4.8 seconds late 2017-11-08 13:55:13,920 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 13:55:48,983 INFO repeater task os75nb_lastensq starting 5.0 seconds late 2017-11-08 13:55:50,639 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 13:55:50,640 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 13:55:34 2017-11-08 13:57:10,781 INFO repeater task os75nb_10pings starting 2.7 seconds late 2017-11-08 13:57:10,899 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:00:11,218 INFO repeater task os75nb_quick starting 3.1 seconds late 2017-11-08 14:00:14,302 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 14:00:14,302 INFO repeater task os75nb_3dayplots starting 6.2 seconds late 2017-11-08 14:00:19,440 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 14:00:19,440 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2017-11-08 14:00:19,441 INFO repeater task os75nb_tsstats starting 11.3 seconds late 2017-11-08 14:00:20,031 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 14:00:20,032 INFO repeater task os75nb_beamplots starting 11.9 seconds late 2017-11-08 14:00:20,993 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 14:00:20,993 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 14:00:20,994 INFO repeater task os75nb_beamplots starting 12.9 seconds late 2017-11-08 14:00:21,978 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 14:00:21,978 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 14:00:21,978 INFO repeater task posmv_hcorrstats starting 13.8 seconds late 2017-11-08 14:00:22,637 INFO repeater task posmv_hcorrstats ended after 0.7 seconds 2017-11-08 14:00:22,638 INFO repeater task posmv_plotposmv starting 14.5 seconds late 2017-11-08 14:00:23,641 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:00:48,679 INFO repeater task os75nb_lastensq starting 4.7 seconds late 2017-11-08 14:00:50,328 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 14:00:50,328 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:00:34 2017-11-08 14:02:30,506 INFO repeater task os75nb_10pings starting 2.4 seconds late 2017-11-08 14:02:30,613 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:05:10,885 INFO repeater task posmv_hcorrstats starting 2.8 seconds late 2017-11-08 14:05:11,509 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:05:11,509 INFO repeater task posmv_plotposmv starting 3.4 seconds late 2017-11-08 14:05:12,522 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:05:47,580 INFO repeater task os75nb_lastensq starting 3.6 seconds late 2017-11-08 14:05:49,201 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 14:05:49,201 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:05:34 2017-11-08 14:07:49,407 INFO repeater task os75nb_10pings starting 1.3 seconds late 2017-11-08 14:07:49,514 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:10:09,765 INFO repeater task posmv_hcorrstats starting 1.6 seconds late 2017-11-08 14:10:10,411 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:10:10,411 INFO repeater task posmv_plotposmv starting 2.3 seconds late 2017-11-08 14:10:11,444 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:10:46,506 INFO repeater task os75nb_lastensq starting 2.5 seconds late 2017-11-08 14:10:48,227 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 14:10:48,227 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:10:34 2017-11-08 14:13:08,470 INFO repeater task os75nb_10pings starting 0.3 seconds late 2017-11-08 14:13:08,575 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:15:08,798 INFO repeater task os75nb_quick starting 0.7 seconds late 2017-11-08 14:15:11,993 INFO repeater task os75nb_quick ended after 3.2 seconds 2017-11-08 14:15:11,993 INFO repeater task os75nb_3dayplots starting 3.9 seconds late 2017-11-08 14:15:17,219 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 14:15:17,219 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 14:15:17,219 INFO repeater task os75nb_tsstats starting 9.1 seconds late 2017-11-08 14:15:17,784 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 14:15:17,785 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 14:15:17,785 INFO repeater task os75nb_beamplots starting 9.7 seconds late 2017-11-08 14:15:18,766 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 14:15:18,766 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 14:15:18,767 INFO repeater task os75nb_beamplots starting 10.6 seconds late 2017-11-08 14:15:19,745 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 14:15:19,746 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 14:15:19,746 INFO repeater task posmv_hcorrstats starting 11.6 seconds late 2017-11-08 14:15:20,373 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:15:20,373 INFO repeater task posmv_plotposmv starting 12.2 seconds late 2017-11-08 14:15:21,392 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:15:46,434 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2017-11-08 14:15:48,082 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 14:15:48,083 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:15:34 2017-11-08 14:18:28,362 INFO repeater task os75nb_10pings starting 0.2 seconds late 2017-11-08 14:18:28,475 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:20:08,659 INFO repeater task posmv_hcorrstats starting 0.5 seconds late 2017-11-08 14:20:09,284 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:20:09,285 INFO repeater task posmv_plotposmv starting 1.2 seconds late 2017-11-08 14:20:10,345 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 14:20:45,409 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2017-11-08 14:20:47,084 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 14:20:47,084 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:20:34 2017-11-08 14:23:52,412 INFO repeater task os75nb_10pings starting 4.3 seconds late 2017-11-08 14:23:52,523 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:25:12,666 INFO repeater task posmv_hcorrstats starting 4.5 seconds late 2017-11-08 14:25:13,294 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:25:13,294 INFO repeater task posmv_plotposmv starting 5.2 seconds late 2017-11-08 14:25:14,313 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:25:44,356 INFO repeater task os75nb_lastensq starting 0.4 seconds late 2017-11-08 14:25:46,007 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 14:25:46,007 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:25:34 2017-11-08 14:29:11,362 INFO repeater task os75nb_10pings starting 3.2 seconds late 2017-11-08 14:29:11,475 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:30:11,585 INFO repeater task os75nb_quick starting 3.5 seconds late 2017-11-08 14:30:14,756 INFO repeater task os75nb_quick ended after 3.2 seconds 2017-11-08 14:30:14,757 INFO repeater task os75nb_3dayplots starting 6.6 seconds late 2017-11-08 14:30:19,976 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 14:30:19,976 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 14:30:19,977 INFO repeater task os75nb_tsstats starting 11.8 seconds late 2017-11-08 14:30:20,545 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2017-11-08 14:30:20,545 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2017-11-08 14:30:20,546 INFO repeater task os75nb_beamplots starting 12.4 seconds late 2017-11-08 14:30:21,543 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 14:30:21,543 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 14:30:21,543 INFO repeater task os75nb_beamplots starting 13.4 seconds late 2017-11-08 14:30:22,502 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-11-08 14:30:22,502 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2017-11-08 14:30:22,502 INFO repeater task posmv_hcorrstats starting 14.4 seconds late 2017-11-08 14:30:23,148 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:30:23,148 INFO repeater task posmv_plotposmv starting 15.0 seconds late 2017-11-08 14:30:24,169 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:30:44,200 INFO repeater task os75nb_lastensq starting 0.2 seconds late 2017-11-08 14:30:45,846 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 14:30:45,846 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:30:34 2017-11-08 14:34:31,234 INFO repeater task os75nb_10pings starting 3.1 seconds late 2017-11-08 14:34:31,341 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:35:11,406 INFO repeater task posmv_hcorrstats starting 3.3 seconds late 2017-11-08 14:35:12,033 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:35:12,033 INFO repeater task posmv_plotposmv starting 3.9 seconds late 2017-11-08 14:35:13,123 INFO repeater task posmv_plotposmv ended after 1.1 seconds 2017-11-08 14:35:48,183 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2017-11-08 14:35:49,890 INFO repeater task os75nb_lastensq ended after 1.7 seconds 2017-11-08 14:35:49,891 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:35:34 2017-11-08 14:39:50,305 INFO repeater task os75nb_10pings starting 2.2 seconds late 2017-11-08 14:39:50,410 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:40:10,447 INFO repeater task posmv_hcorrstats starting 2.3 seconds late 2017-11-08 14:40:11,055 INFO repeater task posmv_hcorrstats ended after 0.6 seconds 2017-11-08 14:40:11,055 INFO repeater task posmv_plotposmv starting 2.9 seconds late 2017-11-08 14:40:12,078 INFO repeater task posmv_plotposmv ended after 1.0 seconds 2017-11-08 14:40:47,137 INFO repeater task os75nb_lastensq starting 3.1 seconds late 2017-11-08 14:40:48,786 INFO repeater task os75nb_lastensq ended after 1.6 seconds 2017-11-08 14:40:48,786 INFO repeater task os75nb_lastensq: Time from /home/data/SH1710/proc/os75nb/load/wrote_ens_time: 2017/11/08 14:40:34 2017-11-08 14:45:09,228 INFO repeater task os75nb_10pings starting 1.1 seconds late 2017-11-08 14:45:09,334 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-08 14:45:09,334 INFO repeater task os75nb_quick starting 1.2 seconds late 2017-11-08 14:45:12,466 INFO repeater task os75nb_quick ended after 3.1 seconds 2017-11-08 14:45:12,466 INFO repeater task os75nb_3dayplots starting 4.3 seconds late 2017-11-08 14:45:17,342 INFO Translate translate exiting, hdg /home/data/SH1710/rbin/gyro 2017-11-08 14:45:17,356 INFO Translate translate exiting, gps /home/data/SH1710/rbin/posmv 2017-11-08 14:45:17,361 INFO Translate translate exiting, gps /home/data/SH1710/rbin/gpsnav 2017-11-08 14:45:17,367 INFO Translate translate exiting, pmv /home/data/SH1710/rbin/posmv 2017-11-08 14:45:17,626 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-11-08 14:45:17,626 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2017-11-08 14:45:22,631 INFO root cleanup_figs not_logging 2017-11-08 14:45:22,643 INFO root Exiting DAS_while_logging.py 2017-11-11 17:34:05,596 INFO root Starting DAS_while_logging.py 2017-11-11 17:34:05,651 INFO root Cruiseid: SH1711 2017-11-11 17:34:05,667 INFO root starting run method 2017-11-11 17:34:05,668 INFO root speedlog_stopflagfile: /home/adcp/flags/DAS_speedlog.stop 2017-11-11 17:34:05,668 INFO root speedlog_runflagfile: /home/adcp/flags/DAS_speedlog.running 2017-11-11 17:34:05,669 INFO root speedlog command is /usr/local/currents/bin/DAS_speedlog.py & 2017-11-11 17:34:05,669 INFO Translate gyro hdg 1 2017-11-11 17:34:05,669 INFO Translate 2017-11-11 17:34:05,669 INFO Translate gpsnav gps 1 2017-11-11 17:34:05,669 INFO Translate 2017-11-11 17:34:05,670 INFO Translate posmv gps 1 2017-11-11 17:34:05,670 INFO Translate 2017-11-11 17:34:05,670 INFO Translate posmv pmv 0 2017-11-11 17:34:05,670 INFO Translate 2017-11-11 17:34:05,672 INFO root active_procdirnames: ['os75nb'] 2017-11-11 17:34:05,672 INFO repeater PeriodicSystem: os75nb_lastensq interval is 310 seconds cd /home/adcp/uhdas_tmp; run_lastensq.py -d os75nb --update_pygbin --averages --plotens_mpl --kt_vecprof --pingwarnN 30 --ktsdir 2017-11-11 17:34:05,673 INFO repeater PeriodicSystem: os75nb_10pings interval is 320 seconds cd /home/adcp/uhdas_tmp; run_grab_lastpings.py -d os75nb 2017-11-11 17:34:05,673 INFO repeater PeriodicSystem: os75nb_quick interval is 900 seconds cd /home/adcp/uhdas_tmp; run_quick.py -d os75nb 2017-11-11 17:34:05,674 INFO repeater PeriodicSystem: os75nb_3dayplots interval is 900 seconds cd /home/adcp/uhdas_tmp; run_3dayplots_mpl.py -d os75nb 2017-11-11 17:34:05,675 INFO repeater PeriodicSystem: os75nb_tsstats interval is 900 seconds cd /home/adcp/uhdas_tmp; run_plot_tsstats.py os75nb 2017-11-11 17:34:05,675 INFO repeater PeriodicSystem: os75nb_beamplots interval is 900 seconds cd /home/adcp/uhdas_tmp; plot_beamdiagnostics.py -d os75nb --savefigs --webcopy -p scattering 2017-11-11 17:34:05,675 INFO repeater PeriodicSystem: os75nb_beamplots interval is 900 seconds cd /home/adcp/uhdas_tmp; plot_beamdiagnostics.py -d os75nb --savefigs --webcopy -p velocity 2017-11-11 17:34:05,681 INFO repeater PeriodicSystem: posmv_hcorrstats interval is 300 seconds cd /home/adcp/uhdas_tmp; run_hbinhcorrstats.py --hcorr_inst posmv --plotdh --printstats 2017-11-11 17:34:05,681 INFO repeater PeriodicSystem: posmv_plotposmv interval is 300 seconds cd /home/adcp/uhdas_tmp; run_plotposmv.py -n25 -d posmv 2017-11-11 17:34:05,681 INFO root cleanup_figs no_figure 2017-11-11 17:39:06,209 INFO repeater task posmv_hcorrstats starting 0.5 seconds late 2017-11-11 17:39:06,396 INFO repeater task posmv_hcorrstats ended after 0.2 seconds 2017-11-11 17:39:06,396 INFO repeater task posmv_plotposmv starting 0.7 seconds late 2017-11-11 17:39:06,790 INFO repeater task posmv_plotposmv ended after 0.4 seconds 2017-11-11 17:39:16,803 INFO repeater task os75nb_lastensq starting 1.1 seconds late 2017-11-11 17:39:17,214 ERROR repeater task os75nb_lastensq returncode 16 2017-11-11 17:39:17,214 INFO repeater task os75nb_lastensq output: Pingavg.read_inputs: no data available? Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/pingavg.py", line 636, in run self.read_inputs() File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/pingavg.py", line 601, in read_inputs yearbase=self.yearbase) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1658, in __init__ if os.path.exists(_log_fnames[0]): IndexError: list index out of range In body of run_lastensq.py Traceback (most recent call last): File "/usr/local/currents/bin/run_lastensq.py", line 336, in cdict = {'beamangle': pavg.beam_angle, AttributeError: 'Pingavg' object has no attribute 'beam_angle' 2017-11-11 17:39:17,214 INFO repeater task os75nb_lastensq ended after 0.4 seconds 2017-11-11 17:39:17,214 WARNING repeater task os75nb_lastensq: Could not stat or read /home/data/SH1711/proc/os75nb/load/wrote_ens_time 2017-11-11 17:39:27,231 INFO repeater task os75nb_10pings starting 1.6 seconds late 2017-11-11 17:39:27,337 ERROR repeater task os75nb_10pings returncode 8 2017-11-11 17:39:27,337 INFO repeater task os75nb_10pings output: In body of run_grab_lastpings.py Traceback (most recent call last): File "/usr/local/currents/bin/run_grab_lastpings.py", line 69, in data=extract_raw(infile, instname[:2], i0, i1, outfile=outfile, verbose=True) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1479, in extract_raw rf.nprofs, rf.header.nbytes)) TypeError: %d format: a number is required, not NoneType 2017-11-11 17:39:27,337 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-11 17:44:07,855 INFO repeater task posmv_hcorrstats starting 2.2 seconds late 2017-11-11 17:44:08,255 INFO repeater task posmv_hcorrstats ended after 0.4 seconds 2017-11-11 17:44:08,255 INFO repeater task posmv_plotposmv starting 2.6 seconds late 2017-11-11 17:44:08,640 INFO repeater task posmv_plotposmv ended after 0.4 seconds 2017-11-11 17:44:28,673 INFO repeater task os75nb_lastensq starting 3.0 seconds late 2017-11-11 17:44:29,001 ERROR repeater task os75nb_lastensq returncode 16 2017-11-11 17:44:29,002 INFO repeater task os75nb_lastensq output: Pingavg.read_inputs: no data available? Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/pingavg.py", line 636, in run self.read_inputs() File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/pingavg.py", line 601, in read_inputs yearbase=self.yearbase) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1658, in __init__ if os.path.exists(_log_fnames[0]): IndexError: list index out of range In body of run_lastensq.py Traceback (most recent call last): File "/usr/local/currents/bin/run_lastensq.py", line 336, in cdict = {'beamangle': pavg.beam_angle, AttributeError: 'Pingavg' object has no attribute 'beam_angle' 2017-11-11 17:44:29,002 INFO repeater task os75nb_lastensq ended after 0.3 seconds 2017-11-11 17:44:29,002 WARNING repeater task os75nb_lastensq: Could not stat or read /home/data/SH1711/proc/os75nb/load/wrote_ens_time 2017-11-11 17:44:49,034 INFO repeater task os75nb_10pings starting 3.4 seconds late 2017-11-11 17:44:49,143 ERROR repeater task os75nb_10pings returncode 8 2017-11-11 17:44:49,143 INFO repeater task os75nb_10pings output: In body of run_grab_lastpings.py Traceback (most recent call last): File "/usr/local/currents/bin/run_grab_lastpings.py", line 69, in data=extract_raw(infile, instname[:2], i0, i1, outfile=outfile, verbose=True) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1479, in extract_raw rf.nprofs, rf.header.nbytes)) TypeError: %d format: a number is required, not NoneType 2017-11-11 17:44:49,143 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-11 17:49:09,649 INFO repeater task os75nb_quick starting 4.0 seconds late 2017-11-11 17:49:09,920 ERROR repeater task os75nb_quick returncode 2 2017-11-11 17:49:09,920 INFO repeater task os75nb_quick output: Failure in: scanping Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_npy.py", line 996, in run_calc Calc() File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_npy.py", line 283, in __call__ gbinsonar= uhdas_cfg.gbinsonar) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1658, in __init__ if os.path.exists(_log_fnames[0]): IndexError: list index out of range cannot run numpy scan Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_adcp.py", line 1100, in quick_adcp proc.run_scanping_npy() #UHDAS data File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_npy.py", line 1014, in run_scanping_npy self.run_calc('scanping', **kw) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_npy.py", line 999, in run_calc raise quickFatalError('Q_npy.runcalc: %s failure ', calcname) quickFatalError: ('Q_npy.runcalc: %s failure ', 'scanping') running quick_adcp Traceback (most recent call last): File "/usr/local/currents/bin/run_quick.py", line 169, in quick_adcp(['--cntfile', cntfile_proc]) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_adcp.py", line 1115, in quick_adcp trddlist = proc.get_time_range(timefile, time_range_string) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_subs.py", line 147, in get_time_range TF = open(timefile,'r') IOError: [Errno 2] No such file or directory: 'scan/a_sh.scn' 2017-11-11 17:49:09,920 INFO repeater task os75nb_quick ended after 0.3 seconds 2017-11-11 17:49:09,920 INFO repeater task os75nb_3dayplots starting 4.2 seconds late 2017-11-11 17:49:10,161 ERROR repeater task os75nb_3dayplots returncode 1 2017-11-11 17:49:10,161 INFO repeater task os75nb_3dayplots output: cannot write codasdb data samples for daily_report Traceback (most recent call last): File "/usr/local/currents/bin/run_3dayplots_mpl.py", line 333, in save_data_sample_db(cid, '_cont', cruiseinfo.daily_dir, -1.2) File "/usr/local/currents/bin/run_3dayplots_mpl.py", line 133, in save_data_sample_db dinfo = get_dbinfo(cid, ndays) File "/usr/local/currents/bin/run_3dayplots_mpl.py", line 202, in get_dbinfo cid['top_plotbin'][cid['procdirname']]) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_codas.py", line 16, in binmaxdepth db = DB(dbname) File "_codas.pyx", line 454, in _codas.DB.__init__ (codas/src/_codas.c:5634) IOError: Can't open database /home/data/SH1711/proc/os75nb/adcpdb/a_sh CODAS ERROR: UNABLE_TO_OPEN ACCESSING: BLOCK_DIR_FILE FROM FUNCTION: open_block_dir IN DATABASE: 1 CODAS ERROR: UNABLE_TO_OPEN ACCESSING: BLOCK_DIR_FILE FROM FUNCTION: DBOPEN IN DATABASE: 1 CODAS ERROR: UNABLE_TO_OPEN ACCESSING: BLOCK_DIR_FILE FROM FUNCTION: open_block_dir IN DATABASE: 1 CODAS ERROR: UNABLE_TO_OPEN ACCESSING: BLOCK_DIR_FILE FROM FUNCTION: DBOPEN IN DATABASE: 1 Traceback (most recent call last): File "/usr/local/currents/bin/run_3dayplots_mpl.py", line 340, in dinfo = get_dbinfo(cid, ndays=-1.5) File "/usr/local/currents/bin/run_3dayplots_mpl.py", line 202, in get_dbinfo cid['top_plotbin'][cid['procdirname']]) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/quick_codas.py", line 16, in binmaxdepth db = DB(dbname) File "_codas.pyx", line 454, in _codas.DB.__init__ (codas/src/_codas.c:5634) IOError: Can't open database /home/data/SH1711/proc/os75nb/adcpdb/a_sh 2017-11-11 17:49:10,161 INFO repeater task os75nb_3dayplots ended after 0.2 seconds 2017-11-11 17:49:10,161 INFO repeater task os75nb_tsstats starting 4.5 seconds late 2017-11-11 17:49:10,337 ERROR repeater task os75nb_tsstats returncode 1 2017-11-11 17:49:10,337 INFO repeater task os75nb_tsstats output: Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_tsstats.py", line 97, in data = get_data(dbname, ndays=ndays) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/tseries_diffstats.py", line 46, in get_data alldata=get_profiles(dbname) File "/usr/local/lib/python2.7/dist-packages/pycurrents/codas/tools.py", line 262, in get_profiles dbname = dbname_from_path(dbname) File "/usr/local/lib/python2.7/dist-packages/pycurrents/codas/tools.py", line 229, in dbname_from_path raise ValueError("path %s has no block directory" % path) ValueError: path /home/data/SH1711/proc/os75nb/adcpdb/a_sh has no block directory 2017-11-11 17:49:10,337 INFO repeater task os75nb_tsstats ended after 0.2 seconds 2017-11-11 17:49:10,337 INFO repeater task os75nb_beamplots starting 4.7 seconds late 2017-11-11 17:49:10,574 ERROR repeater task os75nb_beamplots returncode 1 2017-11-11 17:49:10,574 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin Traceback (most recent call last): File "/usr/local/currents/bin/plot_beamdiagnostics.py", line 125, in savefigs=options.savefigs) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/adcp_diagnostics.py", line 298, in get_plot_recent data = self.read_data(minutes=minutes) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/adcp_diagnostics.py", line 115, in read_data self.m=Multiread(self.lastfiles, self.sonar.model) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1658, in __init__ if os.path.exists(_log_fnames[0]): IndexError: list index out of range 2017-11-11 17:49:10,575 INFO repeater task os75nb_beamplots ended after 0.2 seconds 2017-11-11 17:49:10,575 INFO repeater task os75nb_beamplots starting 4.9 seconds late 2017-11-11 17:49:10,811 ERROR repeater task os75nb_beamplots returncode 1 2017-11-11 17:49:10,811 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin Traceback (most recent call last): File "/usr/local/currents/bin/plot_beamdiagnostics.py", line 125, in savefigs=options.savefigs) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/adcp_diagnostics.py", line 298, in get_plot_recent data = self.read_data(minutes=minutes) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/adcp_diagnostics.py", line 115, in read_data self.m=Multiread(self.lastfiles, self.sonar.model) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1658, in __init__ if os.path.exists(_log_fnames[0]): IndexError: list index out of range 2017-11-11 17:49:10,811 INFO repeater task os75nb_beamplots ended after 0.2 seconds 2017-11-11 17:49:10,812 INFO repeater task posmv_hcorrstats starting 5.1 seconds late 2017-11-11 17:49:11,212 INFO repeater task posmv_hcorrstats ended after 0.4 seconds 2017-11-11 17:49:11,213 INFO repeater task posmv_plotposmv starting 5.5 seconds late 2017-11-11 17:49:11,594 INFO repeater task posmv_plotposmv ended after 0.4 seconds 2017-11-11 17:49:36,634 INFO repeater task os75nb_lastensq starting 1.0 seconds late 2017-11-11 17:49:36,961 ERROR repeater task os75nb_lastensq returncode 16 2017-11-11 17:49:36,961 INFO repeater task os75nb_lastensq output: Pingavg.read_inputs: no data available? Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/pingavg.py", line 636, in run self.read_inputs() File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/pingavg.py", line 601, in read_inputs yearbase=self.yearbase) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1658, in __init__ if os.path.exists(_log_fnames[0]): IndexError: list index out of range In body of run_lastensq.py Traceback (most recent call last): File "/usr/local/currents/bin/run_lastensq.py", line 336, in cdict = {'beamangle': pavg.beam_angle, AttributeError: 'Pingavg' object has no attribute 'beam_angle' 2017-11-11 17:49:36,961 INFO repeater task os75nb_lastensq ended after 0.3 seconds 2017-11-11 17:49:36,961 WARNING repeater task os75nb_lastensq: Could not stat or read /home/data/SH1711/proc/os75nb/load/wrote_ens_time 2017-11-11 17:50:07,010 INFO repeater task os75nb_10pings starting 1.3 seconds late 2017-11-11 17:50:07,113 ERROR repeater task os75nb_10pings returncode 8 2017-11-11 17:50:07,113 INFO repeater task os75nb_10pings output: In body of run_grab_lastpings.py Traceback (most recent call last): File "/usr/local/currents/bin/run_grab_lastpings.py", line 69, in data=extract_raw(infile, instname[:2], i0, i1, outfile=outfile, verbose=True) File "/usr/local/lib/python2.7/dist-packages/pycurrents/adcp/rdiraw.py", line 1479, in extract_raw rf.nprofs, rf.header.nbytes)) TypeError: %d format: a number is required, not NoneType 2017-11-11 17:50:07,113 INFO repeater task os75nb_10pings ended after 0.1 seconds 2017-11-11 17:50:19,942 INFO Translate translate exiting, gps /home/data/SH1711/rbin/posmv 2017-11-11 17:50:19,942 INFO Translate translate exiting, hdg /home/data/SH1711/rbin/gyro 2017-11-11 17:50:19,956 INFO Translate translate exiting, gps /home/data/SH1711/rbin/gpsnav 2017-11-11 17:50:19,960 INFO Translate translate exiting, pmv /home/data/SH1711/rbin/posmv 2017-11-11 17:50:22,145 INFO root cleanup_figs not_logging 2017-11-11 17:50:22,204 INFO root Exiting DAS_while_logging.py 2017-11-11 17:50:38,471 INFO root Starting DAS_while_logging.py 2017-11-11 17:50:38,481 INFO root Cruiseid: SH1711 2017-11-11 17:50:38,492 INFO root starting run method 2017-11-11 17:50:38,493 INFO root speedlog_stopflagfile: /home/adcp/flags/DAS_speedlog.stop 2017-11-11 17:50:38,493 INFO root speedlog_runflagfile: /home/adcp/flags/DAS_speedlog.running 2017-11-11 17:50:38,493 INFO root speedlog command is /usr/local/currents/bin/DAS_speedlog.py & 2017-11-11 17:50:38,494 INFO Translate gyro hdg 1 2017-11-11 17:50:38,494 INFO Translate 2017-11-11 17:50:38,494 INFO Translate gpsnav gps 1 2017-11-11 17:50:38,494 INFO Translate 2017-11-11 17:50:38,494 INFO Translate posmv gps 1 2017-11-11 17:50:38,494 INFO Translate 2017-11-11 17:50:38,494 INFO Translate posmv pmv 0 2017-11-11 17:50:38,494 INFO Translate 2017-11-11 17:50:38,495 INFO root active_procdirnames: ['os75nb'] 2017-11-11 17:50:38,496 INFO repeater PeriodicSystem: os75nb_lastensq interval is 310 seconds cd /home/adcp/uhdas_tmp; run_lastensq.py -d os75nb --update_pygbin --averages --plotens_mpl --kt_vecprof --pingwarnN 30 --ktsdir 2017-11-11 17:50:38,496 INFO repeater PeriodicSystem: os75nb_10pings interval is 320 seconds cd /home/adcp/uhdas_tmp; run_grab_lastpings.py -d os75nb 2017-11-11 17:50:38,496 INFO repeater PeriodicSystem: os75nb_quick interval is 900 seconds cd /home/adcp/uhdas_tmp; run_quick.py -d os75nb 2017-11-11 17:50:38,496 INFO repeater PeriodicSystem: os75nb_3dayplots interval is 900 seconds cd /home/adcp/uhdas_tmp; run_3dayplots_mpl.py -d os75nb 2017-11-11 17:50:38,497 INFO repeater PeriodicSystem: os75nb_tsstats interval is 900 seconds cd /home/adcp/uhdas_tmp; run_plot_tsstats.py os75nb 2017-11-11 17:50:38,497 INFO repeater PeriodicSystem: os75nb_beamplots interval is 900 seconds cd /home/adcp/uhdas_tmp; plot_beamdiagnostics.py -d os75nb --savefigs --webcopy -p scattering 2017-11-11 17:50:38,497 INFO repeater PeriodicSystem: os75nb_beamplots interval is 900 seconds cd /home/adcp/uhdas_tmp; plot_beamdiagnostics.py -d os75nb --savefigs --webcopy -p velocity 2017-11-11 17:50:38,497 INFO repeater PeriodicSystem: posmv_hcorrstats interval is 300 seconds cd /home/adcp/uhdas_tmp; run_hbinhcorrstats.py --hcorr_inst posmv --plotdh --printstats 2017-11-11 17:50:38,497 INFO repeater PeriodicSystem: posmv_plotposmv interval is 300 seconds cd /home/adcp/uhdas_tmp; run_plotposmv.py -n25 -d posmv 2017-11-11 17:50:38,497 INFO root cleanup_figs no_figure 2017-11-11 17:51:07,958 INFO Translate translate exiting, pmv /home/data/SH1711/rbin/posmv 2017-11-11 17:51:07,959 INFO Translate translate exiting, hdg /home/data/SH1711/rbin/gyro 2017-11-11 17:51:07,959 INFO Translate translate exiting, gps /home/data/SH1711/rbin/posmv 2017-11-11 17:51:07,970 INFO Translate translate exiting, gps /home/data/SH1711/rbin/gpsnav 2017-11-11 17:51:08,541 INFO root cleanup_figs not_logging 2017-11-11 17:51:08,549 INFO root Exiting DAS_while_logging.py 2017-11-11 19:21:59,126 INFO root Starting DAS_while_logging.py 2017-11-11 19:21:59,136 INFO root Cruiseid: SH1711 2017-11-11 19:21:59,152 INFO root starting run method 2017-11-11 19:21:59,152 INFO root speedlog_stopflagfile: /home/adcp/flags/DAS_speedlog.stop 2017-11-11 19:21:59,152 INFO root speedlog_runflagfile: /home/adcp/flags/DAS_speedlog.running 2017-11-11 19:21:59,153 INFO root speedlog command is /usr/local/currents/bin/DAS_speedlog.py & 2017-11-11 19:21:59,154 INFO Translate gyro hdg 1 2017-11-11 19:21:59,154 INFO Translate 2017-11-11 19:21:59,154 INFO Translate gpsnav gps 1 2017-11-11 19:21:59,154 INFO Translate 2017-11-11 19:21:59,154 INFO Translate posmv gps 1 2017-11-11 19:21:59,154 INFO Translate 2017-11-11 19:21:59,154 INFO Translate posmv pmv 0 2017-11-11 19:21:59,154 INFO Translate 2017-11-11 19:21:59,156 INFO root active_procdirnames: ['os75nb'] 2017-11-11 19:21:59,156 INFO repeater PeriodicSystem: os75nb_lastensq interval is 310 seconds cd /home/adcp/uhdas_tmp; run_lastensq.py -d os75nb --update_pygbin --averages --plotens_mpl --kt_vecprof --pingwarnN 30 --ktsdir 2017-11-11 19:21:59,157 INFO repeater PeriodicSystem: os75nb_10pings interval is 320 seconds cd /home/adcp/uhdas_tmp; run_grab_lastpings.py -d os75nb 2017-11-11 19:21:59,157 INFO repeater PeriodicSystem: os75nb_quick interval is 900 seconds cd /home/adcp/uhdas_tmp; run_quick.py -d os75nb