2018-09-27 12:28:12,114 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:27:59 2018-09-27 12:28:12,115 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:30:04 2018-09-27 12:30:07,248 INFO repeater task wh300_lastensq starting 3.2 seconds late 2018-09-27 12:30:09,365 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:30:09,365 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:29:59 2018-09-27 12:30:09,365 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:32:04 2018-09-27 12:30:09,365 INFO repeater task wh300_quick starting 5.4 seconds late 2018-09-27 12:30:13,055 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:30:13,056 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:29:59 2018-09-27 12:30:13,056 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:32:04 2018-09-27 12:30:13,056 INFO repeater task wh300_lastvec starting 9.1 seconds late 2018-09-27 12:30:13,412 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:30:13,412 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:29:59 2018-09-27 12:30:13,413 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:32:04 2018-09-27 12:30:23,420 INFO repeater task wh300_3dayplots starting 2.4 seconds late 2018-09-27 12:30:28,678 INFO repeater task wh300_3dayplots ended after 5.3 seconds 2018-09-27 12:30:28,678 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 12:35:21 2018-09-27 12:32:08,802 INFO repeater task wh300_lastensq starting 4.8 seconds late 2018-09-27 12:32:10,959 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-27 12:32:10,959 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:31:59 2018-09-27 12:32:10,959 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:34:04 2018-09-27 12:32:10,959 INFO repeater task wh300_quick starting 7.0 seconds late 2018-09-27 12:32:14,663 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:32:14,663 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:31:59 2018-09-27 12:32:14,664 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:34:04 2018-09-27 12:32:14,664 INFO repeater task wh300_lastvec starting 10.7 seconds late 2018-09-27 12:32:15,034 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:32:15,035 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:31:59 2018-09-27 12:32:15,035 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:34:04 2018-09-27 12:34:05,177 INFO repeater task wh300_lastensq starting 1.2 seconds late 2018-09-27 12:34:07,398 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-27 12:34:07,398 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:33:59 2018-09-27 12:34:07,398 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:36:04 2018-09-27 12:34:07,399 INFO repeater task wh300_quick starting 3.4 seconds late 2018-09-27 12:34:11,105 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:34:11,106 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:33:59 2018-09-27 12:34:11,106 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:36:04 2018-09-27 12:34:11,106 INFO repeater task wh300_lastvec starting 7.1 seconds late 2018-09-27 12:34:11,483 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:34:11,483 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:33:59 2018-09-27 12:34:11,483 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:36:04 2018-09-27 12:35:16,556 INFO repeater task wh300_tsstats starting 0.5 seconds late 2018-09-27 12:35:17,194 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-27 12:35:17,194 INFO repeater task wh300_tsstats next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:17,194 INFO repeater task wh300_10pings starting 1.2 seconds late 2018-09-27 12:35:17,300 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-27 12:35:17,300 INFO repeater task wh300_10pings next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:17,301 INFO repeater task wh300_beamplots starting 1.3 seconds late 2018-09-27 12:35:18,345 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 12:35:18,345 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:18,345 INFO repeater task wh300_beamplots starting 2.3 seconds late 2018-09-27 12:35:19,393 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 12:35:19,394 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:19,394 INFO repeater task adu5_hcorrstats starting 3.2 seconds late 2018-09-27 12:35:20,246 INFO repeater task adu5_hcorrstats ended after 0.9 seconds 2018-09-27 12:35:20,246 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:20,246 INFO repeater task adu800_hcorrstats starting 4.0 seconds late 2018-09-27 12:35:21,770 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-27 12:35:21,770 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:21,771 INFO repeater task adu5_plotqc starting 5.6 seconds late 2018-09-27 12:35:22,717 INFO repeater task adu5_plotqc output: found 50 adu files in /home/data/oc1809b/rbin/adu5 2018-09-27 12:35:22,718 INFO repeater task adu5_plotqc ended after 0.9 seconds 2018-09-27 12:35:22,718 INFO repeater task adu5_plotqc next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:22,718 INFO repeater task adu800_plotqc starting 6.5 seconds late 2018-09-27 12:35:22,924 ERROR repeater task adu800_plotqc returncode 1 2018-09-27 12:35:22,924 INFO repeater task adu800_plotqc output: found 2 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-27 12:35:22,924 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-27 12:35:22,924 INFO repeater task adu800_plotqc next time_to_run is 2018/09/27 12:50:16 2018-09-27 12:35:27,930 INFO repeater task wh300_3dayplots starting 6.9 seconds late 2018-09-27 12:35:33,369 INFO repeater task wh300_3dayplots ended after 5.4 seconds 2018-09-27 12:35:33,369 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 12:40:21 2018-09-27 12:36:08,415 INFO repeater task wh300_lastensq starting 4.4 seconds late 2018-09-27 12:36:10,535 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:36:10,536 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:35:59 2018-09-27 12:36:10,536 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:38:04 2018-09-27 12:36:10,536 INFO repeater task wh300_quick starting 6.5 seconds late 2018-09-27 12:36:14,235 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:36:14,235 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:35:59 2018-09-27 12:36:14,235 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:38:04 2018-09-27 12:36:14,235 INFO repeater task wh300_lastvec starting 10.2 seconds late 2018-09-27 12:36:14,589 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:36:14,590 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:35:59 2018-09-27 12:36:14,590 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:38:04 2018-09-27 12:38:04,725 INFO repeater task wh300_lastensq starting 0.7 seconds late 2018-09-27 12:38:06,879 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-27 12:38:06,879 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:37:59 2018-09-27 12:38:06,879 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:40:04 2018-09-27 12:38:06,880 INFO repeater task wh300_quick starting 2.9 seconds late 2018-09-27 12:38:10,625 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:38:10,625 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:37:59 2018-09-27 12:38:10,625 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:40:04 2018-09-27 12:38:10,625 INFO repeater task wh300_lastvec starting 6.6 seconds late 2018-09-27 12:38:10,981 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:38:10,981 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:37:59 2018-09-27 12:38:10,981 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:40:04 2018-09-27 12:40:06,137 INFO repeater task wh300_lastensq starting 2.1 seconds late 2018-09-27 12:40:08,216 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:40:08,216 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:39:59 2018-09-27 12:40:08,216 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:42:04 2018-09-27 12:40:08,216 INFO repeater task wh300_quick starting 4.2 seconds late 2018-09-27 12:40:11,917 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:40:11,918 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:39:59 2018-09-27 12:40:11,918 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:42:04 2018-09-27 12:40:11,918 INFO repeater task wh300_lastvec starting 7.9 seconds late 2018-09-27 12:40:12,275 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:40:12,275 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:39:59 2018-09-27 12:40:12,275 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:42:04 2018-09-27 12:40:22,287 INFO repeater task wh300_3dayplots starting 1.3 seconds late 2018-09-27 12:40:27,518 INFO repeater task wh300_3dayplots ended after 5.2 seconds 2018-09-27 12:40:27,518 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 12:45:21 2018-09-27 12:42:07,647 INFO repeater task wh300_lastensq starting 3.6 seconds late 2018-09-27 12:42:09,811 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-27 12:42:09,811 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:41:59 2018-09-27 12:42:09,811 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:44:04 2018-09-27 12:42:09,812 INFO repeater task wh300_quick starting 5.8 seconds late 2018-09-27 12:42:13,531 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:42:13,532 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:41:59 2018-09-27 12:42:13,532 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:44:04 2018-09-27 12:42:13,532 INFO repeater task wh300_lastvec starting 9.5 seconds late 2018-09-27 12:42:13,904 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:42:13,904 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:41:59 2018-09-27 12:42:13,904 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:44:04 2018-09-27 12:44:04,051 INFO repeater task wh300_lastensq starting 0.1 seconds late 2018-09-27 12:44:06,181 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:44:06,181 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:43:59 2018-09-27 12:44:06,181 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:46:04 2018-09-27 12:44:06,181 INFO repeater task wh300_quick starting 2.2 seconds late 2018-09-27 12:44:09,866 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:44:09,866 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:43:59 2018-09-27 12:44:09,866 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:46:04 2018-09-27 12:44:09,866 INFO repeater task wh300_lastvec starting 5.9 seconds late 2018-09-27 12:44:10,224 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:44:10,224 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:43:59 2018-09-27 12:44:10,224 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:46:04 2018-09-27 12:45:25,304 INFO repeater task wh300_3dayplots starting 4.3 seconds late 2018-09-27 12:45:30,545 INFO repeater task wh300_3dayplots ended after 5.2 seconds 2018-09-27 12:45:30,545 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 12:50:21 2018-09-27 12:46:05,591 INFO repeater task wh300_lastensq starting 1.6 seconds late 2018-09-27 12:46:07,691 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:46:07,691 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:45:59 2018-09-27 12:46:07,691 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:48:04 2018-09-27 12:46:07,691 INFO repeater task wh300_quick starting 3.7 seconds late 2018-09-27 12:46:11,393 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:46:11,393 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:45:59 2018-09-27 12:46:11,394 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:48:04 2018-09-27 12:46:11,394 INFO repeater task wh300_lastvec starting 7.4 seconds late 2018-09-27 12:46:11,757 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:46:11,757 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:45:59 2018-09-27 12:46:11,757 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:48:04 2018-09-27 12:48:06,905 INFO repeater task wh300_lastensq starting 2.9 seconds late 2018-09-27 12:48:09,077 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-27 12:48:09,077 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:47:59 2018-09-27 12:48:09,077 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:50:04 2018-09-27 12:48:09,077 INFO repeater task wh300_quick starting 5.1 seconds late 2018-09-27 12:48:12,766 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:48:12,766 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:47:59 2018-09-27 12:48:12,766 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:50:04 2018-09-27 12:48:12,766 INFO repeater task wh300_lastvec starting 8.8 seconds late 2018-09-27 12:48:13,133 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:48:13,133 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:47:59 2018-09-27 12:48:13,133 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:50:04 2018-09-27 12:50:08,274 INFO repeater task wh300_lastensq starting 4.3 seconds late 2018-09-27 12:50:10,382 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:50:10,383 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:49:59 2018-09-27 12:50:10,383 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:52:04 2018-09-27 12:50:10,383 INFO repeater task wh300_quick starting 6.4 seconds late 2018-09-27 12:50:14,127 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:50:14,127 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:49:59 2018-09-27 12:50:14,128 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:52:04 2018-09-27 12:50:14,128 INFO repeater task wh300_lastvec starting 10.1 seconds late 2018-09-27 12:50:14,490 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:50:14,490 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:49:59 2018-09-27 12:50:14,490 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:52:04 2018-09-27 12:50:19,497 INFO repeater task wh300_tsstats starting 3.5 seconds late 2018-09-27 12:50:20,131 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-27 12:50:20,131 INFO repeater task wh300_tsstats next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:20,131 INFO repeater task wh300_10pings starting 4.1 seconds late 2018-09-27 12:50:20,243 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-27 12:50:20,243 INFO repeater task wh300_10pings next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:20,243 INFO repeater task wh300_beamplots starting 4.2 seconds late 2018-09-27 12:50:21,350 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-27 12:50:21,350 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:21,350 INFO repeater task wh300_beamplots starting 5.3 seconds late 2018-09-27 12:50:22,384 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 12:50:22,385 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:22,385 INFO repeater task adu5_hcorrstats starting 6.2 seconds late 2018-09-27 12:50:23,217 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-27 12:50:23,217 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:23,218 INFO repeater task adu800_hcorrstats starting 7.0 seconds late 2018-09-27 12:50:24,763 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-27 12:50:24,763 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:24,763 INFO repeater task adu5_plotqc starting 8.6 seconds late 2018-09-27 12:50:25,699 INFO repeater task adu5_plotqc output: found 50 adu files in /home/data/oc1809b/rbin/adu5 2018-09-27 12:50:25,699 INFO repeater task adu5_plotqc ended after 0.9 seconds 2018-09-27 12:50:25,699 INFO repeater task adu5_plotqc next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:25,699 INFO repeater task adu800_plotqc starting 9.5 seconds late 2018-09-27 12:50:25,905 ERROR repeater task adu800_plotqc returncode 1 2018-09-27 12:50:25,906 INFO repeater task adu800_plotqc output: found 2 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-27 12:50:25,906 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-27 12:50:25,906 INFO repeater task adu800_plotqc next time_to_run is 2018/09/27 13:05:16 2018-09-27 12:50:30,912 INFO repeater task wh300_3dayplots starting 9.9 seconds late 2018-09-27 12:50:36,061 INFO repeater task wh300_3dayplots ended after 5.1 seconds 2018-09-27 12:50:36,062 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 12:55:21 2018-09-27 12:52:06,188 INFO repeater task wh300_lastensq starting 2.2 seconds late 2018-09-27 12:52:08,284 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:52:08,284 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:51:59 2018-09-27 12:52:08,284 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:54:04 2018-09-27 12:52:08,284 INFO repeater task wh300_quick starting 4.3 seconds late 2018-09-27 12:52:12,015 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:52:12,015 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:51:59 2018-09-27 12:52:12,015 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:54:04 2018-09-27 12:52:12,016 INFO repeater task wh300_lastvec starting 8.0 seconds late 2018-09-27 12:52:12,381 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:52:12,381 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:51:59 2018-09-27 12:52:12,381 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:54:04 2018-09-27 12:54:07,524 INFO repeater task wh300_lastensq starting 3.5 seconds late 2018-09-27 12:54:09,629 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:54:09,629 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:53:59 2018-09-27 12:54:09,629 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:56:04 2018-09-27 12:54:09,630 INFO repeater task wh300_quick starting 5.6 seconds late 2018-09-27 12:54:13,354 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:54:13,355 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:53:59 2018-09-27 12:54:13,355 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:56:04 2018-09-27 12:54:13,355 INFO repeater task wh300_lastvec starting 9.4 seconds late 2018-09-27 12:54:13,719 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:54:13,719 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:53:59 2018-09-27 12:54:13,719 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:56:04 2018-09-27 12:55:23,811 INFO repeater task wh300_3dayplots starting 2.8 seconds late 2018-09-27 12:55:29,192 INFO repeater task wh300_3dayplots ended after 5.4 seconds 2018-09-27 12:55:29,192 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 13:00:21 2018-09-27 12:56:04,227 INFO repeater task wh300_lastensq starting 0.2 seconds late 2018-09-27 12:56:06,348 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:56:06,348 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:55:59 2018-09-27 12:56:06,348 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 12:58:04 2018-09-27 12:56:06,349 INFO repeater task wh300_quick starting 2.3 seconds late 2018-09-27 12:56:10,046 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 12:56:10,046 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:55:59 2018-09-27 12:56:10,046 INFO repeater task wh300_quick next time_to_run is 2018/09/27 12:58:04 2018-09-27 12:56:10,047 INFO repeater task wh300_lastvec starting 6.0 seconds late 2018-09-27 12:56:10,410 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:56:10,410 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:55:59 2018-09-27 12:56:10,410 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 12:58:04 2018-09-27 12:58:05,541 INFO repeater task wh300_lastensq starting 1.5 seconds late 2018-09-27 12:58:07,639 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 12:58:07,640 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:57:59 2018-09-27 12:58:07,640 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:00:04 2018-09-27 12:58:07,640 INFO repeater task wh300_quick starting 3.6 seconds late 2018-09-27 12:58:11,399 INFO repeater task wh300_quick ended after 3.8 seconds 2018-09-27 12:58:11,399 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:57:59 2018-09-27 12:58:11,399 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:00:04 2018-09-27 12:58:11,400 INFO repeater task wh300_lastvec starting 7.4 seconds late 2018-09-27 12:58:11,768 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 12:58:11,768 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:57:59 2018-09-27 12:58:11,768 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:00:04 2018-09-27 13:00:06,931 INFO repeater task wh300_lastensq starting 2.9 seconds late 2018-09-27 13:00:09,046 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:00:09,047 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:59:59 2018-09-27 13:00:09,047 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:02:04 2018-09-27 13:00:09,047 INFO repeater task wh300_quick starting 5.0 seconds late 2018-09-27 13:00:12,866 INFO repeater task wh300_quick ended after 3.8 seconds 2018-09-27 13:00:12,866 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:59:59 2018-09-27 13:00:12,866 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:02:04 2018-09-27 13:00:12,866 INFO repeater task wh300_lastvec starting 8.9 seconds late 2018-09-27 13:00:13,240 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:00:13,240 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 12:59:59 2018-09-27 13:00:13,240 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:02:04 2018-09-27 13:00:23,254 INFO repeater task wh300_3dayplots starting 2.2 seconds late 2018-09-27 13:00:28,539 INFO repeater task wh300_3dayplots ended after 5.3 seconds 2018-09-27 13:00:28,539 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 13:05:21 2018-09-27 13:02:08,664 INFO repeater task wh300_lastensq starting 4.7 seconds late 2018-09-27 13:02:10,771 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:02:10,771 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:01:59 2018-09-27 13:02:10,771 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:04:04 2018-09-27 13:02:10,772 INFO repeater task wh300_quick starting 6.8 seconds late 2018-09-27 13:02:14,463 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:02:14,463 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:01:59 2018-09-27 13:02:14,463 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:04:04 2018-09-27 13:02:14,463 INFO repeater task wh300_lastvec starting 10.5 seconds late 2018-09-27 13:02:14,833 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:02:14,833 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:01:59 2018-09-27 13:02:14,833 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:04:04 2018-09-27 13:04:04,973 INFO repeater task wh300_lastensq starting 1.0 seconds late 2018-09-27 13:04:07,089 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:04:07,089 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:03:59 2018-09-27 13:04:07,089 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:06:04 2018-09-27 13:04:07,089 INFO repeater task wh300_quick starting 3.1 seconds late 2018-09-27 13:04:10,821 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:04:10,821 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:03:59 2018-09-27 13:04:10,822 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:06:04 2018-09-27 13:04:10,822 INFO repeater task wh300_lastvec starting 6.8 seconds late 2018-09-27 13:04:11,191 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:04:11,191 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:03:59 2018-09-27 13:04:11,191 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:06:04 2018-09-27 13:05:16,266 INFO repeater task wh300_tsstats starting 0.2 seconds late 2018-09-27 13:05:16,908 INFO repeater task wh300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1118: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2018-09-27 13:05:16,908 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-27 13:05:16,908 INFO repeater task wh300_tsstats next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:16,908 INFO repeater task wh300_10pings starting 0.9 seconds late 2018-09-27 13:05:17,016 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-27 13:05:17,016 INFO repeater task wh300_10pings next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:17,016 INFO repeater task wh300_beamplots starting 1.0 seconds late 2018-09-27 13:05:18,051 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 13:05:18,051 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:18,051 INFO repeater task wh300_beamplots starting 2.0 seconds late 2018-09-27 13:05:19,097 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 13:05:19,097 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:19,097 INFO repeater task adu5_hcorrstats starting 2.9 seconds late 2018-09-27 13:05:19,934 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-27 13:05:19,935 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:19,935 INFO repeater task adu800_hcorrstats starting 3.7 seconds late 2018-09-27 13:05:21,526 INFO repeater task adu800_hcorrstats ended after 1.6 seconds 2018-09-27 13:05:21,527 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:21,527 INFO repeater task adu5_plotqc starting 5.3 seconds late 2018-09-27 13:05:22,486 INFO repeater task adu5_plotqc output: found 50 adu files in /home/data/oc1809b/rbin/adu5 2018-09-27 13:05:22,486 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-27 13:05:22,487 INFO repeater task adu5_plotqc next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:22,487 INFO repeater task adu800_plotqc starting 6.3 seconds late 2018-09-27 13:05:22,695 ERROR repeater task adu800_plotqc returncode 1 2018-09-27 13:05:22,695 INFO repeater task adu800_plotqc output: found 2 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-27 13:05:22,695 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-27 13:05:22,695 INFO repeater task adu800_plotqc next time_to_run is 2018/09/27 13:20:16 2018-09-27 13:05:22,696 INFO repeater task report_generator starting 6.5 seconds late 2018-09-27 13:05:26,732 INFO repeater task report_generator output: cruisename is report_dir is uhdas_dir is 0.0 sec : initial png rsync 0.0 sec : initial png rsync 1.0 sec : navplot: plot_rnav.py -s300 --noshow --title oc1809b --outfile /home/data/oc1809b/reports/nav_plot_all /home/data/oc1809b/rbin/gpsnav/*.gps.rbin 0.0 sec : sonar_usage: plot_db_timeranges.py --outfile /home/data/oc1809b/reports/sonar_usage --noshow /home/data/oc1809b 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/nav_plot_all_topo.png /home/data/oc1809b/reports/nav_plot_all_topoT.png 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/nav_plot_all_txy.png /home/data/oc1809b/reports/nav_plot_all_txyT.png 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/sonar_usage.png /home/data/oc1809b/reports/sonar_usageT.png 0.0 sec : png_archive: rsync -a /home/data/oc1809b/proc/wh300/png_archive/ /home/data/oc1809b/reports/wh300_pngarchive 0.0 sec : png_archive: rsync -a /home/data/oc1809b/proc/os75nb/png_archive/ /home/data/oc1809b/reports/os75nb_pngarchive 0.0 sec : uhdas_overview: uhdas_info.py --overview --logfile /home/data/oc1809b/reports/uhdas_overview.txt /home/data/oc1809b 0.0 sec : sonar_summary: sonar_summary.py --outfile /home/data/oc1809b/reports/sonar_summary.txt --uhdas_dir /home/data/oc1809b 0.0 sec : adcp_settings: uhdas_info.py --settings --logfile /home/data/oc1809b/reports/ADCP_settings.txt /home/data/oc1809b 0.0 sec : adcp_cals: uhdas_info.py --cals --logfile /home/data/oc1809b/reports/ADCP_cals.txt /home/data/oc1809b 1.0 sec : ens_hcorr: plot_enshcorr.py --infile /home/data/oc1809b/proc/wh300/cal/rotate/ens_hcorr.asc --outfile /home/data/oc1809b/reports/wh300_hcorr --yearbase 2018 --titlestring ens_hcorr_adu 0.0 sec : ens_hcorr: convert -resize 200 /home/data/oc1809b/reports/wh300_hcorr.png /home/data/oc1809b/reports/wh300_hcorrT.png 1.0 sec : ens_hcorr: plot_enshcorr.py --infile /home/data/oc1809b/proc/os75nb/cal/rotate/ens_hcorr.asc --outfile /home/data/oc1809b/reports/os75nb_hcorr --yearbase 2018 --titlestring ens_hcorr_adu 0.0 sec : ens_hcorr: convert -resize 200 /home/data/oc1809b/reports/os75nb_hcorr.png /home/data/oc1809b/reports/os75nb_hcorrT.png 0.0 sec : pycfg: cp -p /home/data/oc1809b/raw/config/oc1809b_proc.py /home/data/oc1809b/reports/oc1809b_proc.py.txt 0.0 sec : pycfg: cp -p /home/data/oc1809b/raw/config/oc1809b_sensor.py /home/data/oc1809b/reports/oc1809b_sensor.py.txt 4.0 seconds elapsed html page starts here: /home/data/oc1809b/reports/index.html 2018-09-27 13:05:26,732 INFO repeater task report_generator ended after 4.0 seconds 2018-09-27 13:05:26,732 INFO repeater task report_generator next time_to_run is 2018/09/27 14:05:16 2018-09-27 13:05:31,736 INFO repeater task wh300_3dayplots starting 10.7 seconds late 2018-09-27 13:05:36,989 INFO repeater task wh300_3dayplots ended after 5.3 seconds 2018-09-27 13:05:36,989 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 13:10:21 2018-09-27 13:06:07,028 INFO repeater task wh300_lastensq starting 3.0 seconds late 2018-09-27 13:06:09,170 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:06:09,171 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:05:59 2018-09-27 13:06:09,171 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:08:04 2018-09-27 13:06:09,171 INFO repeater task wh300_quick starting 5.2 seconds late 2018-09-27 13:06:12,883 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:06:12,883 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:05:59 2018-09-27 13:06:12,883 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:08:04 2018-09-27 13:06:12,884 INFO repeater task wh300_lastvec starting 8.9 seconds late 2018-09-27 13:06:13,244 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:06:13,245 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:05:59 2018-09-27 13:06:13,245 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:08:04 2018-09-27 13:08:08,384 INFO repeater task wh300_lastensq starting 4.4 seconds late 2018-09-27 13:08:10,498 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:08:10,498 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:07:59 2018-09-27 13:08:10,498 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:10:04 2018-09-27 13:08:10,499 INFO repeater task wh300_quick starting 6.5 seconds late 2018-09-27 13:08:14,206 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:08:14,207 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:07:59 2018-09-27 13:08:14,207 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:10:04 2018-09-27 13:08:14,207 INFO repeater task wh300_lastvec starting 10.2 seconds late 2018-09-27 13:08:14,571 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:08:14,572 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:07:59 2018-09-27 13:08:14,572 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:10:04 2018-09-27 13:10:04,724 INFO repeater task wh300_lastensq starting 0.7 seconds late 2018-09-27 13:10:06,879 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-27 13:10:06,880 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:09:59 2018-09-27 13:10:06,880 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:12:04 2018-09-27 13:10:06,880 INFO repeater task wh300_quick starting 2.9 seconds late 2018-09-27 13:10:10,636 INFO repeater task wh300_quick ended after 3.8 seconds 2018-09-27 13:10:10,636 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:09:59 2018-09-27 13:10:10,636 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:12:04 2018-09-27 13:10:10,637 INFO repeater task wh300_lastvec starting 6.6 seconds late 2018-09-27 13:10:10,992 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:10:10,993 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:09:59 2018-09-27 13:10:10,993 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:12:04 2018-09-27 13:10:26,008 INFO repeater task wh300_3dayplots starting 5.0 seconds late 2018-09-27 13:10:31,340 INFO repeater task wh300_3dayplots ended after 5.3 seconds 2018-09-27 13:10:31,340 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 13:15:21 2018-09-27 13:12:06,446 INFO repeater task wh300_lastensq starting 2.4 seconds late 2018-09-27 13:12:08,554 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:12:08,554 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:11:59 2018-09-27 13:12:08,554 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:14:04 2018-09-27 13:12:08,555 INFO repeater task wh300_quick starting 4.6 seconds late 2018-09-27 13:12:12,249 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:12:12,249 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:11:59 2018-09-27 13:12:12,249 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:14:04 2018-09-27 13:12:12,249 INFO repeater task wh300_lastvec starting 8.2 seconds late 2018-09-27 13:12:12,623 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:12:12,624 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:11:59 2018-09-27 13:12:12,624 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:14:04 2018-09-27 13:14:07,775 INFO repeater task wh300_lastensq starting 3.8 seconds late 2018-09-27 13:14:09,885 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:14:09,885 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:13:59 2018-09-27 13:14:09,885 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:16:04 2018-09-27 13:14:09,885 INFO repeater task wh300_quick starting 5.9 seconds late 2018-09-27 13:14:13,601 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:14:13,601 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:13:59 2018-09-27 13:14:13,602 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:16:04 2018-09-27 13:14:13,602 INFO repeater task wh300_lastvec starting 9.6 seconds late 2018-09-27 13:14:13,967 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:14:13,968 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:13:59 2018-09-27 13:14:13,968 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:16:04 2018-09-27 13:15:24,072 INFO repeater task wh300_3dayplots starting 3.0 seconds late 2018-09-27 13:15:29,302 INFO repeater task wh300_3dayplots ended after 5.2 seconds 2018-09-27 13:15:29,302 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 13:20:21 2018-09-27 13:16:04,336 INFO repeater task wh300_lastensq starting 0.3 seconds late 2018-09-27 13:16:06,451 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:16:06,451 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:15:59 2018-09-27 13:16:06,451 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:18:04 2018-09-27 13:16:06,452 INFO repeater task wh300_quick starting 2.5 seconds late 2018-09-27 13:16:10,160 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:16:10,160 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:15:59 2018-09-27 13:16:10,160 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:18:04 2018-09-27 13:16:10,160 INFO repeater task wh300_lastvec starting 6.2 seconds late 2018-09-27 13:16:10,525 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:16:10,526 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:15:59 2018-09-27 13:16:10,526 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:18:04 2018-09-27 13:18:05,679 INFO repeater task wh300_lastensq starting 1.7 seconds late 2018-09-27 13:18:07,789 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:18:07,789 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:17:59 2018-09-27 13:18:07,789 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:20:04 2018-09-27 13:18:07,789 INFO repeater task wh300_quick starting 3.8 seconds late 2018-09-27 13:18:11,467 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:18:11,467 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:17:59 2018-09-27 13:18:11,467 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:20:04 2018-09-27 13:18:11,467 INFO repeater task wh300_lastvec starting 7.5 seconds late 2018-09-27 13:18:11,832 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:18:11,832 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:17:59 2018-09-27 13:18:11,832 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:20:04 2018-09-27 13:20:06,980 INFO repeater task wh300_lastensq starting 3.0 seconds late 2018-09-27 13:20:09,084 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:20:09,084 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:19:59 2018-09-27 13:20:09,085 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:22:04 2018-09-27 13:20:09,085 INFO repeater task wh300_quick starting 5.1 seconds late 2018-09-27 13:20:12,810 INFO repeater task wh300_quick ended after 3.7 seconds 2018-09-27 13:20:12,811 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:19:59 2018-09-27 13:20:12,811 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:22:04 2018-09-27 13:20:12,811 INFO repeater task wh300_lastvec starting 8.8 seconds late 2018-09-27 13:20:13,172 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:20:13,172 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:19:59 2018-09-27 13:20:13,172 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:22:04 2018-09-27 13:20:18,179 INFO repeater task wh300_tsstats starting 2.1 seconds late 2018-09-27 13:20:18,852 INFO repeater task wh300_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1118: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2018-09-27 13:20:18,852 INFO repeater task wh300_tsstats ended after 0.7 seconds 2018-09-27 13:20:18,852 INFO repeater task wh300_tsstats next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:18,852 INFO repeater task wh300_10pings starting 2.8 seconds late 2018-09-27 13:20:18,962 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-27 13:20:18,962 INFO repeater task wh300_10pings next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:18,963 INFO repeater task wh300_beamplots starting 2.9 seconds late 2018-09-27 13:20:20,003 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 13:20:20,003 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:20,003 INFO repeater task wh300_beamplots starting 4.0 seconds late 2018-09-27 13:20:21,048 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-27 13:20:21,048 INFO repeater task wh300_beamplots next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:21,048 INFO repeater task adu5_hcorrstats starting 4.9 seconds late 2018-09-27 13:20:21,888 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-27 13:20:21,888 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:21,888 INFO repeater task adu800_hcorrstats starting 5.7 seconds late 2018-09-27 13:20:23,443 INFO repeater task adu800_hcorrstats ended after 1.6 seconds 2018-09-27 13:20:23,443 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:23,443 INFO repeater task adu5_plotqc starting 7.2 seconds late 2018-09-27 13:20:24,404 INFO repeater task adu5_plotqc output: found 50 adu files in /home/data/oc1809b/rbin/adu5 2018-09-27 13:20:24,405 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-27 13:20:24,405 INFO repeater task adu5_plotqc next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:24,405 INFO repeater task adu800_plotqc starting 8.2 seconds late 2018-09-27 13:20:24,614 ERROR repeater task adu800_plotqc returncode 1 2018-09-27 13:20:24,614 INFO repeater task adu800_plotqc output: found 2 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-27 13:20:24,614 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-27 13:20:24,614 INFO repeater task adu800_plotqc next time_to_run is 2018/09/27 13:35:16 2018-09-27 13:20:29,620 INFO repeater task wh300_3dayplots starting 8.6 seconds late 2018-09-27 13:20:34,741 INFO repeater task wh300_3dayplots ended after 5.1 seconds 2018-09-27 13:20:34,741 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/27 13:25:21 2018-09-27 13:22:04,869 INFO repeater task wh300_lastensq starting 0.9 seconds late 2018-09-27 13:22:06,963 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-27 13:22:06,963 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:21:59 2018-09-27 13:22:06,964 INFO repeater task wh300_lastensq next time_to_run is 2018/09/27 13:24:04 2018-09-27 13:22:06,964 INFO repeater task wh300_quick starting 3.0 seconds late 2018-09-27 13:22:10,726 INFO repeater task wh300_quick ended after 3.8 seconds 2018-09-27 13:22:10,726 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:21:59 2018-09-27 13:22:10,726 INFO repeater task wh300_quick next time_to_run is 2018/09/27 13:24:04 2018-09-27 13:22:10,726 INFO repeater task wh300_lastvec starting 6.7 seconds late 2018-09-27 13:22:11,106 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-27 13:22:11,106 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/27 13:21:59 2018-09-27 13:22:11,106 INFO repeater task wh300_lastvec next time_to_run is 2018/09/27 13:24:04 2018-09-27 13:23:09,247 INFO incr_class translate exiting, hdg /home/data/oc1809b/rbin/gyro 2018-09-27 13:23:09,249 INFO incr_class translate exiting, adu /home/data/oc1809b/rbin/adu5 2018-09-27 13:23:09,263 INFO incr_class translate exiting, gps /home/data/oc1809b/rbin/adu5 2018-09-27 13:23:09,269 INFO incr_class translate exiting, gps /home/data/oc1809b/rbin/gpsnav 2018-09-27 13:23:09,281 INFO incr_class translate exiting, gps /home/data/oc1809b/rbin/adu800 2018-09-27 13:23:09,281 INFO incr_class translate exiting, hdg /home/data/oc1809b/rbin/adu800 2018-09-27 13:23:09,281 INFO incr_class translate exiting, adu /home/data/oc1809b/rbin/adu800 2018-09-27 13:23:11,187 INFO repeater Running task report_generator at exit 2018-09-27 13:23:11,188 INFO repeater task report_generator starting -2525.0 seconds late 2018-09-27 13:23:15,125 INFO repeater task report_generator output: cruisename is report_dir is uhdas_dir is 0.0 sec : initial png rsync 0.0 sec : initial png rsync 1.0 sec : navplot: plot_rnav.py -s300 --noshow --title oc1809b --outfile /home/data/oc1809b/reports/nav_plot_all /home/data/oc1809b/rbin/gpsnav/*.gps.rbin 0.0 sec : sonar_usage: plot_db_timeranges.py --outfile /home/data/oc1809b/reports/sonar_usage --noshow /home/data/oc1809b 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/nav_plot_all_topo.png /home/data/oc1809b/reports/nav_plot_all_topoT.png 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/nav_plot_all_txy.png /home/data/oc1809b/reports/nav_plot_all_txyT.png 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/sonar_usage.png /home/data/oc1809b/reports/sonar_usageT.png 0.0 sec : png_archive: rsync -a /home/data/oc1809b/proc/wh300/png_archive/ /home/data/oc1809b/reports/wh300_pngarchive 0.0 sec : png_archive: rsync -a /home/data/oc1809b/proc/os75nb/png_archive/ /home/data/oc1809b/reports/os75nb_pngarchive 0.0 sec : uhdas_overview: uhdas_info.py --overview --logfile /home/data/oc1809b/reports/uhdas_overview.txt /home/data/oc1809b 0.0 sec : sonar_summary: sonar_summary.py --outfile /home/data/oc1809b/reports/sonar_summary.txt --uhdas_dir /home/data/oc1809b 0.0 sec : adcp_settings: uhdas_info.py --settings --logfile /home/data/oc1809b/reports/ADCP_settings.txt /home/data/oc1809b 0.0 sec : adcp_cals: uhdas_info.py --cals --logfile /home/data/oc1809b/reports/ADCP_cals.txt /home/data/oc1809b 1.0 sec : ens_hcorr: plot_enshcorr.py --infile /home/data/oc1809b/proc/wh300/cal/rotate/ens_hcorr.asc --outfile /home/data/oc1809b/reports/wh300_hcorr --yearbase 2018 --titlestring ens_hcorr_adu 0.0 sec : ens_hcorr: convert -resize 200 /home/data/oc1809b/reports/wh300_hcorr.png /home/data/oc1809b/reports/wh300_hcorrT.png 1.0 sec : ens_hcorr: plot_enshcorr.py --infile /home/data/oc1809b/proc/os75nb/cal/rotate/ens_hcorr.asc --outfile /home/data/oc1809b/reports/os75nb_hcorr --yearbase 2018 --titlestring ens_hcorr_adu 0.0 sec : ens_hcorr: convert -resize 200 /home/data/oc1809b/reports/os75nb_hcorr.png /home/data/oc1809b/reports/os75nb_hcorrT.png 0.0 sec : pycfg: cp -p /home/data/oc1809b/raw/config/oc1809b_proc.py /home/data/oc1809b/reports/oc1809b_proc.py.txt 0.0 sec : pycfg: cp -p /home/data/oc1809b/raw/config/oc1809b_sensor.py /home/data/oc1809b/reports/oc1809b_sensor.py.txt 4.0 seconds elapsed html page starts here: /home/data/oc1809b/reports/index.html 2018-09-27 13:23:15,125 INFO repeater task report_generator ended after 3.9 seconds 2018-09-27 13:23:15,125 INFO repeater task report_generator next time_to_run is 2018/09/27 15:05:16 2018-09-27 13:23:15,126 INFO DAS_while_logging cleanup_figs not_logging 2018-09-27 13:23:15,131 INFO DAS_while_logging Exiting DAS_while_logging.py