2018-09-26 22:32:08,187 INFO repeater task wh300_lastensq starting 4.2 seconds late 2018-09-26 22:32:10,278 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:32:10,278 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:31:59 2018-09-26 22:32:10,278 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:34:04 2018-09-26 22:32:10,278 INFO repeater task wh300_quick starting 6.3 seconds late 2018-09-26 22:32:13,581 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:32:13,581 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:31:59 2018-09-26 22:32:13,581 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:34:04 2018-09-26 22:32:13,581 INFO repeater task wh300_lastvec starting 9.6 seconds late 2018-09-26 22:32:13,959 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:32:13,959 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:31:59 2018-09-26 22:32:13,959 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:34:04 2018-09-26 22:34:04,136 INFO repeater task wh300_lastensq starting 0.1 seconds late 2018-09-26 22:34:06,251 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:34:06,251 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:33:59 2018-09-26 22:34:06,251 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:36:04 2018-09-26 22:34:06,251 INFO repeater task wh300_quick starting 2.3 seconds late 2018-09-26 22:34:09,563 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:34:09,563 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:33:59 2018-09-26 22:34:09,563 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:36:04 2018-09-26 22:34:09,564 INFO repeater task wh300_lastvec starting 5.6 seconds late 2018-09-26 22:34:09,934 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:34:09,934 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:33:59 2018-09-26 22:34:09,935 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:36:04 2018-09-26 22:35:05,035 INFO repeater task wh300_tsstats starting 4.9 seconds late 2018-09-26 22:35:05,709 INFO repeater task wh300_tsstats ended after 0.7 seconds 2018-09-26 22:35:05,709 INFO repeater task wh300_tsstats next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:05,709 INFO repeater task wh300_10pings starting 5.6 seconds late 2018-09-26 22:35:05,819 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-26 22:35:05,819 INFO repeater task wh300_10pings next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:05,820 INFO repeater task wh300_beamplots starting 5.7 seconds late 2018-09-26 22:35:06,869 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-26 22:35:06,869 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:06,869 INFO repeater task wh300_beamplots starting 6.8 seconds late 2018-09-26 22:35:07,912 INFO repeater task wh300_beamplots ended after 1.0 seconds 2018-09-26 22:35:07,912 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:07,912 INFO repeater task os75nb_lastensq starting 1.9 seconds late 2018-09-26 22:35:09,350 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 22:35:09,350 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:35:01 2018-09-26 22:35:09,350 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 22:40:06 2018-09-26 22:35:09,350 INFO repeater task os75nb_quick starting 3.4 seconds late 2018-09-26 22:35:12,133 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 22:35:12,133 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:35:01 2018-09-26 22:35:12,133 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 22:40:06 2018-09-26 22:35:12,134 INFO repeater task os75nb_lastvec starting 6.1 seconds late 2018-09-26 22:35:12,493 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 22:35:12,494 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:35:01 2018-09-26 22:35:12,494 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 22:40:06 2018-09-26 22:35:12,494 INFO repeater task os75nb_3dayplots starting 7.4 seconds late 2018-09-26 22:35:16,900 INFO repeater task os75nb_3dayplots ended after 4.4 seconds 2018-09-26 22:35:16,900 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 22:40:05 2018-09-26 22:35:16,900 INFO repeater task os75nb_tsstats starting 16.8 seconds late 2018-09-26 22:35:17,495 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2018-09-26 22:35:17,495 INFO repeater task os75nb_tsstats next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:17,495 INFO repeater task os75nb_10pings starting 17.4 seconds late 2018-09-26 22:35:17,602 INFO repeater task os75nb_10pings ended after 0.1 seconds 2018-09-26 22:35:17,602 INFO repeater task os75nb_10pings next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:17,603 INFO repeater task os75nb_beamplots starting 17.5 seconds late 2018-09-26 22:35:18,627 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 22:35:18,627 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:18,628 INFO repeater task os75nb_beamplots starting 18.5 seconds late 2018-09-26 22:35:19,660 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 22:35:19,660 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:19,660 INFO repeater task adu5_hcorrstats starting 19.4 seconds late 2018-09-26 22:35:20,474 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-26 22:35:20,474 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:20,474 INFO repeater task adu800_hcorrstats starting 20.2 seconds late 2018-09-26 22:35:22,006 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-26 22:35:22,006 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:22,006 INFO repeater task adu5_plotqc starting 21.8 seconds late 2018-09-26 22:35:22,977 INFO repeater task adu5_plotqc output: found 42 adu files in /home/data/oc1809b/rbin/adu5 2018-09-26 22:35:22,977 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-26 22:35:22,977 INFO repeater task adu5_plotqc next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:22,977 INFO repeater task adu800_plotqc starting 22.7 seconds late 2018-09-26 22:35:23,192 ERROR repeater task adu800_plotqc returncode 1 2018-09-26 22:35:23,192 INFO repeater task adu800_plotqc output: found 1 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-26 22:35:23,193 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-26 22:35:23,193 INFO repeater task adu800_plotqc next time_to_run is 2018/09/26 22:50:00 2018-09-26 22:35:28,196 INFO repeater task wh300_3dayplots starting 23.1 seconds late 2018-09-26 22:35:33,174 INFO repeater task wh300_3dayplots ended after 5.0 seconds 2018-09-26 22:35:33,174 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 22:40:05 2018-09-26 22:36:08,224 INFO repeater task wh300_lastensq starting 4.2 seconds late 2018-09-26 22:36:10,322 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:36:10,322 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:35:59 2018-09-26 22:36:10,322 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:38:04 2018-09-26 22:36:10,322 INFO repeater task wh300_quick starting 6.3 seconds late 2018-09-26 22:36:13,655 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:36:13,655 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:35:59 2018-09-26 22:36:13,655 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:38:04 2018-09-26 22:36:13,656 INFO repeater task wh300_lastvec starting 9.7 seconds late 2018-09-26 22:36:14,038 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:36:14,038 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:35:59 2018-09-26 22:36:14,039 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:38:04 2018-09-26 22:38:04,200 INFO repeater task wh300_lastensq starting 0.2 seconds late 2018-09-26 22:38:06,302 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:38:06,303 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:37:59 2018-09-26 22:38:06,303 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:40:04 2018-09-26 22:38:06,303 INFO repeater task wh300_quick starting 2.3 seconds late 2018-09-26 22:38:09,702 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 22:38:09,702 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:37:59 2018-09-26 22:38:09,702 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:40:04 2018-09-26 22:38:09,702 INFO repeater task wh300_lastvec starting 5.7 seconds late 2018-09-26 22:38:10,082 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:38:10,082 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:37:59 2018-09-26 22:38:10,082 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:40:04 2018-09-26 22:40:05,262 INFO repeater task wh300_lastensq starting 1.3 seconds late 2018-09-26 22:40:07,389 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:40:07,390 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:39:59 2018-09-26 22:40:07,390 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:42:04 2018-09-26 22:40:07,390 INFO repeater task wh300_quick starting 3.4 seconds late 2018-09-26 22:40:10,773 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 22:40:10,774 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:39:59 2018-09-26 22:40:10,774 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:42:04 2018-09-26 22:40:10,774 INFO repeater task wh300_lastvec starting 6.8 seconds late 2018-09-26 22:40:11,153 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:40:11,153 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:39:59 2018-09-26 22:40:11,154 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:42:04 2018-09-26 22:40:11,154 INFO repeater task wh300_3dayplots starting 6.1 seconds late 2018-09-26 22:40:16,238 INFO repeater task wh300_3dayplots ended after 5.1 seconds 2018-09-26 22:40:16,239 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 22:45:05 2018-09-26 22:40:16,239 INFO repeater task os75nb_lastensq starting 10.2 seconds late 2018-09-26 22:40:17,646 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 22:40:17,646 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:40:01 2018-09-26 22:40:17,646 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 22:45:06 2018-09-26 22:40:17,647 INFO repeater task os75nb_quick starting 11.6 seconds late 2018-09-26 22:40:20,542 INFO repeater task os75nb_quick ended after 2.9 seconds 2018-09-26 22:40:20,542 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:40:01 2018-09-26 22:40:20,542 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 22:45:06 2018-09-26 22:40:20,542 INFO repeater task os75nb_lastvec starting 14.5 seconds late 2018-09-26 22:40:20,907 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 22:40:20,908 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:40:01 2018-09-26 22:40:20,908 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 22:45:06 2018-09-26 22:40:20,908 INFO repeater task os75nb_3dayplots starting 15.8 seconds late 2018-09-26 22:40:25,334 INFO repeater task os75nb_3dayplots ended after 4.4 seconds 2018-09-26 22:40:25,334 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 22:45:05 2018-09-26 22:42:05,475 INFO repeater task wh300_lastensq starting 1.5 seconds late 2018-09-26 22:42:07,605 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:42:07,605 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:41:59 2018-09-26 22:42:07,605 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:44:04 2018-09-26 22:42:07,606 INFO repeater task wh300_quick starting 3.6 seconds late 2018-09-26 22:42:10,940 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:42:10,940 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:41:59 2018-09-26 22:42:10,940 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:44:04 2018-09-26 22:42:10,941 INFO repeater task wh300_lastvec starting 6.9 seconds late 2018-09-26 22:42:11,327 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:42:11,327 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:41:59 2018-09-26 22:42:11,327 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:44:04 2018-09-26 22:44:06,492 INFO repeater task wh300_lastensq starting 2.5 seconds late 2018-09-26 22:44:08,612 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:44:08,613 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:43:59 2018-09-26 22:44:08,613 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:46:04 2018-09-26 22:44:08,613 INFO repeater task wh300_quick starting 4.6 seconds late 2018-09-26 22:44:11,983 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 22:44:11,983 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:43:59 2018-09-26 22:44:11,983 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:46:04 2018-09-26 22:44:11,983 INFO repeater task wh300_lastvec starting 8.0 seconds late 2018-09-26 22:44:12,371 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:44:12,371 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:43:59 2018-09-26 22:44:12,371 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:46:04 2018-09-26 22:45:07,449 INFO repeater task wh300_3dayplots starting 2.4 seconds late 2018-09-26 22:45:12,323 INFO repeater task wh300_3dayplots ended after 4.9 seconds 2018-09-26 22:45:12,323 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 22:50:05 2018-09-26 22:45:12,324 INFO repeater task os75nb_lastensq starting 6.3 seconds late 2018-09-26 22:45:13,747 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 22:45:13,747 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:45:01 2018-09-26 22:45:13,747 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 22:50:06 2018-09-26 22:45:13,747 INFO repeater task os75nb_quick starting 7.7 seconds late 2018-09-26 22:45:16,606 INFO repeater task os75nb_quick ended after 2.9 seconds 2018-09-26 22:45:16,606 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:45:01 2018-09-26 22:45:16,606 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 22:50:06 2018-09-26 22:45:16,607 INFO repeater task os75nb_lastvec starting 10.6 seconds late 2018-09-26 22:45:16,969 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 22:45:16,969 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:45:01 2018-09-26 22:45:16,969 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 22:50:06 2018-09-26 22:45:16,969 INFO repeater task os75nb_3dayplots starting 11.9 seconds late 2018-09-26 22:45:21,505 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 22:45:21,505 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 22:50:05 2018-09-26 22:46:06,560 INFO repeater task wh300_lastensq starting 2.6 seconds late 2018-09-26 22:46:08,665 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:46:08,666 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:45:59 2018-09-26 22:46:08,666 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:48:04 2018-09-26 22:46:08,666 INFO repeater task wh300_quick starting 4.7 seconds late 2018-09-26 22:46:12,010 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:46:12,011 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:45:59 2018-09-26 22:46:12,011 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:48:04 2018-09-26 22:46:12,011 INFO repeater task wh300_lastvec starting 8.0 seconds late 2018-09-26 22:46:12,400 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:46:12,400 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:45:59 2018-09-26 22:46:12,400 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:48:04 2018-09-26 22:48:07,538 INFO repeater task wh300_lastensq starting 3.5 seconds late 2018-09-26 22:48:09,681 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:48:09,682 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:47:59 2018-09-26 22:48:09,682 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:50:04 2018-09-26 22:48:09,682 INFO repeater task wh300_quick starting 5.7 seconds late 2018-09-26 22:48:13,015 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:48:13,016 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:47:59 2018-09-26 22:48:13,016 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:50:04 2018-09-26 22:48:13,016 INFO repeater task wh300_lastvec starting 9.0 seconds late 2018-09-26 22:48:13,378 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:48:13,379 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:47:59 2018-09-26 22:48:13,379 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:50:04 2018-09-26 22:50:03,536 INFO repeater task wh300_tsstats starting 3.4 seconds late 2018-09-26 22:50:04,171 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-26 22:50:04,172 INFO repeater task wh300_tsstats next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:04,172 INFO repeater task wh300_10pings starting 4.1 seconds late 2018-09-26 22:50:04,284 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-26 22:50:04,284 INFO repeater task wh300_10pings next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:04,284 INFO repeater task wh300_beamplots starting 4.2 seconds late 2018-09-26 22:50:05,382 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 22:50:05,382 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:05,383 INFO repeater task wh300_beamplots starting 5.3 seconds late 2018-09-26 22:50:06,474 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 22:50:06,474 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:06,474 INFO repeater task os75nb_lastensq starting 0.5 seconds late 2018-09-26 22:50:07,925 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2018-09-26 22:50:07,926 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:50:01 2018-09-26 22:50:07,926 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 22:55:06 2018-09-26 22:50:07,926 INFO repeater task os75nb_quick starting 1.9 seconds late 2018-09-26 22:50:10,749 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 22:50:10,750 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:50:01 2018-09-26 22:50:10,750 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 22:55:06 2018-09-26 22:50:10,750 INFO repeater task os75nb_lastvec starting 4.8 seconds late 2018-09-26 22:50:11,143 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 22:50:11,143 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:50:01 2018-09-26 22:50:11,143 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 22:55:06 2018-09-26 22:50:11,143 INFO repeater task os75nb_3dayplots starting 6.1 seconds late 2018-09-26 22:50:15,673 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 22:50:15,673 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 22:55:05 2018-09-26 22:50:15,673 INFO repeater task os75nb_tsstats starting 15.6 seconds late 2018-09-26 22:50:16,275 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2018-09-26 22:50:16,275 INFO repeater task os75nb_tsstats next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:16,276 INFO repeater task os75nb_10pings starting 16.2 seconds late 2018-09-26 22:50:16,386 INFO repeater task os75nb_10pings ended after 0.1 seconds 2018-09-26 22:50:16,386 INFO repeater task os75nb_10pings next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:16,386 INFO repeater task os75nb_beamplots starting 16.3 seconds late 2018-09-26 22:50:17,388 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 22:50:17,388 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:17,388 INFO repeater task os75nb_beamplots starting 17.3 seconds late 2018-09-26 22:50:18,408 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 22:50:18,408 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:18,409 INFO repeater task adu5_hcorrstats starting 18.2 seconds late 2018-09-26 22:50:19,231 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-26 22:50:19,231 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:19,232 INFO repeater task adu800_hcorrstats starting 19.0 seconds late 2018-09-26 22:50:20,771 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-26 22:50:20,771 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:20,771 INFO repeater task adu5_plotqc starting 20.5 seconds late 2018-09-26 22:50:21,749 INFO repeater task adu5_plotqc output: found 42 adu files in /home/data/oc1809b/rbin/adu5 2018-09-26 22:50:21,749 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-26 22:50:21,750 INFO repeater task adu5_plotqc next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:21,750 INFO repeater task adu800_plotqc starting 21.5 seconds late 2018-09-26 22:50:21,968 ERROR repeater task adu800_plotqc returncode 1 2018-09-26 22:50:21,968 INFO repeater task adu800_plotqc output: found 1 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-26 22:50:21,968 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-26 22:50:21,968 INFO repeater task adu800_plotqc next time_to_run is 2018/09/26 23:05:00 2018-09-26 22:50:21,968 INFO repeater task report_generator starting 21.7 seconds late 2018-09-26 22:50:25,889 INFO repeater task report_generator output: cruisename is report_dir is uhdas_dir is 0.0 sec : initial png rsync 0.0 sec : initial png rsync 1.0 sec : navplot: plot_rnav.py -s300 --noshow --title oc1809b --outfile /home/data/oc1809b/reports/nav_plot_all /home/data/oc1809b/rbin/gpsnav/*.gps.rbin 0.0 sec : sonar_usage: plot_db_timeranges.py --outfile /home/data/oc1809b/reports/sonar_usage --noshow /home/data/oc1809b 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/nav_plot_all_topo.png /home/data/oc1809b/reports/nav_plot_all_topoT.png 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/nav_plot_all_txy.png /home/data/oc1809b/reports/nav_plot_all_txyT.png 0.0 sec : sonar_usage: convert -resize 200 /home/data/oc1809b/reports/sonar_usage.png /home/data/oc1809b/reports/sonar_usageT.png 0.0 sec : png_archive: rsync -a /home/data/oc1809b/proc/wh300/png_archive/ /home/data/oc1809b/reports/wh300_pngarchive 0.0 sec : png_archive: rsync -a /home/data/oc1809b/proc/os75nb/png_archive/ /home/data/oc1809b/reports/os75nb_pngarchive 0.0 sec : uhdas_overview: uhdas_info.py --overview --logfile /home/data/oc1809b/reports/uhdas_overview.txt /home/data/oc1809b 0.0 sec : sonar_summary: sonar_summary.py --outfile /home/data/oc1809b/reports/sonar_summary.txt --uhdas_dir /home/data/oc1809b 0.0 sec : adcp_settings: uhdas_info.py --settings --logfile /home/data/oc1809b/reports/ADCP_settings.txt /home/data/oc1809b 0.0 sec : adcp_cals: uhdas_info.py --cals --logfile /home/data/oc1809b/reports/ADCP_cals.txt /home/data/oc1809b 1.0 sec : ens_hcorr: plot_enshcorr.py --infile /home/data/oc1809b/proc/wh300/cal/rotate/ens_hcorr.asc --outfile /home/data/oc1809b/reports/wh300_hcorr --yearbase 2018 --titlestring ens_hcorr_adu 0.0 sec : ens_hcorr: convert -resize 200 /home/data/oc1809b/reports/wh300_hcorr.png /home/data/oc1809b/reports/wh300_hcorrT.png 1.0 sec : ens_hcorr: plot_enshcorr.py --infile /home/data/oc1809b/proc/os75nb/cal/rotate/ens_hcorr.asc --outfile /home/data/oc1809b/reports/os75nb_hcorr --yearbase 2018 --titlestring ens_hcorr_adu 0.0 sec : ens_hcorr: convert -resize 200 /home/data/oc1809b/reports/os75nb_hcorr.png /home/data/oc1809b/reports/os75nb_hcorrT.png 0.0 sec : pycfg: cp -p /home/data/oc1809b/raw/config/oc1809b_proc.py /home/data/oc1809b/reports/oc1809b_proc.py.txt 0.0 sec : pycfg: cp -p /home/data/oc1809b/raw/config/oc1809b_sensor.py /home/data/oc1809b/reports/oc1809b_sensor.py.txt 4.0 seconds elapsed html page starts here: /home/data/oc1809b/reports/index.html 2018-09-26 22:50:25,889 INFO repeater task report_generator ended after 3.9 seconds 2018-09-26 22:50:25,889 INFO repeater task report_generator next time_to_run is 2018/09/26 23:50:00 2018-09-26 22:50:30,894 INFO repeater task wh300_lastensq starting 26.9 seconds late 2018-09-26 22:50:33,037 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:50:33,037 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:49:59 2018-09-26 22:50:33,037 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:52:04 2018-09-26 22:50:33,038 INFO repeater task wh300_quick starting 29.0 seconds late 2018-09-26 22:50:36,367 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:50:36,367 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:49:59 2018-09-26 22:50:36,367 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:52:04 2018-09-26 22:50:36,367 INFO repeater task wh300_lastvec starting 32.4 seconds late 2018-09-26 22:50:36,735 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:50:36,735 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:49:59 2018-09-26 22:50:36,735 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:52:04 2018-09-26 22:50:36,735 INFO repeater task wh300_3dayplots starting 31.6 seconds late 2018-09-26 22:50:41,604 INFO repeater task wh300_3dayplots ended after 4.9 seconds 2018-09-26 22:50:41,604 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 22:55:05 2018-09-26 22:52:06,724 INFO repeater task wh300_lastensq starting 2.7 seconds late 2018-09-26 22:52:08,827 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:52:08,827 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:51:59 2018-09-26 22:52:08,827 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:54:04 2018-09-26 22:52:08,828 INFO repeater task wh300_quick starting 4.8 seconds late 2018-09-26 22:52:12,179 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 22:52:12,179 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:51:59 2018-09-26 22:52:12,179 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:54:04 2018-09-26 22:52:12,179 INFO repeater task wh300_lastvec starting 8.2 seconds late 2018-09-26 22:52:12,546 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:52:12,546 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:51:59 2018-09-26 22:52:12,546 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:54:04 2018-09-26 22:54:07,720 INFO repeater task wh300_lastensq starting 3.7 seconds late 2018-09-26 22:54:09,812 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:54:09,812 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:53:59 2018-09-26 22:54:09,812 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:56:04 2018-09-26 22:54:09,812 INFO repeater task wh300_quick starting 5.8 seconds late 2018-09-26 22:54:13,201 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 22:54:13,202 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:53:59 2018-09-26 22:54:13,202 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:56:04 2018-09-26 22:54:13,202 INFO repeater task wh300_lastvec starting 9.2 seconds late 2018-09-26 22:54:13,567 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:54:13,567 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:53:59 2018-09-26 22:54:13,567 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:56:04 2018-09-26 22:55:08,650 INFO repeater task wh300_3dayplots starting 3.6 seconds late 2018-09-26 22:55:13,874 INFO repeater task wh300_3dayplots ended after 5.2 seconds 2018-09-26 22:55:13,874 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:00:05 2018-09-26 22:55:13,875 INFO repeater task os75nb_lastensq starting 7.9 seconds late 2018-09-26 22:55:15,279 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 22:55:15,279 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:55:01 2018-09-26 22:55:15,279 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:00:06 2018-09-26 22:55:15,280 INFO repeater task os75nb_quick starting 9.3 seconds late 2018-09-26 22:55:18,180 INFO repeater task os75nb_quick ended after 2.9 seconds 2018-09-26 22:55:18,181 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:55:01 2018-09-26 22:55:18,181 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:00:06 2018-09-26 22:55:18,181 INFO repeater task os75nb_lastvec starting 12.2 seconds late 2018-09-26 22:55:18,557 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 22:55:18,557 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 22:55:01 2018-09-26 22:55:18,557 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:00:06 2018-09-26 22:55:18,557 INFO repeater task os75nb_3dayplots starting 13.5 seconds late 2018-09-26 22:55:23,124 INFO repeater task os75nb_3dayplots ended after 4.6 seconds 2018-09-26 22:55:23,124 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:00:05 2018-09-26 22:56:08,192 INFO repeater task wh300_lastensq starting 4.2 seconds late 2018-09-26 22:56:10,328 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:56:10,328 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:55:59 2018-09-26 22:56:10,328 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 22:58:04 2018-09-26 22:56:10,328 INFO repeater task wh300_quick starting 6.3 seconds late 2018-09-26 22:56:13,661 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 22:56:13,661 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:55:59 2018-09-26 22:56:13,662 INFO repeater task wh300_quick next time_to_run is 2018/09/26 22:58:04 2018-09-26 22:56:13,662 INFO repeater task wh300_lastvec starting 9.7 seconds late 2018-09-26 22:56:14,026 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:56:14,027 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:55:59 2018-09-26 22:56:14,027 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 22:58:04 2018-09-26 22:58:04,188 INFO repeater task wh300_lastensq starting 0.2 seconds late 2018-09-26 22:58:06,280 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 22:58:06,281 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:57:59 2018-09-26 22:58:06,281 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:00:04 2018-09-26 22:58:06,281 INFO repeater task wh300_quick starting 2.3 seconds late 2018-09-26 22:58:09,657 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 22:58:09,657 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:57:59 2018-09-26 22:58:09,657 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:00:04 2018-09-26 22:58:09,658 INFO repeater task wh300_lastvec starting 5.7 seconds late 2018-09-26 22:58:10,039 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 22:58:10,039 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:57:59 2018-09-26 22:58:10,039 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:00:04 2018-09-26 23:00:05,225 INFO repeater task wh300_lastensq starting 1.2 seconds late 2018-09-26 23:00:07,368 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:00:07,368 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:59:59 2018-09-26 23:00:07,368 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:02:04 2018-09-26 23:00:07,368 INFO repeater task wh300_quick starting 3.4 seconds late 2018-09-26 23:00:10,678 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:00:10,678 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:59:59 2018-09-26 23:00:10,678 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:02:04 2018-09-26 23:00:10,678 INFO repeater task wh300_lastvec starting 6.7 seconds late 2018-09-26 23:00:11,074 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:00:11,074 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 22:59:59 2018-09-26 23:00:11,074 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:02:04 2018-09-26 23:00:11,074 INFO repeater task wh300_3dayplots starting 6.0 seconds late 2018-09-26 23:00:16,051 INFO repeater task wh300_3dayplots ended after 5.0 seconds 2018-09-26 23:00:16,051 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:05:05 2018-09-26 23:00:16,052 INFO repeater task os75nb_lastensq starting 10.1 seconds late 2018-09-26 23:00:17,490 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:00:17,490 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:00:01 2018-09-26 23:00:17,490 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:05:06 2018-09-26 23:00:17,490 INFO repeater task os75nb_quick starting 11.5 seconds late 2018-09-26 23:00:20,360 INFO repeater task os75nb_quick ended after 2.9 seconds 2018-09-26 23:00:20,360 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:00:01 2018-09-26 23:00:20,361 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:05:06 2018-09-26 23:00:20,361 INFO repeater task os75nb_lastvec starting 14.4 seconds late 2018-09-26 23:00:20,733 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:00:20,733 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:00:01 2018-09-26 23:00:20,733 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:05:06 2018-09-26 23:00:20,733 INFO repeater task os75nb_3dayplots starting 15.6 seconds late 2018-09-26 23:00:25,383 INFO repeater task os75nb_3dayplots ended after 4.7 seconds 2018-09-26 23:00:25,383 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:05:05 2018-09-26 23:02:05,510 INFO repeater task wh300_lastensq starting 1.5 seconds late 2018-09-26 23:02:07,622 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:02:07,622 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:01:59 2018-09-26 23:02:07,622 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:04:04 2018-09-26 23:02:07,623 INFO repeater task wh300_quick starting 3.6 seconds late 2018-09-26 23:02:11,020 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:02:11,020 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:01:59 2018-09-26 23:02:11,020 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:04:04 2018-09-26 23:02:11,020 INFO repeater task wh300_lastvec starting 7.0 seconds late 2018-09-26 23:02:11,400 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:02:11,401 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:01:59 2018-09-26 23:02:11,401 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:04:04 2018-09-26 23:04:06,570 INFO repeater task wh300_lastensq starting 2.6 seconds late 2018-09-26 23:04:08,734 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-26 23:04:08,734 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:03:59 2018-09-26 23:04:08,734 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:06:04 2018-09-26 23:04:08,735 INFO repeater task wh300_quick starting 4.7 seconds late 2018-09-26 23:04:12,138 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:04:12,138 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:03:59 2018-09-26 23:04:12,138 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:06:04 2018-09-26 23:04:12,139 INFO repeater task wh300_lastvec starting 8.1 seconds late 2018-09-26 23:04:12,517 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:04:12,517 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:03:59 2018-09-26 23:04:12,517 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:06:04 2018-09-26 23:05:02,576 INFO repeater task wh300_tsstats starting 2.5 seconds late 2018-09-26 23:05:03,202 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-26 23:05:03,202 INFO repeater task wh300_tsstats next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:03,202 INFO repeater task wh300_10pings starting 3.1 seconds late 2018-09-26 23:05:03,309 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-26 23:05:03,309 INFO repeater task wh300_10pings next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:03,310 INFO repeater task wh300_beamplots starting 3.2 seconds late 2018-09-26 23:05:04,367 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 23:05:04,367 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:04,368 INFO repeater task wh300_beamplots starting 4.3 seconds late 2018-09-26 23:05:05,437 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 23:05:05,437 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:05,438 INFO repeater task os75nb_3dayplots starting 0.3 seconds late 2018-09-26 23:05:09,947 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 23:05:09,947 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:10:05 2018-09-26 23:05:09,948 INFO repeater task os75nb_tsstats starting 9.9 seconds late 2018-09-26 23:05:10,540 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2018-09-26 23:05:10,540 INFO repeater task os75nb_tsstats next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:10,540 INFO repeater task os75nb_10pings starting 10.4 seconds late 2018-09-26 23:05:10,650 INFO repeater task os75nb_10pings ended after 0.1 seconds 2018-09-26 23:05:10,651 INFO repeater task os75nb_10pings next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:10,651 INFO repeater task os75nb_beamplots starting 10.6 seconds late 2018-09-26 23:05:11,645 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 23:05:11,645 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:11,646 INFO repeater task os75nb_beamplots starting 11.6 seconds late 2018-09-26 23:05:12,686 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 23:05:12,686 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:12,686 INFO repeater task adu5_hcorrstats starting 12.4 seconds late 2018-09-26 23:05:13,496 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-26 23:05:13,496 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:13,496 INFO repeater task adu800_hcorrstats starting 13.2 seconds late 2018-09-26 23:05:15,034 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-26 23:05:15,034 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:15,034 INFO repeater task adu5_plotqc starting 14.8 seconds late 2018-09-26 23:05:15,994 INFO repeater task adu5_plotqc output: found 42 adu files in /home/data/oc1809b/rbin/adu5 2018-09-26 23:05:15,994 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-26 23:05:15,994 INFO repeater task adu5_plotqc next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:15,995 INFO repeater task adu800_plotqc starting 15.7 seconds late 2018-09-26 23:05:16,206 ERROR repeater task adu800_plotqc returncode 1 2018-09-26 23:05:16,206 INFO repeater task adu800_plotqc output: found 1 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-26 23:05:16,206 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-26 23:05:16,206 INFO repeater task adu800_plotqc next time_to_run is 2018/09/26 23:20:00 2018-09-26 23:05:21,212 INFO repeater task wh300_3dayplots starting 16.1 seconds late 2018-09-26 23:05:26,084 INFO repeater task wh300_3dayplots ended after 4.9 seconds 2018-09-26 23:05:26,085 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:10:05 2018-09-26 23:05:26,085 INFO repeater task os75nb_lastensq starting 20.1 seconds late 2018-09-26 23:05:27,544 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2018-09-26 23:05:27,545 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:05:01 2018-09-26 23:05:27,545 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:10:06 2018-09-26 23:05:27,545 INFO repeater task os75nb_quick starting 21.5 seconds late 2018-09-26 23:05:30,364 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 23:05:30,365 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:05:01 2018-09-26 23:05:30,365 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:10:06 2018-09-26 23:05:30,365 INFO repeater task os75nb_lastvec starting 24.4 seconds late 2018-09-26 23:05:30,736 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:05:30,737 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:05:01 2018-09-26 23:05:30,737 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:10:06 2018-09-26 23:06:05,788 INFO repeater task wh300_lastensq starting 1.8 seconds late 2018-09-26 23:06:07,920 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:06:07,920 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:05:59 2018-09-26 23:06:07,920 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:08:04 2018-09-26 23:06:07,920 INFO repeater task wh300_quick starting 3.9 seconds late 2018-09-26 23:06:11,255 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:06:11,255 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:05:59 2018-09-26 23:06:11,255 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:08:04 2018-09-26 23:06:11,255 INFO repeater task wh300_lastvec starting 7.3 seconds late 2018-09-26 23:06:11,638 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:06:11,638 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:05:59 2018-09-26 23:06:11,638 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:08:04 2018-09-26 23:08:06,797 INFO repeater task wh300_lastensq starting 2.8 seconds late 2018-09-26 23:08:08,896 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:08:08,896 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:07:59 2018-09-26 23:08:08,896 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:10:04 2018-09-26 23:08:08,897 INFO repeater task wh300_quick starting 4.9 seconds late 2018-09-26 23:08:12,290 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:08:12,290 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:07:59 2018-09-26 23:08:12,290 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:10:04 2018-09-26 23:08:12,290 INFO repeater task wh300_lastvec starting 8.3 seconds late 2018-09-26 23:08:12,677 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:08:12,677 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:07:59 2018-09-26 23:08:12,677 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:10:04 2018-09-26 23:10:07,848 INFO repeater task wh300_lastensq starting 3.8 seconds late 2018-09-26 23:10:09,983 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:10:09,983 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:09:59 2018-09-26 23:10:09,983 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:12:04 2018-09-26 23:10:09,983 INFO repeater task wh300_quick starting 6.0 seconds late 2018-09-26 23:10:13,304 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:10:13,304 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:09:59 2018-09-26 23:10:13,304 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:12:04 2018-09-26 23:10:13,305 INFO repeater task wh300_lastvec starting 9.3 seconds late 2018-09-26 23:10:13,685 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:10:13,686 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:09:59 2018-09-26 23:10:13,686 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:12:04 2018-09-26 23:10:13,686 INFO repeater task wh300_3dayplots starting 8.6 seconds late 2018-09-26 23:10:18,599 INFO repeater task wh300_3dayplots ended after 4.9 seconds 2018-09-26 23:10:18,599 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:15:05 2018-09-26 23:10:18,600 INFO repeater task os75nb_lastensq starting 12.6 seconds late 2018-09-26 23:10:19,977 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:10:19,977 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:10:01 2018-09-26 23:10:19,977 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:15:06 2018-09-26 23:10:19,978 INFO repeater task os75nb_quick starting 14.0 seconds late 2018-09-26 23:10:22,837 INFO repeater task os75nb_quick ended after 2.9 seconds 2018-09-26 23:10:22,837 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:10:01 2018-09-26 23:10:22,837 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:15:06 2018-09-26 23:10:22,838 INFO repeater task os75nb_lastvec starting 16.8 seconds late 2018-09-26 23:10:23,204 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:10:23,205 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:10:01 2018-09-26 23:10:23,205 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:15:06 2018-09-26 23:10:23,205 INFO repeater task os75nb_3dayplots starting 18.1 seconds late 2018-09-26 23:10:27,730 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 23:10:27,730 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:15:05 2018-09-26 23:12:07,896 INFO repeater task wh300_lastensq starting 3.9 seconds late 2018-09-26 23:12:10,016 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:12:10,016 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:11:59 2018-09-26 23:12:10,016 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:14:04 2018-09-26 23:12:10,016 INFO repeater task wh300_quick starting 6.0 seconds late 2018-09-26 23:12:13,366 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:12:13,366 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:11:59 2018-09-26 23:12:13,367 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:14:04 2018-09-26 23:12:13,367 INFO repeater task wh300_lastvec starting 9.4 seconds late 2018-09-26 23:12:13,747 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:12:13,747 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:11:59 2018-09-26 23:12:13,747 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:14:04 2018-09-26 23:14:08,916 INFO repeater task wh300_lastensq starting 4.9 seconds late 2018-09-26 23:14:11,037 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:14:11,037 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:13:59 2018-09-26 23:14:11,037 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:16:04 2018-09-26 23:14:11,038 INFO repeater task wh300_quick starting 7.0 seconds late 2018-09-26 23:14:14,374 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:14:14,374 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:13:59 2018-09-26 23:14:14,375 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:16:04 2018-09-26 23:14:14,375 INFO repeater task wh300_lastvec starting 10.4 seconds late 2018-09-26 23:14:14,759 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:14:14,759 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:13:59 2018-09-26 23:14:14,759 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:16:04 2018-09-26 23:15:09,849 INFO repeater task wh300_3dayplots starting 4.8 seconds late 2018-09-26 23:15:14,846 INFO repeater task wh300_3dayplots ended after 5.0 seconds 2018-09-26 23:15:14,846 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:20:05 2018-09-26 23:15:14,847 INFO repeater task os75nb_lastensq starting 8.8 seconds late 2018-09-26 23:15:16,231 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:15:16,231 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:15:01 2018-09-26 23:15:16,231 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:20:06 2018-09-26 23:15:16,232 INFO repeater task os75nb_quick starting 10.2 seconds late 2018-09-26 23:15:19,077 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 23:15:19,078 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:15:01 2018-09-26 23:15:19,078 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:20:06 2018-09-26 23:15:19,078 INFO repeater task os75nb_lastvec starting 13.1 seconds late 2018-09-26 23:15:19,443 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:15:19,444 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:15:01 2018-09-26 23:15:19,444 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:20:06 2018-09-26 23:15:19,444 INFO repeater task os75nb_3dayplots starting 14.4 seconds late 2018-09-26 23:15:23,954 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 23:15:23,954 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:20:05 2018-09-26 23:16:09,002 INFO repeater task wh300_lastensq starting 5.0 seconds late 2018-09-26 23:16:11,166 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-26 23:16:11,166 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:15:59 2018-09-26 23:16:11,166 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:18:04 2018-09-26 23:16:11,166 INFO repeater task wh300_quick starting 7.2 seconds late 2018-09-26 23:16:14,517 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:16:14,518 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:15:59 2018-09-26 23:16:14,518 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:18:04 2018-09-26 23:16:14,518 INFO repeater task wh300_lastvec starting 10.5 seconds late 2018-09-26 23:16:14,897 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:16:14,898 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:15:59 2018-09-26 23:16:14,898 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:18:04 2018-09-26 23:18:05,050 INFO repeater task wh300_lastensq starting 1.1 seconds late 2018-09-26 23:18:07,239 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-26 23:18:07,239 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:17:59 2018-09-26 23:18:07,239 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:20:04 2018-09-26 23:18:07,239 INFO repeater task wh300_quick starting 3.2 seconds late 2018-09-26 23:18:10,570 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:18:10,570 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:17:59 2018-09-26 23:18:10,570 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:20:04 2018-09-26 23:18:10,571 INFO repeater task wh300_lastvec starting 6.6 seconds late 2018-09-26 23:18:10,960 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:18:10,960 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:17:59 2018-09-26 23:18:10,960 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:20:04 2018-09-26 23:20:01,128 INFO repeater task wh300_tsstats starting 1.0 seconds late 2018-09-26 23:20:01,754 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-26 23:20:01,754 INFO repeater task wh300_tsstats next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:01,755 INFO repeater task wh300_10pings starting 1.7 seconds late 2018-09-26 23:20:01,864 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-26 23:20:01,864 INFO repeater task wh300_10pings next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:01,864 INFO repeater task wh300_beamplots starting 1.8 seconds late 2018-09-26 23:20:02,914 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 23:20:02,915 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:02,915 INFO repeater task wh300_beamplots starting 2.8 seconds late 2018-09-26 23:20:04,002 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 23:20:04,002 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:04,003 INFO repeater task os75nb_tsstats starting 3.9 seconds late 2018-09-26 23:20:04,589 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2018-09-26 23:20:04,589 INFO repeater task os75nb_tsstats next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:04,589 INFO repeater task os75nb_10pings starting 4.5 seconds late 2018-09-26 23:20:04,694 INFO repeater task os75nb_10pings ended after 0.1 seconds 2018-09-26 23:20:04,695 INFO repeater task os75nb_10pings next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:04,695 INFO repeater task os75nb_beamplots starting 4.6 seconds late 2018-09-26 23:20:05,733 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 23:20:05,733 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:05,733 INFO repeater task os75nb_beamplots starting 5.6 seconds late 2018-09-26 23:20:06,737 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 23:20:06,738 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:06,738 INFO repeater task adu5_hcorrstats starting 6.5 seconds late 2018-09-26 23:20:07,579 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-26 23:20:07,579 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:07,579 INFO repeater task adu800_hcorrstats starting 7.3 seconds late 2018-09-26 23:20:09,127 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-26 23:20:09,127 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:09,127 INFO repeater task adu5_plotqc starting 8.9 seconds late 2018-09-26 23:20:10,088 INFO repeater task adu5_plotqc output: found 42 adu files in /home/data/oc1809b/rbin/adu5 2018-09-26 23:20:10,088 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-26 23:20:10,088 INFO repeater task adu5_plotqc next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:10,088 INFO repeater task adu800_plotqc starting 9.8 seconds late 2018-09-26 23:20:10,298 ERROR repeater task adu800_plotqc returncode 1 2018-09-26 23:20:10,298 INFO repeater task adu800_plotqc output: found 1 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-26 23:20:10,298 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-26 23:20:10,298 INFO repeater task adu800_plotqc next time_to_run is 2018/09/26 23:35:00 2018-09-26 23:20:15,304 INFO repeater task wh300_lastensq starting 11.3 seconds late 2018-09-26 23:20:17,425 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:20:17,425 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:19:59 2018-09-26 23:20:17,425 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:22:04 2018-09-26 23:20:17,425 INFO repeater task wh300_quick starting 13.4 seconds late 2018-09-26 23:20:20,768 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:20:20,769 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:19:59 2018-09-26 23:20:20,769 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:22:04 2018-09-26 23:20:20,769 INFO repeater task wh300_lastvec starting 16.8 seconds late 2018-09-26 23:20:21,144 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:20:21,144 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:19:59 2018-09-26 23:20:21,144 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:22:04 2018-09-26 23:20:21,144 INFO repeater task wh300_3dayplots starting 16.1 seconds late 2018-09-26 23:20:26,088 INFO repeater task wh300_3dayplots ended after 4.9 seconds 2018-09-26 23:20:26,088 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:25:05 2018-09-26 23:20:26,089 INFO repeater task os75nb_lastensq starting 20.1 seconds late 2018-09-26 23:20:27,476 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:20:27,476 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:20:01 2018-09-26 23:20:27,476 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:25:06 2018-09-26 23:20:27,477 INFO repeater task os75nb_quick starting 21.5 seconds late 2018-09-26 23:20:30,263 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 23:20:30,264 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:20:01 2018-09-26 23:20:30,264 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:25:06 2018-09-26 23:20:30,264 INFO repeater task os75nb_lastvec starting 24.3 seconds late 2018-09-26 23:20:30,638 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:20:30,638 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:20:01 2018-09-26 23:20:30,638 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:25:06 2018-09-26 23:20:30,639 INFO repeater task os75nb_3dayplots starting 25.5 seconds late 2018-09-26 23:20:35,155 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 23:20:35,156 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:25:05 2018-09-26 23:22:05,297 INFO repeater task wh300_lastensq starting 1.3 seconds late 2018-09-26 23:22:07,440 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:22:07,441 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:21:59 2018-09-26 23:22:07,441 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:24:04 2018-09-26 23:22:07,441 INFO repeater task wh300_quick starting 3.4 seconds late 2018-09-26 23:22:10,769 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:22:10,769 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:21:59 2018-09-26 23:22:10,769 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:24:04 2018-09-26 23:22:10,769 INFO repeater task wh300_lastvec starting 6.8 seconds late 2018-09-26 23:22:11,147 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:22:11,147 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:21:59 2018-09-26 23:22:11,147 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:24:04 2018-09-26 23:24:06,312 INFO repeater task wh300_lastensq starting 2.3 seconds late 2018-09-26 23:24:08,545 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-26 23:24:08,546 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:23:59 2018-09-26 23:24:08,546 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:26:04 2018-09-26 23:24:08,546 INFO repeater task wh300_quick starting 4.5 seconds late 2018-09-26 23:24:11,900 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:24:11,900 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:23:59 2018-09-26 23:24:11,900 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:26:04 2018-09-26 23:24:11,901 INFO repeater task wh300_lastvec starting 7.9 seconds late 2018-09-26 23:24:12,287 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:24:12,287 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:23:59 2018-09-26 23:24:12,287 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:26:04 2018-09-26 23:25:07,373 INFO repeater task wh300_3dayplots starting 2.3 seconds late 2018-09-26 23:25:12,368 INFO repeater task wh300_3dayplots ended after 5.0 seconds 2018-09-26 23:25:12,368 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:30:05 2018-09-26 23:25:12,369 INFO repeater task os75nb_lastensq starting 6.4 seconds late 2018-09-26 23:25:13,760 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:25:13,760 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:25:01 2018-09-26 23:25:13,760 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:30:06 2018-09-26 23:25:13,760 INFO repeater task os75nb_quick starting 7.8 seconds late 2018-09-26 23:25:16,696 INFO repeater task os75nb_quick ended after 2.9 seconds 2018-09-26 23:25:16,696 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:25:01 2018-09-26 23:25:16,696 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:30:06 2018-09-26 23:25:16,696 INFO repeater task os75nb_lastvec starting 10.7 seconds late 2018-09-26 23:25:17,067 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:25:17,067 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:25:01 2018-09-26 23:25:17,067 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:30:06 2018-09-26 23:25:17,067 INFO repeater task os75nb_3dayplots starting 12.0 seconds late 2018-09-26 23:25:21,516 INFO repeater task os75nb_3dayplots ended after 4.4 seconds 2018-09-26 23:25:21,516 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:30:05 2018-09-26 23:26:06,583 INFO repeater task wh300_lastensq starting 2.6 seconds late 2018-09-26 23:26:08,705 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:26:08,706 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:25:59 2018-09-26 23:26:08,706 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:28:04 2018-09-26 23:26:08,706 INFO repeater task wh300_quick starting 4.7 seconds late 2018-09-26 23:26:12,062 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:26:12,062 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:25:59 2018-09-26 23:26:12,062 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:28:04 2018-09-26 23:26:12,062 INFO repeater task wh300_lastvec starting 8.1 seconds late 2018-09-26 23:26:12,444 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:26:12,445 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:25:59 2018-09-26 23:26:12,445 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:28:04 2018-09-26 23:28:07,601 INFO repeater task wh300_lastensq starting 3.6 seconds late 2018-09-26 23:28:09,742 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:28:09,742 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:27:59 2018-09-26 23:28:09,742 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:30:04 2018-09-26 23:28:09,743 INFO repeater task wh300_quick starting 5.7 seconds late 2018-09-26 23:28:13,049 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:28:13,049 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:27:59 2018-09-26 23:28:13,049 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:30:04 2018-09-26 23:28:13,050 INFO repeater task wh300_lastvec starting 9.1 seconds late 2018-09-26 23:28:13,440 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:28:13,440 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:27:59 2018-09-26 23:28:13,440 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:30:04 2018-09-26 23:30:08,606 INFO repeater task wh300_lastensq starting 4.6 seconds late 2018-09-26 23:30:10,711 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:30:10,711 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:29:59 2018-09-26 23:30:10,712 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:32:04 2018-09-26 23:30:10,712 INFO repeater task wh300_quick starting 6.7 seconds late 2018-09-26 23:30:14,032 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:30:14,033 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:29:59 2018-09-26 23:30:14,033 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:32:04 2018-09-26 23:30:14,033 INFO repeater task wh300_lastvec starting 10.0 seconds late 2018-09-26 23:30:14,415 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:30:14,416 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:29:59 2018-09-26 23:30:14,416 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:32:04 2018-09-26 23:30:14,416 INFO repeater task wh300_3dayplots starting 9.3 seconds late 2018-09-26 23:30:19,392 INFO repeater task wh300_3dayplots ended after 5.0 seconds 2018-09-26 23:30:19,392 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:35:05 2018-09-26 23:30:19,393 INFO repeater task os75nb_lastensq starting 13.4 seconds late 2018-09-26 23:30:20,771 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:30:20,771 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:30:01 2018-09-26 23:30:20,771 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:35:06 2018-09-26 23:30:20,771 INFO repeater task os75nb_quick starting 14.8 seconds late 2018-09-26 23:30:23,600 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 23:30:23,600 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:30:01 2018-09-26 23:30:23,600 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:35:06 2018-09-26 23:30:23,600 INFO repeater task os75nb_lastvec starting 17.6 seconds late 2018-09-26 23:30:23,969 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:30:23,970 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:30:01 2018-09-26 23:30:23,970 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:35:06 2018-09-26 23:30:23,970 INFO repeater task os75nb_3dayplots starting 18.9 seconds late 2018-09-26 23:30:28,520 INFO repeater task os75nb_3dayplots ended after 4.6 seconds 2018-09-26 23:30:28,520 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:35:05 2018-09-26 23:32:08,660 INFO repeater task wh300_lastensq starting 4.7 seconds late 2018-09-26 23:32:10,776 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:32:10,776 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:31:59 2018-09-26 23:32:10,777 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:34:04 2018-09-26 23:32:10,777 INFO repeater task wh300_quick starting 6.8 seconds late 2018-09-26 23:32:14,124 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:32:14,124 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:31:59 2018-09-26 23:32:14,124 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:34:04 2018-09-26 23:32:14,125 INFO repeater task wh300_lastvec starting 10.1 seconds late 2018-09-26 23:32:14,520 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:32:14,521 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:31:59 2018-09-26 23:32:14,521 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:34:04 2018-09-26 23:34:04,668 INFO repeater task wh300_lastensq starting 0.7 seconds late 2018-09-26 23:34:06,799 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:34:06,799 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:33:59 2018-09-26 23:34:06,799 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:36:04 2018-09-26 23:34:06,799 INFO repeater task wh300_quick starting 2.8 seconds late 2018-09-26 23:34:10,152 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:34:10,152 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:33:59 2018-09-26 23:34:10,152 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:36:04 2018-09-26 23:34:10,153 INFO repeater task wh300_lastvec starting 6.2 seconds late 2018-09-26 23:34:10,538 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:34:10,538 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:33:59 2018-09-26 23:34:10,538 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:36:04 2018-09-26 23:35:00,604 INFO repeater task wh300_tsstats starting 0.5 seconds late 2018-09-26 23:35:01,229 INFO repeater task wh300_tsstats ended after 0.6 seconds 2018-09-26 23:35:01,229 INFO repeater task wh300_tsstats next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:01,230 INFO repeater task wh300_10pings starting 1.1 seconds late 2018-09-26 23:35:01,338 INFO repeater task wh300_10pings ended after 0.1 seconds 2018-09-26 23:35:01,338 INFO repeater task wh300_10pings next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:01,338 INFO repeater task wh300_beamplots starting 1.2 seconds late 2018-09-26 23:35:02,391 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 23:35:02,391 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:02,391 INFO repeater task wh300_beamplots starting 2.3 seconds late 2018-09-26 23:35:03,447 INFO repeater task wh300_beamplots ended after 1.1 seconds 2018-09-26 23:35:03,447 INFO repeater task wh300_beamplots next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:03,448 INFO repeater task os75nb_tsstats starting 3.4 seconds late 2018-09-26 23:35:04,055 INFO repeater task os75nb_tsstats ended after 0.6 seconds 2018-09-26 23:35:04,055 INFO repeater task os75nb_tsstats next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:04,055 INFO repeater task os75nb_10pings starting 4.0 seconds late 2018-09-26 23:35:04,165 INFO repeater task os75nb_10pings ended after 0.1 seconds 2018-09-26 23:35:04,165 INFO repeater task os75nb_10pings next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:04,165 INFO repeater task os75nb_beamplots starting 4.1 seconds late 2018-09-26 23:35:05,193 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 23:35:05,193 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:05,193 INFO repeater task os75nb_beamplots starting 5.1 seconds late 2018-09-26 23:35:06,178 INFO repeater task os75nb_beamplots ended after 1.0 seconds 2018-09-26 23:35:06,178 INFO repeater task os75nb_beamplots next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:06,178 INFO repeater task adu5_hcorrstats starting 5.9 seconds late 2018-09-26 23:35:06,986 INFO repeater task adu5_hcorrstats ended after 0.8 seconds 2018-09-26 23:35:06,986 INFO repeater task adu5_hcorrstats next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:06,987 INFO repeater task adu800_hcorrstats starting 6.7 seconds late 2018-09-26 23:35:08,534 INFO repeater task adu800_hcorrstats ended after 1.5 seconds 2018-09-26 23:35:08,534 INFO repeater task adu800_hcorrstats next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:08,535 INFO repeater task adu5_plotqc starting 8.3 seconds late 2018-09-26 23:35:09,493 INFO repeater task adu5_plotqc output: found 42 adu files in /home/data/oc1809b/rbin/adu5 2018-09-26 23:35:09,494 INFO repeater task adu5_plotqc ended after 1.0 seconds 2018-09-26 23:35:09,494 INFO repeater task adu5_plotqc next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:09,494 INFO repeater task adu800_plotqc starting 9.2 seconds late 2018-09-26 23:35:09,701 ERROR repeater task adu800_plotqc returncode 1 2018-09-26 23:35:09,701 INFO repeater task adu800_plotqc output: found 1 adu files in /home/data/oc1809b/rbin/adu800 Traceback (most recent call last): File "/usr/local/currents/bin/run_plot_ashqc.py", line 80, in nhours = float(options.nhours)) File "/usr/local/currents/bin/run_plot_ashqc.py", line 51, in get_adu last_dday = adu.ends[adu.cname][-1] IndexError: index -1 is out of bounds for axis 0 with size 0 2018-09-26 23:35:09,701 INFO repeater task adu800_plotqc ended after 0.2 seconds 2018-09-26 23:35:09,701 INFO repeater task adu800_plotqc next time_to_run is 2018/09/26 23:50:00 2018-09-26 23:35:14,707 INFO repeater task wh300_3dayplots starting 9.6 seconds late 2018-09-26 23:35:19,864 INFO repeater task wh300_3dayplots ended after 5.2 seconds 2018-09-26 23:35:19,865 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:40:05 2018-09-26 23:35:19,866 INFO repeater task os75nb_lastensq starting 13.9 seconds late 2018-09-26 23:35:21,311 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:35:21,311 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:35:01 2018-09-26 23:35:21,311 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:40:06 2018-09-26 23:35:21,311 INFO repeater task os75nb_quick starting 15.3 seconds late 2018-09-26 23:35:24,117 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 23:35:24,117 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:35:01 2018-09-26 23:35:24,117 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:40:06 2018-09-26 23:35:24,118 INFO repeater task os75nb_lastvec starting 18.1 seconds late 2018-09-26 23:35:24,482 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:35:24,482 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:35:01 2018-09-26 23:35:24,482 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:40:06 2018-09-26 23:35:24,483 INFO repeater task os75nb_3dayplots starting 19.4 seconds late 2018-09-26 23:35:28,924 INFO repeater task os75nb_3dayplots ended after 4.4 seconds 2018-09-26 23:35:28,924 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:40:05 2018-09-26 23:36:08,980 INFO repeater task wh300_lastensq starting 5.0 seconds late 2018-09-26 23:36:11,171 INFO repeater task wh300_lastensq ended after 2.2 seconds 2018-09-26 23:36:11,171 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:35:59 2018-09-26 23:36:11,171 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:38:04 2018-09-26 23:36:11,171 INFO repeater task wh300_quick starting 7.2 seconds late 2018-09-26 23:36:14,554 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:36:14,554 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:35:59 2018-09-26 23:36:14,554 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:38:04 2018-09-26 23:36:14,554 INFO repeater task wh300_lastvec starting 10.6 seconds late 2018-09-26 23:36:14,938 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:36:14,939 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:35:59 2018-09-26 23:36:14,939 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:38:04 2018-09-26 23:38:05,091 INFO repeater task wh300_lastensq starting 1.1 seconds late 2018-09-26 23:38:07,206 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:38:07,206 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:37:59 2018-09-26 23:38:07,206 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:40:04 2018-09-26 23:38:07,206 INFO repeater task wh300_quick starting 3.2 seconds late 2018-09-26 23:38:10,538 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:38:10,538 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:37:59 2018-09-26 23:38:10,538 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:40:04 2018-09-26 23:38:10,539 INFO repeater task wh300_lastvec starting 6.5 seconds late 2018-09-26 23:38:10,934 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:38:10,934 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:37:59 2018-09-26 23:38:10,934 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:40:04 2018-09-26 23:40:06,103 INFO repeater task wh300_lastensq starting 2.1 seconds late 2018-09-26 23:40:08,219 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:40:08,219 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:39:59 2018-09-26 23:40:08,219 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:42:04 2018-09-26 23:40:08,219 INFO repeater task wh300_quick starting 4.2 seconds late 2018-09-26 23:40:11,569 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:40:11,569 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:39:59 2018-09-26 23:40:11,569 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:42:04 2018-09-26 23:40:11,569 INFO repeater task wh300_lastvec starting 7.6 seconds late 2018-09-26 23:40:11,946 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:40:11,946 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:39:59 2018-09-26 23:40:11,946 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:42:04 2018-09-26 23:40:11,947 INFO repeater task wh300_3dayplots starting 6.9 seconds late 2018-09-26 23:40:16,970 INFO repeater task wh300_3dayplots ended after 5.0 seconds 2018-09-26 23:40:16,970 INFO repeater task wh300_3dayplots next time_to_run is 2018/09/26 23:45:05 2018-09-26 23:40:16,971 INFO repeater task os75nb_lastensq starting 11.0 seconds late 2018-09-26 23:40:18,362 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2018-09-26 23:40:18,362 INFO repeater task os75nb_lastensq: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:40:01 2018-09-26 23:40:18,363 INFO repeater task os75nb_lastensq next time_to_run is 2018/09/26 23:45:06 2018-09-26 23:40:18,363 INFO repeater task os75nb_quick starting 12.4 seconds late 2018-09-26 23:40:21,190 INFO repeater task os75nb_quick ended after 2.8 seconds 2018-09-26 23:40:21,190 INFO repeater task os75nb_quick: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:40:01 2018-09-26 23:40:21,191 INFO repeater task os75nb_quick next time_to_run is 2018/09/26 23:45:06 2018-09-26 23:40:21,191 INFO repeater task os75nb_lastvec starting 15.2 seconds late 2018-09-26 23:40:21,572 INFO repeater task os75nb_lastvec ended after 0.4 seconds 2018-09-26 23:40:21,572 INFO repeater task os75nb_lastvec: Time from /home/data/oc1809b/proc/os75nb/load/wrote_ens_time: 2018/09/26 23:40:01 2018-09-26 23:40:21,572 INFO repeater task os75nb_lastvec next time_to_run is 2018/09/26 23:45:06 2018-09-26 23:40:21,572 INFO repeater task os75nb_3dayplots starting 16.5 seconds late 2018-09-26 23:40:26,050 INFO repeater task os75nb_3dayplots ended after 4.5 seconds 2018-09-26 23:40:26,051 INFO repeater task os75nb_3dayplots next time_to_run is 2018/09/26 23:45:05 2018-09-26 23:42:06,212 INFO repeater task wh300_lastensq starting 2.2 seconds late 2018-09-26 23:42:08,326 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:42:08,327 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:41:59 2018-09-26 23:42:08,327 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:44:04 2018-09-26 23:42:08,327 INFO repeater task wh300_quick starting 4.3 seconds late 2018-09-26 23:42:11,672 INFO repeater task wh300_quick ended after 3.3 seconds 2018-09-26 23:42:11,673 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:41:59 2018-09-26 23:42:11,673 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:44:04 2018-09-26 23:42:11,673 INFO repeater task wh300_lastvec starting 7.7 seconds late 2018-09-26 23:42:12,056 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:42:12,056 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:41:59 2018-09-26 23:42:12,056 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:44:04 2018-09-26 23:44:07,225 INFO repeater task wh300_lastensq starting 3.2 seconds late 2018-09-26 23:44:09,329 INFO repeater task wh300_lastensq ended after 2.1 seconds 2018-09-26 23:44:09,329 INFO repeater task wh300_lastensq: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:43:59 2018-09-26 23:44:09,329 INFO repeater task wh300_lastensq next time_to_run is 2018/09/26 23:46:04 2018-09-26 23:44:09,330 INFO repeater task wh300_quick starting 5.3 seconds late 2018-09-26 23:44:12,707 INFO repeater task wh300_quick ended after 3.4 seconds 2018-09-26 23:44:12,708 INFO repeater task wh300_quick: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:43:59 2018-09-26 23:44:12,708 INFO repeater task wh300_quick next time_to_run is 2018/09/26 23:46:04 2018-09-26 23:44:12,708 INFO repeater task wh300_lastvec starting 8.7 seconds late 2018-09-26 23:44:13,083 INFO repeater task wh300_lastvec ended after 0.4 seconds 2018-09-26 23:44:13,084 INFO repeater task wh300_lastvec: Time from /home/data/oc1809b/proc/wh300/load/wrote_ens_time: 2018/09/26 23:43:59 2018-09-26 23:44:13,084 INFO repeater task wh300_lastvec next time_to_run is 2018/09/26 23:46:04 2018-09-26 23:45:08,174 INFO repeater task wh300_3dayplots starting 3.1 seconds late