2015-03-18 15:54:07,018 INFO repeater task phins3_hcorrstats starting 31.1 seconds late 2015-03-18 15:54:07,748 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 15:54:12,750 INFO repeater task os150nb_10pings starting 16.9 seconds late 2015-03-18 15:54:12,873 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 15:54:47,922 INFO repeater task os150nb_lastensq starting 1.9 seconds late 2015-03-18 15:54:49,748 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 15:54:49,748 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 15:54:36 2015-03-18 15:55:19,785 INFO repeater task os75nb_10pings starting 3.9 seconds late 2015-03-18 15:55:19,913 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 15:56:19,986 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-03-18 15:56:20,115 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 15:56:50,158 INFO repeater task os150nb_lastensq starting 4.2 seconds late 2015-03-18 15:56:52,057 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-18 15:56:52,057 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 15:56:36 2015-03-18 15:56:52,058 INFO repeater task os75nb_lastensq starting 6.1 seconds late 2015-03-18 15:56:53,551 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-18 15:56:53,551 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 15:56:36 2015-03-18 15:58:38,680 INFO repeater task os150nb_10pings starting 2.8 seconds late 2015-03-18 15:58:38,803 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 15:58:38,804 INFO repeater task os75nb_10pings starting 2.9 seconds late 2015-03-18 15:58:38,927 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 15:58:38,928 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2015-03-18 15:58:39,733 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 15:58:39,733 INFO repeater task phins3_hcorrstats starting 3.8 seconds late 2015-03-18 15:58:40,480 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 15:58:50,493 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2015-03-18 15:58:52,333 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 15:58:52,333 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 15:58:36 2015-03-18 15:59:47,411 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2015-03-18 15:59:48,846 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 15:59:48,846 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 15:59:36 2015-03-18 16:00:48,970 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-03-18 16:00:50,804 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:00:50,805 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:00:36 2015-03-18 16:01:00,818 INFO repeater task os150nb_10pings starting 4.9 seconds late 2015-03-18 16:01:00,947 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:01:56,025 INFO repeater task os75nb_10pings starting 0.1 seconds late 2015-03-18 16:01:56,151 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:02:46,224 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-18 16:02:47,984 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:02:47,984 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:02:36 2015-03-18 16:02:47,985 INFO repeater task os75nb_lastensq starting 2.0 seconds late 2015-03-18 16:02:49,408 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:02:49,408 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:02:36 2015-03-18 16:03:19,449 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-03-18 16:03:19,578 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:03:39,605 INFO repeater task ashtech_hcorrstats starting 3.7 seconds late 2015-03-18 16:03:40,393 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:03:40,393 INFO repeater task phins3_hcorrstats starting 4.5 seconds late 2015-03-18 16:03:41,148 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 16:04:46,242 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-18 16:04:47,971 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:04:47,971 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:04:36 2015-03-18 16:05:18,011 INFO repeater task os75nb_10pings starting 2.1 seconds late 2015-03-18 16:05:18,137 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:05:38,162 INFO repeater task os150nb_10pings starting 2.3 seconds late 2015-03-18 16:05:38,297 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:05:48,312 INFO repeater task os75nb_lastensq starting 2.3 seconds late 2015-03-18 16:05:49,668 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:05:49,668 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:05:36 2015-03-18 16:06:49,753 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2015-03-18 16:06:51,500 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:06:51,500 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:06:36 2015-03-18 16:07:56,587 INFO repeater task os150nb_10pings starting 0.7 seconds late 2015-03-18 16:07:56,708 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:08:36,763 INFO repeater task os150nb_quick starting 0.9 seconds late 2015-03-18 16:08:41,115 INFO repeater task os150nb_quick ended after 4.4 seconds 2015-03-18 16:08:41,116 INFO repeater task os150nb_3dayplots starting 5.2 seconds late 2015-03-18 16:08:46,569 INFO repeater task os150nb_3dayplots ended after 5.5 seconds 2015-03-18 16:08:46,569 INFO repeater task os150nb_tsstats starting 10.7 seconds late 2015-03-18 16:08:47,346 INFO repeater task os150nb_tsstats ended after 0.8 seconds 2015-03-18 16:08:47,347 INFO repeater task os75nb_lastensq starting 1.3 seconds late 2015-03-18 16:08:48,686 INFO repeater task os75nb_lastensq ended after 1.3 seconds 2015-03-18 16:08:48,686 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:08:36 2015-03-18 16:08:48,687 INFO repeater task os75nb_10pings starting 12.8 seconds late 2015-03-18 16:08:48,812 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:08:48,813 INFO repeater task os75nb_quick starting 12.9 seconds late 2015-03-18 16:08:52,702 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-18 16:08:52,702 INFO repeater task os75nb_3dayplots starting 16.8 seconds late 2015-03-18 16:08:58,159 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 16:08:58,159 INFO repeater task os75nb_3dayplots ended after 5.5 seconds 2015-03-18 16:08:58,159 INFO repeater task os75nb_tsstats starting 22.3 seconds late 2015-03-18 16:08:58,894 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-18 16:08:58,894 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 16:08:58,894 INFO repeater task os150nb_beamplots starting 23.0 seconds late 2015-03-18 16:09:00,082 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:09:00,082 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:09:00,083 INFO repeater task os150nb_beamplots starting 24.2 seconds late 2015-03-18 16:09:01,249 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:09:01,249 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:09:01,250 INFO repeater task os75nb_beamplots starting 25.3 seconds late 2015-03-18 16:09:02,352 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:09:02,352 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 16:09:02,353 INFO repeater task os75nb_beamplots starting 26.5 seconds late 2015-03-18 16:09:03,488 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:09:03,489 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 16:09:03,489 INFO repeater task ashtech_hcorrstats starting 27.6 seconds late 2015-03-18 16:09:04,274 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:09:04,274 INFO repeater task phins3_hcorrstats starting 28.4 seconds late 2015-03-18 16:09:05,007 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:09:10,010 INFO repeater task os150nb_lastensq starting 24.0 seconds late 2015-03-18 16:09:11,745 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:09:11,746 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:08:36 2015-03-18 16:10:16,836 INFO repeater task os150nb_10pings starting 0.9 seconds late 2015-03-18 16:10:16,973 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:10:47,016 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2015-03-18 16:10:48,761 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:10:48,761 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:10:36 2015-03-18 16:11:48,842 INFO repeater task os75nb_lastensq starting 2.8 seconds late 2015-03-18 16:11:50,295 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-18 16:11:50,295 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:11:36 2015-03-18 16:12:00,306 INFO repeater task os75nb_10pings starting 4.4 seconds late 2015-03-18 16:12:00,429 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:12:40,484 INFO repeater task os150nb_10pings starting 4.6 seconds late 2015-03-18 16:12:40,608 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:12:50,618 INFO repeater task os150nb_lastensq starting 4.6 seconds late 2015-03-18 16:12:52,385 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:12:52,385 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:12:36 2015-03-18 16:13:37,444 INFO repeater task ashtech_hcorrstats starting 1.5 seconds late 2015-03-18 16:13:38,230 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:13:38,230 INFO repeater task phins3_hcorrstats starting 2.3 seconds late 2015-03-18 16:13:38,960 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:14:49,048 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-03-18 16:14:50,779 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:14:50,779 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:14:36 2015-03-18 16:14:50,780 INFO repeater task os75nb_lastensq starting 4.8 seconds late 2015-03-18 16:14:52,158 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:14:52,159 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:14:36 2015-03-18 16:14:57,165 INFO repeater task os150nb_10pings starting 1.3 seconds late 2015-03-18 16:14:57,289 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:15:17,316 INFO repeater task os75nb_10pings starting 1.4 seconds late 2015-03-18 16:15:17,457 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:16:47,586 INFO repeater task os150nb_lastensq starting 1.6 seconds late 2015-03-18 16:16:49,343 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:16:49,343 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:16:36 2015-03-18 16:17:19,380 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-03-18 16:17:19,507 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:17:49,551 INFO repeater task os75nb_lastensq starting 3.6 seconds late 2015-03-18 16:17:50,927 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:17:50,927 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:17:36 2015-03-18 16:18:35,981 INFO repeater task os75nb_10pings starting 0.1 seconds late 2015-03-18 16:18:36,107 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:18:36,108 INFO repeater task ashtech_hcorrstats starting 0.2 seconds late 2015-03-18 16:18:36,931 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:18:36,931 INFO repeater task phins3_hcorrstats starting 1.0 seconds late 2015-03-18 16:18:37,665 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:18:47,679 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-03-18 16:18:49,482 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:18:49,482 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:18:36 2015-03-18 16:19:39,554 INFO repeater task os150nb_10pings starting 3.7 seconds late 2015-03-18 16:19:39,678 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:20:49,758 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2015-03-18 16:20:51,490 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:20:51,491 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:20:36 2015-03-18 16:20:51,492 INFO repeater task os75nb_lastensq starting 5.5 seconds late 2015-03-18 16:20:52,887 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:20:52,888 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:20:36 2015-03-18 16:21:57,967 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-03-18 16:21:58,097 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:21:58,098 INFO repeater task os75nb_10pings starting 2.2 seconds late 2015-03-18 16:21:58,223 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:22:48,291 INFO repeater task os150nb_lastensq starting 2.3 seconds late 2015-03-18 16:22:50,129 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:22:50,129 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:22:36 2015-03-18 16:23:40,203 INFO repeater task os150nb_quick starting 4.3 seconds late 2015-03-18 16:23:44,751 INFO repeater task os150nb_quick ended after 4.5 seconds 2015-03-18 16:23:44,752 INFO repeater task os150nb_3dayplots starting 8.9 seconds late 2015-03-18 16:23:50,195 INFO repeater task os150nb_3dayplots ended after 5.4 seconds 2015-03-18 16:23:50,195 INFO repeater task os150nb_tsstats starting 14.3 seconds late 2015-03-18 16:23:50,916 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-18 16:23:50,916 INFO repeater task os75nb_lastensq starting 4.9 seconds late 2015-03-18 16:23:52,286 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:23:52,287 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:23:36 2015-03-18 16:23:52,287 INFO repeater task os75nb_quick starting 16.4 seconds late 2015-03-18 16:23:56,228 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-18 16:23:56,229 INFO repeater task os75nb_3dayplots starting 20.3 seconds late 2015-03-18 16:24:01,768 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 16:24:01,768 INFO repeater task os75nb_3dayplots ended after 5.5 seconds 2015-03-18 16:24:01,769 INFO repeater task os75nb_tsstats starting 25.9 seconds late 2015-03-18 16:24:02,455 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-18 16:24:02,455 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 16:24:02,455 INFO repeater task os150nb_beamplots starting 26.6 seconds late 2015-03-18 16:24:03,631 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:24:03,632 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:24:03,632 INFO repeater task os150nb_beamplots starting 27.7 seconds late 2015-03-18 16:24:04,882 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:24:04,882 INFO repeater task os150nb_beamplots ended after 1.3 seconds 2015-03-18 16:24:04,882 INFO repeater task os75nb_beamplots starting 29.0 seconds late 2015-03-18 16:24:06,002 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:24:06,002 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 16:24:06,003 INFO repeater task os75nb_beamplots starting 30.1 seconds late 2015-03-18 16:24:07,090 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:24:07,090 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 16:24:07,091 INFO repeater task ashtech_hcorrstats starting 31.2 seconds late 2015-03-18 16:24:07,876 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:24:07,877 INFO repeater task phins3_hcorrstats starting 32.0 seconds late 2015-03-18 16:24:08,607 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:24:18,620 INFO repeater task os150nb_10pings starting 2.7 seconds late 2015-03-18 16:24:18,748 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:24:48,794 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2015-03-18 16:24:50,682 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-18 16:24:50,682 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:24:36 2015-03-18 16:25:20,723 INFO repeater task os75nb_10pings starting 4.8 seconds late 2015-03-18 16:25:20,851 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:26:35,949 INFO repeater task os150nb_10pings starting 0.1 seconds late 2015-03-18 16:26:36,077 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:26:46,087 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-03-18 16:26:47,807 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:26:47,807 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:26:36 2015-03-18 16:26:47,808 INFO repeater task os75nb_lastensq starting 1.8 seconds late 2015-03-18 16:26:49,203 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:26:49,204 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:26:36 2015-03-18 16:28:39,334 INFO repeater task os75nb_10pings starting 3.4 seconds late 2015-03-18 16:28:39,475 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:28:39,477 INFO repeater task ashtech_hcorrstats starting 3.6 seconds late 2015-03-18 16:28:40,251 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:28:40,252 INFO repeater task phins3_hcorrstats starting 4.4 seconds late 2015-03-18 16:28:40,993 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:28:51,006 INFO repeater task os150nb_lastensq starting 5.0 seconds late 2015-03-18 16:28:52,734 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:28:52,734 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:28:36 2015-03-18 16:28:57,742 INFO repeater task os150nb_10pings starting 1.8 seconds late 2015-03-18 16:28:57,867 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:29:47,930 INFO repeater task os75nb_lastensq starting 1.9 seconds late 2015-03-18 16:29:49,264 INFO repeater task os75nb_lastensq ended after 1.3 seconds 2015-03-18 16:29:49,264 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:29:36 2015-03-18 16:30:49,339 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2015-03-18 16:30:51,186 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:30:51,186 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:30:36 2015-03-18 16:31:16,218 INFO repeater task os150nb_10pings starting 0.3 seconds late 2015-03-18 16:31:16,342 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:31:56,397 INFO repeater task os75nb_10pings starting 0.5 seconds late 2015-03-18 16:31:56,520 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:32:46,582 INFO repeater task os150nb_lastensq starting 0.6 seconds late 2015-03-18 16:32:48,361 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:32:48,362 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:32:36 2015-03-18 16:32:48,363 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2015-03-18 16:32:49,725 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:32:49,725 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:32:36 2015-03-18 16:33:39,792 INFO repeater task os150nb_10pings starting 3.9 seconds late 2015-03-18 16:33:39,914 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:33:39,916 INFO repeater task ashtech_hcorrstats starting 4.0 seconds late 2015-03-18 16:33:40,757 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:33:40,757 INFO repeater task phins3_hcorrstats starting 4.9 seconds late 2015-03-18 16:33:41,493 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:34:46,582 INFO repeater task os150nb_lastensq starting 0.6 seconds late 2015-03-18 16:34:48,339 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:34:48,340 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:34:36 2015-03-18 16:35:18,385 INFO repeater task os75nb_10pings starting 2.5 seconds late 2015-03-18 16:35:18,511 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:35:48,555 INFO repeater task os75nb_lastensq starting 2.6 seconds late 2015-03-18 16:35:49,985 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:35:49,985 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:35:36 2015-03-18 16:36:00,000 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-03-18 16:36:00,126 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:36:50,191 INFO repeater task os150nb_lastensq starting 4.2 seconds late 2015-03-18 16:36:51,962 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:36:51,962 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:36:36 2015-03-18 16:38:17,077 INFO repeater task os150nb_10pings starting 1.2 seconds late 2015-03-18 16:38:17,205 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:38:37,235 INFO repeater task os150nb_quick starting 1.3 seconds late 2015-03-18 16:38:41,663 INFO repeater task os150nb_quick ended after 4.4 seconds 2015-03-18 16:38:41,664 INFO repeater task os150nb_3dayplots starting 5.8 seconds late 2015-03-18 16:38:47,013 INFO repeater task os150nb_3dayplots ended after 5.3 seconds 2015-03-18 16:38:47,014 INFO repeater task os150nb_tsstats starting 11.1 seconds late 2015-03-18 16:38:47,732 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-18 16:38:47,732 INFO repeater task os75nb_lastensq starting 1.7 seconds late 2015-03-18 16:38:49,062 INFO repeater task os75nb_lastensq ended after 1.3 seconds 2015-03-18 16:38:49,062 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:38:36 2015-03-18 16:38:49,063 INFO repeater task os75nb_10pings starting 13.2 seconds late 2015-03-18 16:38:49,189 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:38:49,189 INFO repeater task os75nb_quick starting 13.3 seconds late 2015-03-18 16:38:53,210 INFO repeater task os75nb_quick ended after 4.0 seconds 2015-03-18 16:38:53,211 INFO repeater task os75nb_3dayplots starting 17.3 seconds late 2015-03-18 16:38:58,607 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 16:38:58,607 INFO repeater task os75nb_3dayplots ended after 5.4 seconds 2015-03-18 16:38:58,607 INFO repeater task os75nb_tsstats starting 22.7 seconds late 2015-03-18 16:38:59,335 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 16:38:59,335 INFO repeater task os150nb_beamplots starting 23.4 seconds late 2015-03-18 16:39:00,504 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:39:00,504 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:39:00,505 INFO repeater task os150nb_beamplots starting 24.6 seconds late 2015-03-18 16:39:01,689 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:39:01,689 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:39:01,689 INFO repeater task os75nb_beamplots starting 25.8 seconds late 2015-03-18 16:39:02,776 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:39:02,776 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 16:39:02,777 INFO repeater task os75nb_beamplots starting 26.9 seconds late 2015-03-18 16:39:03,950 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:39:03,950 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-18 16:39:03,950 INFO repeater task ashtech_hcorrstats starting 28.0 seconds late 2015-03-18 16:39:04,774 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:39:04,774 INFO repeater task phins3_hcorrstats starting 28.9 seconds late 2015-03-18 16:39:05,498 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:39:10,503 INFO repeater task os150nb_lastensq starting 24.5 seconds late 2015-03-18 16:39:12,277 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:39:12,277 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:38:36 2015-03-18 16:40:37,400 INFO repeater task os150nb_10pings starting 1.5 seconds late 2015-03-18 16:40:37,524 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:40:47,540 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2015-03-18 16:40:49,276 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:40:49,276 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:40:36 2015-03-18 16:41:49,362 INFO repeater task os75nb_lastensq starting 3.4 seconds late 2015-03-18 16:41:50,785 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:41:50,785 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:41:36 2015-03-18 16:42:00,800 INFO repeater task os75nb_10pings starting 4.9 seconds late 2015-03-18 16:42:00,935 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:42:46,001 INFO repeater task os150nb_lastensq starting 0.0 seconds late 2015-03-18 16:42:47,747 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:42:47,748 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:42:36 2015-03-18 16:42:57,762 INFO repeater task os150nb_10pings starting 1.9 seconds late 2015-03-18 16:42:57,920 INFO repeater task os150nb_10pings ended after 0.2 seconds 2015-03-18 16:43:37,973 INFO repeater task ashtech_hcorrstats starting 2.1 seconds late 2015-03-18 16:43:38,794 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:43:38,794 INFO repeater task phins3_hcorrstats starting 2.9 seconds late 2015-03-18 16:43:39,568 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 16:44:49,665 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-03-18 16:44:51,500 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:44:51,501 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:44:36 2015-03-18 16:44:51,502 INFO repeater task os75nb_lastensq starting 5.5 seconds late 2015-03-18 16:44:52,980 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-18 16:44:52,980 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:44:36 2015-03-18 16:45:18,016 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-03-18 16:45:18,152 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:45:18,153 INFO repeater task os75nb_10pings starting 2.3 seconds late 2015-03-18 16:45:18,277 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:46:48,395 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-03-18 16:46:50,140 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:46:50,140 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:46:36 2015-03-18 16:47:40,209 INFO repeater task os150nb_10pings starting 4.3 seconds late 2015-03-18 16:47:40,339 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:47:50,352 INFO repeater task os75nb_lastensq starting 4.4 seconds late 2015-03-18 16:47:51,796 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:47:51,796 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:47:36 2015-03-18 16:48:36,856 INFO repeater task os75nb_10pings starting 1.0 seconds late 2015-03-18 16:48:36,980 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:48:36,982 INFO repeater task ashtech_hcorrstats starting 1.1 seconds late 2015-03-18 16:48:37,841 INFO repeater task ashtech_hcorrstats ended after 0.9 seconds 2015-03-18 16:48:37,842 INFO repeater task phins3_hcorrstats starting 1.9 seconds late 2015-03-18 16:48:38,593 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 16:48:48,605 INFO repeater task os150nb_lastensq starting 2.6 seconds late 2015-03-18 16:48:50,500 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-18 16:48:50,500 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:48:36 2015-03-18 16:50:00,590 INFO repeater task os150nb_10pings starting 4.7 seconds late 2015-03-18 16:50:00,725 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:50:50,789 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-03-18 16:50:52,547 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:50:52,547 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:50:36 2015-03-18 16:50:52,548 INFO repeater task os75nb_lastensq starting 6.5 seconds late 2015-03-18 16:50:53,961 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:50:53,961 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:50:36 2015-03-18 16:51:59,060 INFO repeater task os75nb_10pings starting 3.2 seconds late 2015-03-18 16:51:59,186 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:52:19,210 INFO repeater task os150nb_10pings starting 3.3 seconds late 2015-03-18 16:52:19,340 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:52:49,383 INFO repeater task os150nb_lastensq starting 3.4 seconds late 2015-03-18 16:52:51,173 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:52:51,173 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:52:36 2015-03-18 16:53:36,241 INFO repeater task os150nb_quick starting 0.3 seconds late 2015-03-18 16:53:40,721 INFO repeater task os150nb_quick ended after 4.5 seconds 2015-03-18 16:53:40,721 INFO repeater task os150nb_3dayplots starting 4.8 seconds late 2015-03-18 16:53:46,385 INFO repeater task os150nb_3dayplots ended after 5.7 seconds 2015-03-18 16:53:46,385 INFO repeater task os150nb_tsstats starting 10.5 seconds late 2015-03-18 16:53:47,104 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-18 16:53:47,105 INFO repeater task os75nb_lastensq starting 1.1 seconds late 2015-03-18 16:53:48,581 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-18 16:53:48,581 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:53:36 2015-03-18 16:53:48,582 INFO repeater task os75nb_quick starting 12.7 seconds late 2015-03-18 16:53:52,537 INFO repeater task os75nb_quick ended after 4.0 seconds 2015-03-18 16:53:52,537 INFO repeater task os75nb_3dayplots starting 16.6 seconds late 2015-03-18 16:53:59,145 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 16:53:59,145 INFO repeater task os75nb_3dayplots ended after 6.6 seconds 2015-03-18 16:53:59,145 INFO repeater task os75nb_tsstats starting 23.2 seconds late 2015-03-18 16:53:59,819 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 16:53:59,820 INFO repeater task os150nb_beamplots starting 23.9 seconds late 2015-03-18 16:54:01,003 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:54:01,003 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:54:01,004 INFO repeater task os150nb_beamplots starting 25.1 seconds late 2015-03-18 16:54:02,214 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:54:02,214 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 16:54:02,214 INFO repeater task os75nb_beamplots starting 26.3 seconds late 2015-03-18 16:54:03,350 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:54:03,351 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 16:54:03,351 INFO repeater task os75nb_beamplots starting 27.5 seconds late 2015-03-18 16:54:04,503 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 16:54:04,503 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-18 16:54:04,504 INFO repeater task ashtech_hcorrstats starting 28.6 seconds late 2015-03-18 16:54:05,293 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:54:05,293 INFO repeater task phins3_hcorrstats starting 29.4 seconds late 2015-03-18 16:54:06,071 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 16:54:36,105 INFO repeater task os150nb_10pings starting 0.2 seconds late 2015-03-18 16:54:36,229 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:54:46,242 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-18 16:54:48,009 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:54:48,009 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:54:36 2015-03-18 16:55:18,059 INFO repeater task os75nb_10pings starting 2.2 seconds late 2015-03-18 16:55:18,196 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:56:48,329 INFO repeater task os150nb_lastensq starting 2.3 seconds late 2015-03-18 16:56:50,096 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 16:56:50,096 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:56:36 2015-03-18 16:56:50,097 INFO repeater task os75nb_lastensq starting 4.1 seconds late 2015-03-18 16:56:51,493 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:56:51,493 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:56:36 2015-03-18 16:56:56,495 INFO repeater task os150nb_10pings starting 0.6 seconds late 2015-03-18 16:56:56,625 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:58:36,763 INFO repeater task os75nb_10pings starting 0.9 seconds late 2015-03-18 16:58:36,890 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 16:58:36,891 INFO repeater task ashtech_hcorrstats starting 1.0 seconds late 2015-03-18 16:58:37,655 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 16:58:37,655 INFO repeater task phins3_hcorrstats starting 1.8 seconds late 2015-03-18 16:58:38,388 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 16:58:48,398 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-03-18 16:58:50,141 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 16:58:50,141 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 16:58:36 2015-03-18 16:59:20,183 INFO repeater task os150nb_10pings starting 4.3 seconds late 2015-03-18 16:59:20,313 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 16:59:50,358 INFO repeater task os75nb_lastensq starting 4.4 seconds late 2015-03-18 16:59:51,726 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 16:59:51,727 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 16:59:36 2015-03-18 17:00:46,797 INFO repeater task os150nb_lastensq starting 0.8 seconds late 2015-03-18 17:00:48,532 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 17:00:48,532 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:00:36 2015-03-18 17:01:38,594 INFO repeater task os150nb_10pings starting 2.7 seconds late 2015-03-18 17:01:38,718 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:01:58,747 INFO repeater task os75nb_10pings starting 2.8 seconds late 2015-03-18 17:01:58,871 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:02:48,938 INFO repeater task os150nb_lastensq starting 2.9 seconds late 2015-03-18 17:02:50,709 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:02:50,709 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:02:36 2015-03-18 17:02:50,710 INFO repeater task os75nb_lastensq starting 4.7 seconds late 2015-03-18 17:02:52,109 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:02:52,109 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:02:36 2015-03-18 17:03:37,171 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2015-03-18 17:03:37,991 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:03:37,992 INFO repeater task phins3_hcorrstats starting 2.1 seconds late 2015-03-18 17:03:38,758 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:03:58,785 INFO repeater task os150nb_10pings starting 2.9 seconds late 2015-03-18 17:03:58,918 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:04:48,982 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-03-18 17:04:50,869 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-18 17:04:50,869 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:04:36 2015-03-18 17:05:15,906 INFO repeater task os75nb_10pings starting 0.0 seconds late 2015-03-18 17:05:16,037 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:05:46,082 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2015-03-18 17:05:47,450 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:05:47,450 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:05:36 2015-03-18 17:06:17,483 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-03-18 17:06:17,610 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:06:47,652 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-03-18 17:06:49,415 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:06:49,415 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:06:36 2015-03-18 17:08:39,558 INFO repeater task os150nb_10pings starting 3.7 seconds late 2015-03-18 17:08:39,682 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:08:39,682 INFO repeater task os150nb_quick starting 3.8 seconds late 2015-03-18 17:08:44,211 INFO repeater task os150nb_quick ended after 4.5 seconds 2015-03-18 17:08:44,212 INFO repeater task os150nb_3dayplots starting 8.3 seconds late 2015-03-18 17:08:49,841 INFO repeater task os150nb_3dayplots ended after 5.6 seconds 2015-03-18 17:08:49,841 INFO repeater task os150nb_tsstats starting 13.9 seconds late 2015-03-18 17:08:50,584 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-18 17:08:50,585 INFO repeater task os75nb_lastensq starting 4.6 seconds late 2015-03-18 17:08:51,994 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:08:51,994 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:08:36 2015-03-18 17:08:51,995 INFO repeater task os75nb_10pings starting 16.1 seconds late 2015-03-18 17:08:52,122 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:08:52,123 INFO repeater task os75nb_quick starting 16.2 seconds late 2015-03-18 17:08:56,059 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-18 17:08:56,059 INFO repeater task os75nb_3dayplots starting 20.2 seconds late 2015-03-18 17:09:01,603 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 17:09:01,603 INFO repeater task os75nb_3dayplots ended after 5.5 seconds 2015-03-18 17:09:01,604 INFO repeater task os75nb_tsstats starting 25.7 seconds late 2015-03-18 17:09:02,327 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 17:09:02,328 INFO repeater task os150nb_beamplots starting 26.4 seconds late 2015-03-18 17:09:03,522 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:09:03,522 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 17:09:03,523 INFO repeater task os150nb_beamplots starting 27.6 seconds late 2015-03-18 17:09:04,748 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:09:04,748 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 17:09:04,749 INFO repeater task os75nb_beamplots starting 28.8 seconds late 2015-03-18 17:09:05,892 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:09:05,893 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 17:09:05,893 INFO repeater task os75nb_beamplots starting 30.0 seconds late 2015-03-18 17:09:07,053 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:09:07,054 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-18 17:09:07,054 INFO repeater task ashtech_hcorrstats starting 31.2 seconds late 2015-03-18 17:09:07,831 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:09:07,832 INFO repeater task phins3_hcorrstats starting 31.9 seconds late 2015-03-18 17:09:08,575 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 17:09:13,580 INFO repeater task os150nb_lastensq starting 27.6 seconds late 2015-03-18 17:09:15,382 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:09:15,382 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:08:36 2015-03-18 17:10:50,512 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2015-03-18 17:10:52,286 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:10:52,287 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:10:36 2015-03-18 17:10:57,294 INFO repeater task os150nb_10pings starting 1.4 seconds late 2015-03-18 17:10:57,418 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:11:47,487 INFO repeater task os75nb_lastensq starting 1.5 seconds late 2015-03-18 17:11:48,819 INFO repeater task os75nb_lastensq ended after 1.3 seconds 2015-03-18 17:11:48,819 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:11:36 2015-03-18 17:11:58,830 INFO repeater task os75nb_10pings starting 2.9 seconds late 2015-03-18 17:11:58,953 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:12:49,023 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-03-18 17:12:50,829 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:12:50,829 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:12:36 2015-03-18 17:13:20,868 INFO repeater task os150nb_10pings starting 5.0 seconds late 2015-03-18 17:13:20,995 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:13:36,019 INFO repeater task ashtech_hcorrstats starting 0.1 seconds late 2015-03-18 17:13:36,811 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:13:36,811 INFO repeater task phins3_hcorrstats starting 0.9 seconds late 2015-03-18 17:13:37,582 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:14:47,676 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-03-18 17:14:49,452 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:14:49,452 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:14:36 2015-03-18 17:14:49,453 INFO repeater task os75nb_lastensq starting 3.5 seconds late 2015-03-18 17:14:50,858 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:14:50,858 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:14:36 2015-03-18 17:15:20,897 INFO repeater task os75nb_10pings starting 5.0 seconds late 2015-03-18 17:15:21,042 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:15:36,066 INFO repeater task os150nb_10pings starting 0.2 seconds late 2015-03-18 17:15:36,202 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:16:46,294 INFO repeater task os150nb_lastensq starting 0.3 seconds late 2015-03-18 17:16:48,004 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-18 17:16:48,004 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:16:36 2015-03-18 17:17:48,081 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2015-03-18 17:17:49,471 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:17:49,471 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:17:36 2015-03-18 17:17:59,482 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-03-18 17:17:59,610 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:18:39,667 INFO repeater task os75nb_10pings starting 3.8 seconds late 2015-03-18 17:18:39,794 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:18:39,795 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2015-03-18 17:18:40,573 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:18:40,573 INFO repeater task phins3_hcorrstats starting 4.7 seconds late 2015-03-18 17:18:41,338 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:18:46,344 INFO repeater task os150nb_lastensq starting 0.3 seconds late 2015-03-18 17:18:48,142 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:18:48,142 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:18:36 2015-03-18 17:20:18,264 INFO repeater task os150nb_10pings starting 2.4 seconds late 2015-03-18 17:20:18,393 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:20:48,434 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-03-18 17:20:50,191 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:20:50,191 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:20:36 2015-03-18 17:20:50,193 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2015-03-18 17:20:51,561 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:20:51,561 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:20:36 2015-03-18 17:21:56,649 INFO repeater task os75nb_10pings starting 0.8 seconds late 2015-03-18 17:21:56,779 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:22:36,836 INFO repeater task os150nb_10pings starting 0.9 seconds late 2015-03-18 17:22:36,966 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:22:46,988 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2015-03-18 17:22:48,781 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:22:48,782 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:22:36 2015-03-18 17:23:38,846 INFO repeater task os150nb_quick starting 2.9 seconds late 2015-03-18 17:23:43,345 INFO repeater task os150nb_quick ended after 4.5 seconds 2015-03-18 17:23:43,346 INFO repeater task os150nb_3dayplots starting 7.4 seconds late 2015-03-18 17:23:48,674 INFO repeater task os150nb_3dayplots ended after 5.3 seconds 2015-03-18 17:23:48,674 INFO repeater task os150nb_tsstats starting 12.8 seconds late 2015-03-18 17:23:49,381 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-18 17:23:49,382 INFO repeater task os75nb_lastensq starting 3.4 seconds late 2015-03-18 17:23:50,800 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:23:50,800 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:23:36 2015-03-18 17:23:50,800 INFO repeater task os75nb_quick starting 14.9 seconds late 2015-03-18 17:23:54,744 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-18 17:23:54,744 INFO repeater task os75nb_3dayplots starting 18.8 seconds late 2015-03-18 17:24:00,131 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 17:24:00,131 INFO repeater task os75nb_3dayplots ended after 5.4 seconds 2015-03-18 17:24:00,132 INFO repeater task os75nb_tsstats starting 24.2 seconds late 2015-03-18 17:24:00,826 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 17:24:00,826 INFO repeater task os150nb_beamplots starting 24.9 seconds late 2015-03-18 17:24:02,006 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:24:02,007 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 17:24:02,007 INFO repeater task os150nb_beamplots starting 26.1 seconds late 2015-03-18 17:24:03,208 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:24:03,208 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 17:24:03,208 INFO repeater task os75nb_beamplots starting 27.3 seconds late 2015-03-18 17:24:04,330 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:24:04,330 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 17:24:04,330 INFO repeater task os75nb_beamplots starting 28.4 seconds late 2015-03-18 17:24:05,426 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:24:05,426 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 17:24:05,427 INFO repeater task ashtech_hcorrstats starting 29.5 seconds late 2015-03-18 17:24:06,212 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:24:06,212 INFO repeater task phins3_hcorrstats starting 30.3 seconds late 2015-03-18 17:24:06,981 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:24:47,035 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2015-03-18 17:24:48,853 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:24:48,853 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:24:36 2015-03-18 17:24:58,868 INFO repeater task os150nb_10pings starting 3.0 seconds late 2015-03-18 17:24:58,996 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:25:19,025 INFO repeater task os75nb_10pings starting 3.1 seconds late 2015-03-18 17:25:19,151 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:26:49,286 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2015-03-18 17:26:51,086 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:26:51,087 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:26:36 2015-03-18 17:26:51,088 INFO repeater task os75nb_lastensq starting 5.1 seconds late 2015-03-18 17:26:52,488 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:26:52,488 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:26:36 2015-03-18 17:27:17,522 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-03-18 17:27:17,659 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:28:37,783 INFO repeater task os75nb_10pings starting 1.9 seconds late 2015-03-18 17:28:37,920 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:28:37,921 INFO repeater task ashtech_hcorrstats starting 2.0 seconds late 2015-03-18 17:28:38,741 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:28:38,741 INFO repeater task phins3_hcorrstats starting 2.8 seconds late 2015-03-18 17:28:39,499 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:28:49,510 INFO repeater task os150nb_lastensq starting 3.5 seconds late 2015-03-18 17:28:51,346 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:28:51,347 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:28:36 2015-03-18 17:29:36,408 INFO repeater task os150nb_10pings starting 0.5 seconds late 2015-03-18 17:29:36,540 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:29:46,554 INFO repeater task os75nb_lastensq starting 0.6 seconds late 2015-03-18 17:29:48,019 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-18 17:29:48,020 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:29:36 2015-03-18 17:30:48,102 INFO repeater task os150nb_lastensq starting 2.1 seconds late 2015-03-18 17:30:49,910 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:30:49,910 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:30:36 2015-03-18 17:32:00,009 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-03-18 17:32:00,137 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:32:00,139 INFO repeater task os75nb_10pings starting 4.2 seconds late 2015-03-18 17:32:00,270 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:32:50,336 INFO repeater task os150nb_lastensq starting 4.3 seconds late 2015-03-18 17:32:52,102 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:32:52,103 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:32:36 2015-03-18 17:32:52,103 INFO repeater task os75nb_lastensq starting 6.1 seconds late 2015-03-18 17:32:53,521 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:32:53,521 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:32:36 2015-03-18 17:33:38,599 INFO repeater task ashtech_hcorrstats starting 2.7 seconds late 2015-03-18 17:33:39,408 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:33:39,409 INFO repeater task phins3_hcorrstats starting 3.5 seconds late 2015-03-18 17:33:40,167 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:34:20,222 INFO repeater task os150nb_10pings starting 4.3 seconds late 2015-03-18 17:34:20,356 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:34:50,395 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2015-03-18 17:34:52,183 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:34:52,183 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:34:36 2015-03-18 17:35:17,219 INFO repeater task os75nb_10pings starting 1.3 seconds late 2015-03-18 17:35:17,346 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:35:47,387 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2015-03-18 17:35:48,842 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-18 17:35:48,842 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:35:36 2015-03-18 17:36:38,915 INFO repeater task os150nb_10pings starting 3.0 seconds late 2015-03-18 17:36:39,057 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:36:49,074 INFO repeater task os150nb_lastensq starting 3.1 seconds late 2015-03-18 17:36:50,859 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:36:50,860 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:36:36 2015-03-18 17:38:36,001 INFO repeater task os150nb_quick starting 0.1 seconds late 2015-03-18 17:38:40,365 INFO repeater task os150nb_quick ended after 4.4 seconds 2015-03-18 17:38:40,365 INFO repeater task os150nb_3dayplots starting 4.5 seconds late 2015-03-18 17:38:45,753 INFO repeater task os150nb_3dayplots ended after 5.4 seconds 2015-03-18 17:38:45,754 INFO repeater task os150nb_tsstats starting 9.9 seconds late 2015-03-18 17:38:46,516 INFO repeater task os150nb_tsstats ended after 0.8 seconds 2015-03-18 17:38:46,517 INFO repeater task os75nb_lastensq starting 0.5 seconds late 2015-03-18 17:38:47,922 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:38:47,922 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:38:36 2015-03-18 17:38:47,923 INFO repeater task os75nb_10pings starting 12.0 seconds late 2015-03-18 17:38:48,047 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:38:48,048 INFO repeater task os75nb_quick starting 12.1 seconds late 2015-03-18 17:38:51,993 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-18 17:38:51,993 INFO repeater task os75nb_3dayplots starting 16.1 seconds late 2015-03-18 17:38:57,203 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-18 17:38:57,203 INFO repeater task os75nb_3dayplots ended after 5.2 seconds 2015-03-18 17:38:57,203 INFO repeater task os75nb_tsstats starting 21.3 seconds late 2015-03-18 17:38:57,854 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-18 17:38:57,854 INFO repeater task os150nb_beamplots starting 22.0 seconds late 2015-03-18 17:38:58,983 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:38:58,983 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-03-18 17:38:58,983 INFO repeater task os150nb_beamplots starting 23.1 seconds late 2015-03-18 17:39:00,166 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:39:00,166 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-18 17:39:00,167 INFO repeater task os75nb_beamplots starting 24.3 seconds late 2015-03-18 17:39:01,358 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:39:01,358 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-18 17:39:01,358 INFO repeater task os75nb_beamplots starting 25.5 seconds late 2015-03-18 17:39:02,460 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-18 17:39:02,460 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-18 17:39:02,461 INFO repeater task ashtech_hcorrstats starting 26.6 seconds late 2015-03-18 17:39:03,283 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:39:03,284 INFO repeater task phins3_hcorrstats starting 27.4 seconds late 2015-03-18 17:39:04,051 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:39:09,056 INFO repeater task os150nb_lastensq starting 23.1 seconds late 2015-03-18 17:39:10,871 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:39:10,871 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:38:36 2015-03-18 17:39:10,871 INFO repeater task os150nb_10pings starting 15.0 seconds late 2015-03-18 17:39:10,994 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:40:46,123 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-03-18 17:40:47,903 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:40:47,903 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:40:36 2015-03-18 17:41:17,948 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-03-18 17:41:18,078 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:41:48,119 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2015-03-18 17:41:49,506 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:41:49,506 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:41:36 2015-03-18 17:41:59,518 INFO repeater task os75nb_10pings starting 3.6 seconds late 2015-03-18 17:41:59,650 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:42:49,723 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-03-18 17:42:51,519 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:42:51,519 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:42:36 2015-03-18 17:43:36,579 INFO repeater task os150nb_10pings starting 0.7 seconds late 2015-03-18 17:43:36,708 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:43:36,710 INFO repeater task ashtech_hcorrstats starting 0.8 seconds late 2015-03-18 17:43:37,531 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:43:37,532 INFO repeater task phins3_hcorrstats starting 1.6 seconds late 2015-03-18 17:43:38,279 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-18 17:44:48,377 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-03-18 17:44:50,170 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:44:50,170 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:44:36 2015-03-18 17:44:50,171 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2015-03-18 17:44:51,576 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:44:51,576 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:44:36 2015-03-18 17:45:16,609 INFO repeater task os75nb_10pings starting 0.7 seconds late 2015-03-18 17:45:16,738 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:45:56,788 INFO repeater task os150nb_10pings starting 0.9 seconds late 2015-03-18 17:45:56,914 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:46:46,979 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2015-03-18 17:46:48,777 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:46:48,777 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:46:36 2015-03-18 17:47:48,861 INFO repeater task os75nb_lastensq starting 2.9 seconds late 2015-03-18 17:47:50,288 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-18 17:47:50,289 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/18 17:47:36 2015-03-18 17:48:20,326 INFO repeater task os150nb_10pings starting 4.4 seconds late 2015-03-18 17:48:20,452 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-18 17:48:40,478 INFO repeater task os75nb_10pings starting 4.6 seconds late 2015-03-18 17:48:40,608 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-18 17:48:40,610 INFO repeater task ashtech_hcorrstats starting 4.7 seconds late 2015-03-18 17:48:41,421 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-18 17:48:41,421 INFO repeater task phins3_hcorrstats starting 5.5 seconds late 2015-03-18 17:48:42,178 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-18 17:48:47,184 INFO repeater task os150nb_lastensq starting 1.2 seconds late 2015-03-18 17:48:48,974 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-18 17:48:48,974 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/18 17:48:36 2015-03-18 17:49:22,117 INFO Translate translate exiting, gps /home/data/RR1504/rbin/gpsnav 2015-03-18 17:49:22,211 INFO Translate translate exiting, gps /home/data/RR1504/rbin/hydrins 2015-03-18 17:49:22,239 INFO Translate translate exiting, hdg /home/data/RR1504/rbin/hydrins 2015-03-18 17:49:22,275 INFO Translate translate exiting, gps /home/data/RR1504/rbin/ashtech 2015-03-18 17:49:22,276 INFO Translate translate exiting, hdg /home/data/RR1504/rbin/phins3 2015-03-18 17:49:22,303 INFO Translate translate exiting, adu /home/data/RR1504/rbin/ashtech 2015-03-18 17:49:24,019 INFO root cleanup_figs not_logging 2015-03-18 17:49:24,026 INFO root Exiting DAS_while_logging.py