2014-12-20 06:21:15,339 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:21:02 2014-12-20 06:21:15,340 INFO repeater task os75nb_lastensq starting 6.3 seconds late 2014-12-20 06:21:16,723 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:21:16,723 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:20:59 2014-12-20 06:21:41,752 INFO repeater task os150nb_10pings starting 0.7 seconds late 2014-12-20 06:21:41,879 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:21:41,880 INFO repeater task os75nb_10pings starting 0.9 seconds late 2014-12-20 06:21:42,003 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:23:12,122 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2014-12-20 06:23:13,939 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:23:13,939 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:23:02 2014-12-20 06:24:04,008 INFO repeater task os150nb_10pings starting 3.0 seconds late 2014-12-20 06:24:04,135 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:24:09,143 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2014-12-20 06:24:10,556 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:24:10,556 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:23:59 2014-12-20 06:25:05,628 INFO repeater task os75nb_10pings starting 4.6 seconds late 2014-12-20 06:25:05,760 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:25:05,762 INFO repeater task ashtech_hcorrstats starting 4.7 seconds late 2014-12-20 06:25:06,555 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 06:25:06,556 INFO repeater task phins3_hcorrstats starting 5.5 seconds late 2014-12-20 06:25:07,345 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 06:25:12,350 INFO repeater task os150nb_lastensq starting 0.4 seconds late 2014-12-20 06:25:14,134 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:25:14,135 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:25:02 2014-12-20 06:26:24,224 INFO repeater task os150nb_10pings starting 3.2 seconds late 2014-12-20 06:26:24,358 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:27:09,422 INFO repeater task os75nb_lastensq starting 0.4 seconds late 2014-12-20 06:27:10,796 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:27:10,796 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:26:59 2014-12-20 06:27:15,802 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2014-12-20 06:27:17,571 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:27:17,572 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:27:02 2014-12-20 06:28:22,656 INFO repeater task os75nb_10pings starting 1.6 seconds late 2014-12-20 06:28:22,778 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:28:42,806 INFO repeater task os150nb_10pings starting 1.8 seconds late 2014-12-20 06:28:42,940 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:29:12,978 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2014-12-20 06:29:14,752 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:29:14,752 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:29:02 2014-12-20 06:30:04,815 INFO repeater task os150nb_quick starting 3.8 seconds late 2014-12-20 06:30:14,104 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 06:30:14,104 INFO repeater task os150nb_quick ended after 9.3 seconds 2014-12-20 06:30:14,105 INFO repeater task os150nb_3dayplots starting 13.1 seconds late 2014-12-20 06:30:26,621 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 06:30:26,621 INFO repeater task os150nb_3dayplots ended after 12.5 seconds 2014-12-20 06:30:26,621 INFO repeater task os150nb_tsstats starting 25.6 seconds late 2014-12-20 06:30:27,678 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 06:30:27,678 INFO repeater task os75nb_lastensq starting 18.7 seconds late 2014-12-20 06:30:29,161 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 06:30:29,161 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:29:59 2014-12-20 06:30:29,162 INFO repeater task os75nb_quick starting 28.1 seconds late 2014-12-20 06:30:36,146 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 06:30:36,146 INFO repeater task os75nb_quick ended after 7.0 seconds 2014-12-20 06:30:36,147 INFO repeater task os75nb_3dayplots starting 35.1 seconds late 2014-12-20 06:30:49,346 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 06:30:49,347 INFO repeater task os75nb_3dayplots ended after 13.2 seconds 2014-12-20 06:30:49,347 INFO repeater task os75nb_tsstats starting 48.3 seconds late 2014-12-20 06:30:50,292 INFO repeater task os75nb_tsstats ended after 0.9 seconds 2014-12-20 06:30:50,292 INFO repeater task os150nb_beamplots starting 49.3 seconds late 2014-12-20 06:30:51,456 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:30:51,456 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 06:30:51,456 INFO repeater task os150nb_beamplots starting 50.4 seconds late 2014-12-20 06:30:52,634 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:30:52,634 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 06:30:52,634 INFO repeater task os75nb_beamplots starting 51.6 seconds late 2014-12-20 06:30:53,743 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:30:53,743 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 06:30:53,743 INFO repeater task os75nb_beamplots starting 52.7 seconds late 2014-12-20 06:30:54,831 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:30:54,832 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 06:30:54,832 INFO repeater task ashtech_hcorrstats starting 53.8 seconds late 2014-12-20 06:30:55,618 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 06:30:55,618 INFO repeater task phins3_hcorrstats starting 54.6 seconds late 2014-12-20 06:30:56,367 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 06:31:01,373 INFO repeater task os150nb_10pings starting 0.3 seconds late 2014-12-20 06:31:01,504 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:31:16,526 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2014-12-20 06:31:18,307 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:31:18,307 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:31:02 2014-12-20 06:31:43,335 INFO repeater task os75nb_10pings starting 2.3 seconds late 2014-12-20 06:31:43,461 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:33:13,585 INFO repeater task os150nb_lastensq starting 1.6 seconds late 2014-12-20 06:33:15,362 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:33:15,363 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:33:02 2014-12-20 06:33:15,364 INFO repeater task os75nb_lastensq starting 6.4 seconds late 2014-12-20 06:33:16,849 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 06:33:16,850 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:32:59 2014-12-20 06:33:21,857 INFO repeater task os150nb_10pings starting 0.8 seconds late 2014-12-20 06:33:21,981 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:35:02,115 INFO repeater task os75nb_10pings starting 1.1 seconds late 2014-12-20 06:35:02,241 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:35:02,243 INFO repeater task ashtech_hcorrstats starting 1.2 seconds late 2014-12-20 06:35:03,033 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 06:35:03,033 INFO repeater task phins3_hcorrstats starting 2.0 seconds late 2014-12-20 06:35:03,816 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 06:35:13,825 INFO repeater task os150nb_lastensq starting 1.8 seconds late 2014-12-20 06:35:15,586 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:35:15,586 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:35:02 2014-12-20 06:35:45,619 INFO repeater task os150nb_10pings starting 4.6 seconds late 2014-12-20 06:35:45,745 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:36:10,783 INFO repeater task os75nb_lastensq starting 1.8 seconds late 2014-12-20 06:36:12,161 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:36:12,162 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:35:59 2014-12-20 06:37:12,231 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2014-12-20 06:37:14,023 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:37:14,024 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:37:02 2014-12-20 06:38:04,090 INFO repeater task os150nb_10pings starting 3.1 seconds late 2014-12-20 06:38:04,218 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:38:24,246 INFO repeater task os75nb_10pings starting 3.2 seconds late 2014-12-20 06:38:24,373 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:39:09,431 INFO repeater task os75nb_lastensq starting 0.4 seconds late 2014-12-20 06:39:10,834 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:39:10,835 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:38:59 2014-12-20 06:39:15,841 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2014-12-20 06:39:17,632 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:39:17,633 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:39:02 2014-12-20 06:40:02,694 INFO repeater task ashtech_hcorrstats starting 1.7 seconds late 2014-12-20 06:40:03,502 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 06:40:03,503 INFO repeater task phins3_hcorrstats starting 2.5 seconds late 2014-12-20 06:40:04,292 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 06:40:24,319 INFO repeater task os150nb_10pings starting 3.3 seconds late 2014-12-20 06:40:24,447 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:41:14,502 INFO repeater task os150nb_lastensq starting 2.5 seconds late 2014-12-20 06:41:16,269 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:41:16,269 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:41:02 2014-12-20 06:41:41,303 INFO repeater task os75nb_10pings starting 0.3 seconds late 2014-12-20 06:41:41,438 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:42:11,471 INFO repeater task os75nb_lastensq starting 2.5 seconds late 2014-12-20 06:42:12,952 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 06:42:12,952 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:41:59 2014-12-20 06:42:42,991 INFO repeater task os150nb_10pings starting 2.0 seconds late 2014-12-20 06:42:43,126 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:43:13,166 INFO repeater task os150nb_lastensq starting 1.2 seconds late 2014-12-20 06:43:14,959 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:43:14,959 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:43:02 2014-12-20 06:45:05,106 INFO repeater task os150nb_10pings starting 4.1 seconds late 2014-12-20 06:45:05,235 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:45:05,235 INFO repeater task os150nb_quick starting 4.2 seconds late 2014-12-20 06:45:14,214 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 06:45:14,215 INFO repeater task os150nb_quick ended after 9.0 seconds 2014-12-20 06:45:14,215 INFO repeater task os150nb_3dayplots starting 13.2 seconds late 2014-12-20 06:45:26,917 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 06:45:26,917 INFO repeater task os150nb_3dayplots ended after 12.7 seconds 2014-12-20 06:45:26,918 INFO repeater task os150nb_tsstats starting 25.9 seconds late 2014-12-20 06:45:28,031 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 06:45:28,032 INFO repeater task os75nb_lastensq starting 19.0 seconds late 2014-12-20 06:45:29,456 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:45:29,456 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:44:59 2014-12-20 06:45:29,457 INFO repeater task os75nb_10pings starting 28.4 seconds late 2014-12-20 06:45:29,579 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:45:29,580 INFO repeater task os75nb_quick starting 28.6 seconds late 2014-12-20 06:45:36,643 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 06:45:36,643 INFO repeater task os75nb_quick ended after 7.1 seconds 2014-12-20 06:45:36,643 INFO repeater task os75nb_3dayplots starting 35.6 seconds late 2014-12-20 06:45:49,702 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 06:45:49,703 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2014-12-20 06:45:49,703 INFO repeater task os75nb_tsstats starting 48.7 seconds late 2014-12-20 06:45:50,618 INFO repeater task os75nb_tsstats ended after 0.9 seconds 2014-12-20 06:45:50,618 INFO repeater task os150nb_beamplots starting 49.6 seconds late 2014-12-20 06:45:51,843 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:45:51,843 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 06:45:51,844 INFO repeater task os150nb_beamplots starting 50.8 seconds late 2014-12-20 06:45:53,090 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:45:53,090 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 06:45:53,091 INFO repeater task os75nb_beamplots starting 52.1 seconds late 2014-12-20 06:45:54,204 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:45:54,205 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 06:45:54,205 INFO repeater task os75nb_beamplots starting 53.2 seconds late 2014-12-20 06:45:55,330 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 06:45:55,330 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 06:45:55,331 INFO repeater task ashtech_hcorrstats starting 54.3 seconds late 2014-12-20 06:45:56,184 INFO repeater task ashtech_hcorrstats ended after 0.9 seconds 2014-12-20 06:45:56,184 INFO repeater task phins3_hcorrstats starting 55.2 seconds late 2014-12-20 06:45:56,931 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 06:46:01,933 INFO repeater task os150nb_lastensq starting 49.9 seconds late 2014-12-20 06:46:03,770 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:46:03,770 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:45:02 2014-12-20 06:47:13,865 INFO repeater task os150nb_lastensq starting 1.9 seconds late 2014-12-20 06:47:15,626 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:47:15,627 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:47:02 2014-12-20 06:47:25,642 INFO repeater task os150nb_10pings starting 4.6 seconds late 2014-12-20 06:47:25,765 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:48:10,823 INFO repeater task os75nb_lastensq starting 1.8 seconds late 2014-12-20 06:48:12,290 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 06:48:12,290 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:47:59 2014-12-20 06:48:22,305 INFO repeater task os75nb_10pings starting 1.3 seconds late 2014-12-20 06:48:22,432 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:49:12,494 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2014-12-20 06:49:14,298 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:49:14,298 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:49:02 2014-12-20 06:49:44,341 INFO repeater task os150nb_10pings starting 3.3 seconds late 2014-12-20 06:49:44,470 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:50:04,501 INFO repeater task ashtech_hcorrstats starting 3.5 seconds late 2014-12-20 06:50:05,347 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 06:50:05,347 INFO repeater task phins3_hcorrstats starting 4.3 seconds late 2014-12-20 06:50:06,096 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 06:51:11,178 INFO repeater task os75nb_lastensq starting 2.2 seconds late 2014-12-20 06:51:12,618 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:51:12,618 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:50:59 2014-12-20 06:51:17,625 INFO repeater task os150nb_lastensq starting 5.6 seconds late 2014-12-20 06:51:19,418 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:51:19,418 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:51:02 2014-12-20 06:51:44,447 INFO repeater task os75nb_10pings starting 3.4 seconds late 2014-12-20 06:51:44,579 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:52:04,606 INFO repeater task os150nb_10pings starting 3.6 seconds late 2014-12-20 06:52:04,736 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:53:14,826 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2014-12-20 06:53:16,684 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 06:53:16,684 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:53:02 2014-12-20 06:54:11,769 INFO repeater task os75nb_lastensq starting 2.8 seconds late 2014-12-20 06:54:13,187 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:54:13,187 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:53:59 2014-12-20 06:54:23,201 INFO repeater task os150nb_10pings starting 2.2 seconds late 2014-12-20 06:54:23,326 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:55:03,385 INFO repeater task os75nb_10pings starting 2.4 seconds late 2014-12-20 06:55:03,521 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:55:03,522 INFO repeater task ashtech_hcorrstats starting 2.5 seconds late 2014-12-20 06:55:04,323 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 06:55:04,323 INFO repeater task phins3_hcorrstats starting 3.3 seconds late 2014-12-20 06:55:05,087 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 06:55:15,100 INFO repeater task os150nb_lastensq starting 3.1 seconds late 2014-12-20 06:55:16,899 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:55:16,900 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:55:02 2014-12-20 06:56:42,012 INFO repeater task os150nb_10pings starting 1.0 seconds late 2014-12-20 06:56:42,142 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:57:12,182 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2014-12-20 06:57:14,047 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 06:57:14,048 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:57:02 2014-12-20 06:57:14,049 INFO repeater task os75nb_lastensq starting 5.0 seconds late 2014-12-20 06:57:15,496 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 06:57:15,496 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:56:59 2014-12-20 06:58:25,589 INFO repeater task os75nb_10pings starting 4.6 seconds late 2014-12-20 06:58:25,724 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 06:59:05,774 INFO repeater task os150nb_10pings starting 4.7 seconds late 2014-12-20 06:59:05,904 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 06:59:15,918 INFO repeater task os150nb_lastensq starting 3.9 seconds late 2014-12-20 06:59:17,708 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 06:59:17,708 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 06:59:02 2014-12-20 07:00:02,771 INFO repeater task os150nb_quick starting 1.7 seconds late 2014-12-20 07:00:12,125 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:00:12,125 INFO repeater task os150nb_quick ended after 9.4 seconds 2014-12-20 07:00:12,126 INFO repeater task os150nb_3dayplots starting 11.1 seconds late 2014-12-20 07:00:24,443 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:00:24,443 INFO repeater task os150nb_3dayplots ended after 12.3 seconds 2014-12-20 07:00:24,443 INFO repeater task os150nb_tsstats starting 23.4 seconds late 2014-12-20 07:00:25,506 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 07:00:25,507 INFO repeater task os75nb_lastensq starting 16.5 seconds late 2014-12-20 07:00:26,960 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:00:26,961 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 06:59:59 2014-12-20 07:00:26,961 INFO repeater task os75nb_quick starting 25.9 seconds late 2014-12-20 07:00:34,167 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:00:34,168 INFO repeater task os75nb_quick ended after 7.2 seconds 2014-12-20 07:00:34,168 INFO repeater task os75nb_3dayplots starting 33.1 seconds late 2014-12-20 07:00:47,047 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:00:47,047 INFO repeater task os75nb_3dayplots ended after 12.9 seconds 2014-12-20 07:00:47,047 INFO repeater task os75nb_tsstats starting 46.0 seconds late 2014-12-20 07:00:47,992 INFO repeater task os75nb_tsstats ended after 0.9 seconds 2014-12-20 07:00:47,993 INFO repeater task os150nb_beamplots starting 47.0 seconds late 2014-12-20 07:00:49,169 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:00:49,170 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:00:49,170 INFO repeater task os150nb_beamplots starting 48.1 seconds late 2014-12-20 07:00:50,360 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:00:50,360 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:00:50,360 INFO repeater task os75nb_beamplots starting 49.3 seconds late 2014-12-20 07:00:51,510 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:00:51,510 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2014-12-20 07:00:51,511 INFO repeater task os75nb_beamplots starting 50.5 seconds late 2014-12-20 07:00:52,633 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:00:52,634 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 07:00:52,634 INFO repeater task ashtech_hcorrstats starting 51.6 seconds late 2014-12-20 07:00:53,489 INFO repeater task ashtech_hcorrstats ended after 0.9 seconds 2014-12-20 07:00:53,489 INFO repeater task phins3_hcorrstats starting 52.5 seconds late 2014-12-20 07:00:54,236 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:01:14,262 INFO repeater task os150nb_lastensq starting 2.3 seconds late 2014-12-20 07:01:16,148 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:01:16,148 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:01:02 2014-12-20 07:01:21,156 INFO repeater task os150nb_10pings starting 0.1 seconds late 2014-12-20 07:01:21,290 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:01:41,320 INFO repeater task os75nb_10pings starting 0.3 seconds late 2014-12-20 07:01:41,451 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:03:11,570 INFO repeater task os75nb_lastensq starting 2.6 seconds late 2014-12-20 07:03:13,044 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:03:13,044 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:02:59 2014-12-20 07:03:18,051 INFO repeater task os150nb_lastensq starting 6.1 seconds late 2014-12-20 07:03:19,936 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:03:19,936 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:03:02 2014-12-20 07:03:44,973 INFO repeater task os150nb_10pings starting 3.9 seconds late 2014-12-20 07:03:45,109 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:05:05,215 INFO repeater task os75nb_10pings starting 4.2 seconds late 2014-12-20 07:05:05,347 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:05:05,348 INFO repeater task ashtech_hcorrstats starting 4.3 seconds late 2014-12-20 07:05:06,209 INFO repeater task ashtech_hcorrstats ended after 0.9 seconds 2014-12-20 07:05:06,209 INFO repeater task phins3_hcorrstats starting 5.2 seconds late 2014-12-20 07:05:07,005 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 07:05:12,006 INFO repeater task os150nb_lastensq starting 0.0 seconds late 2014-12-20 07:05:13,881 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:05:13,882 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:05:02 2014-12-20 07:06:03,950 INFO repeater task os150nb_10pings starting 2.9 seconds late 2014-12-20 07:06:04,083 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:06:09,091 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2014-12-20 07:06:10,576 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:06:10,577 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:05:59 2014-12-20 07:07:15,658 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2014-12-20 07:07:17,425 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:07:17,425 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:07:02 2014-12-20 07:08:22,509 INFO repeater task os150nb_10pings starting 1.5 seconds late 2014-12-20 07:08:22,638 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:08:22,639 INFO repeater task os75nb_10pings starting 1.6 seconds late 2014-12-20 07:08:22,768 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:09:12,827 INFO repeater task os150nb_lastensq starting 0.8 seconds late 2014-12-20 07:09:14,637 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:09:14,637 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:09:02 2014-12-20 07:09:14,638 INFO repeater task os75nb_lastensq starting 5.6 seconds late 2014-12-20 07:09:16,018 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:09:16,019 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:08:59 2014-12-20 07:10:01,076 INFO repeater task ashtech_hcorrstats starting 0.0 seconds late 2014-12-20 07:10:01,886 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:10:01,887 INFO repeater task phins3_hcorrstats starting 0.9 seconds late 2014-12-20 07:10:02,626 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:10:42,673 INFO repeater task os150nb_10pings starting 1.6 seconds late 2014-12-20 07:10:42,808 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:11:12,846 INFO repeater task os150nb_lastensq starting 0.8 seconds late 2014-12-20 07:11:14,629 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:11:14,629 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:11:02 2014-12-20 07:11:44,663 INFO repeater task os75nb_10pings starting 3.6 seconds late 2014-12-20 07:11:44,788 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:12:09,821 INFO repeater task os75nb_lastensq starting 0.8 seconds late 2014-12-20 07:12:11,257 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:12:11,258 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:11:59 2014-12-20 07:13:01,322 INFO repeater task os150nb_10pings starting 0.3 seconds late 2014-12-20 07:13:01,446 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:13:16,467 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2014-12-20 07:13:18,360 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:13:18,360 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:13:02 2014-12-20 07:15:03,492 INFO repeater task os150nb_quick starting 2.5 seconds late 2014-12-20 07:15:12,516 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:15:12,517 INFO repeater task os150nb_quick ended after 9.0 seconds 2014-12-20 07:15:12,517 INFO repeater task os150nb_3dayplots starting 11.5 seconds late 2014-12-20 07:15:25,228 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:15:25,228 INFO repeater task os150nb_3dayplots ended after 12.7 seconds 2014-12-20 07:15:25,228 INFO repeater task os150nb_tsstats starting 24.2 seconds late 2014-12-20 07:15:26,329 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 07:15:26,329 INFO repeater task os75nb_lastensq starting 17.3 seconds late 2014-12-20 07:15:27,735 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:15:27,735 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:14:59 2014-12-20 07:15:27,736 INFO repeater task os75nb_10pings starting 26.7 seconds late 2014-12-20 07:15:27,863 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:15:27,863 INFO repeater task os75nb_quick starting 26.8 seconds late 2014-12-20 07:15:34,728 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:15:34,729 INFO repeater task os75nb_quick ended after 6.9 seconds 2014-12-20 07:15:34,729 INFO repeater task os75nb_3dayplots starting 33.7 seconds late 2014-12-20 07:15:47,712 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:15:47,712 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2014-12-20 07:15:47,712 INFO repeater task os75nb_tsstats starting 46.7 seconds late 2014-12-20 07:15:48,644 INFO repeater task os75nb_tsstats ended after 0.9 seconds 2014-12-20 07:15:48,645 INFO repeater task os150nb_beamplots starting 47.6 seconds late 2014-12-20 07:15:49,888 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:15:49,888 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:15:49,888 INFO repeater task os150nb_beamplots starting 48.9 seconds late 2014-12-20 07:15:51,092 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:15:51,092 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:15:51,092 INFO repeater task os75nb_beamplots starting 50.1 seconds late 2014-12-20 07:15:52,290 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:15:52,290 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2014-12-20 07:15:52,291 INFO repeater task os75nb_beamplots starting 51.3 seconds late 2014-12-20 07:15:53,434 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:15:53,434 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 07:15:53,434 INFO repeater task ashtech_hcorrstats starting 52.4 seconds late 2014-12-20 07:15:54,221 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:15:54,221 INFO repeater task phins3_hcorrstats starting 53.2 seconds late 2014-12-20 07:15:54,948 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:15:59,953 INFO repeater task os150nb_lastensq starting 48.0 seconds late 2014-12-20 07:16:01,742 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:16:01,742 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:15:02 2014-12-20 07:16:01,742 INFO repeater task os150nb_10pings starting 40.7 seconds late 2014-12-20 07:16:01,870 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:17:16,975 INFO repeater task os150nb_lastensq starting 5.0 seconds late 2014-12-20 07:17:18,772 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:17:18,773 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:17:02 2014-12-20 07:17:43,807 INFO repeater task os150nb_10pings starting 2.8 seconds late 2014-12-20 07:17:43,935 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:18:13,976 INFO repeater task os75nb_lastensq starting 5.0 seconds late 2014-12-20 07:18:15,380 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:18:15,381 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:17:59 2014-12-20 07:18:25,395 INFO repeater task os75nb_10pings starting 4.4 seconds late 2014-12-20 07:18:25,525 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:19:15,580 INFO repeater task os150nb_lastensq starting 3.6 seconds late 2014-12-20 07:19:17,442 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:19:17,442 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:19:02 2014-12-20 07:20:02,496 INFO repeater task os150nb_10pings starting 1.5 seconds late 2014-12-20 07:20:02,626 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:20:02,627 INFO repeater task ashtech_hcorrstats starting 1.6 seconds late 2014-12-20 07:20:03,437 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:20:03,437 INFO repeater task phins3_hcorrstats starting 2.4 seconds late 2014-12-20 07:20:04,222 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 07:21:09,310 INFO repeater task os75nb_lastensq starting 0.3 seconds late 2014-12-20 07:21:10,778 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:21:10,778 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:20:59 2014-12-20 07:21:15,784 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2014-12-20 07:21:17,632 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:21:17,632 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:21:02 2014-12-20 07:21:42,663 INFO repeater task os75nb_10pings starting 1.6 seconds late 2014-12-20 07:21:42,793 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:22:22,842 INFO repeater task os150nb_10pings starting 1.8 seconds late 2014-12-20 07:22:22,975 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:23:13,046 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2014-12-20 07:23:14,904 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:23:14,904 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:23:02 2014-12-20 07:24:09,978 INFO repeater task os75nb_lastensq starting 1.0 seconds late 2014-12-20 07:24:11,436 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:24:11,436 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:23:59 2014-12-20 07:24:41,476 INFO repeater task os150nb_10pings starting 0.5 seconds late 2014-12-20 07:24:41,603 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:25:01,633 INFO repeater task os75nb_10pings starting 0.6 seconds late 2014-12-20 07:25:01,759 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:25:01,760 INFO repeater task ashtech_hcorrstats starting 0.7 seconds late 2014-12-20 07:25:02,641 INFO repeater task ashtech_hcorrstats ended after 0.9 seconds 2014-12-20 07:25:02,641 INFO repeater task phins3_hcorrstats starting 1.6 seconds late 2014-12-20 07:25:03,368 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:25:13,377 INFO repeater task os150nb_lastensq starting 1.4 seconds late 2014-12-20 07:25:15,277 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:25:15,277 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:25:02 2014-12-20 07:27:05,416 INFO repeater task os150nb_10pings starting 4.4 seconds late 2014-12-20 07:27:05,552 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:27:10,560 INFO repeater task os75nb_lastensq starting 1.6 seconds late 2014-12-20 07:27:11,962 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:27:11,963 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:26:59 2014-12-20 07:27:16,969 INFO repeater task os150nb_lastensq starting 5.0 seconds late 2014-12-20 07:27:18,796 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:27:18,796 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:27:02 2014-12-20 07:28:23,886 INFO repeater task os75nb_10pings starting 2.9 seconds late 2014-12-20 07:28:24,009 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:29:14,072 INFO repeater task os150nb_lastensq starting 2.1 seconds late 2014-12-20 07:29:16,002 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:29:16,002 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:29:02 2014-12-20 07:29:26,017 INFO repeater task os150nb_10pings starting 5.0 seconds late 2014-12-20 07:29:26,140 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:30:01,185 INFO repeater task os150nb_quick starting 0.2 seconds late 2014-12-20 07:30:10,357 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:30:10,357 INFO repeater task os150nb_quick ended after 9.2 seconds 2014-12-20 07:30:10,357 INFO repeater task os150nb_3dayplots starting 9.3 seconds late 2014-12-20 07:30:23,245 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:30:23,245 INFO repeater task os150nb_3dayplots ended after 12.9 seconds 2014-12-20 07:30:23,245 INFO repeater task os150nb_tsstats starting 22.2 seconds late 2014-12-20 07:30:24,338 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 07:30:24,338 INFO repeater task os75nb_lastensq starting 15.3 seconds late 2014-12-20 07:30:25,722 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:30:25,722 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:29:59 2014-12-20 07:30:25,723 INFO repeater task os75nb_quick starting 24.7 seconds late 2014-12-20 07:30:32,573 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:30:32,573 INFO repeater task os75nb_quick ended after 6.9 seconds 2014-12-20 07:30:32,574 INFO repeater task os75nb_3dayplots starting 31.5 seconds late 2014-12-20 07:30:45,269 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:30:45,269 INFO repeater task os75nb_3dayplots ended after 12.7 seconds 2014-12-20 07:30:45,270 INFO repeater task os75nb_tsstats starting 44.2 seconds late 2014-12-20 07:30:46,223 INFO repeater task os75nb_tsstats ended after 1.0 seconds 2014-12-20 07:30:46,223 INFO repeater task os150nb_beamplots starting 45.2 seconds late 2014-12-20 07:30:47,439 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:30:47,439 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:30:47,439 INFO repeater task os150nb_beamplots starting 46.4 seconds late 2014-12-20 07:30:48,617 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:30:48,617 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:30:48,617 INFO repeater task os75nb_beamplots starting 47.6 seconds late 2014-12-20 07:30:49,781 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:30:49,781 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2014-12-20 07:30:49,782 INFO repeater task os75nb_beamplots starting 48.8 seconds late 2014-12-20 07:30:50,951 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:30:50,951 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2014-12-20 07:30:50,951 INFO repeater task ashtech_hcorrstats starting 49.9 seconds late 2014-12-20 07:30:51,765 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:30:51,765 INFO repeater task phins3_hcorrstats starting 50.7 seconds late 2014-12-20 07:30:52,522 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 07:31:12,545 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2014-12-20 07:31:14,403 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:31:14,403 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:31:02 2014-12-20 07:31:44,438 INFO repeater task os150nb_10pings starting 3.4 seconds late 2014-12-20 07:31:44,563 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:31:44,564 INFO repeater task os75nb_10pings starting 3.5 seconds late 2014-12-20 07:31:44,685 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:33:09,795 INFO repeater task os75nb_lastensq starting 0.8 seconds late 2014-12-20 07:33:11,176 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:33:11,177 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:32:59 2014-12-20 07:33:16,182 INFO repeater task os150nb_lastensq starting 4.2 seconds late 2014-12-20 07:33:18,020 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:33:18,021 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:33:02 2014-12-20 07:34:03,083 INFO repeater task os150nb_10pings starting 2.1 seconds late 2014-12-20 07:34:03,211 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:35:03,296 INFO repeater task os75nb_10pings starting 2.3 seconds late 2014-12-20 07:35:03,417 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:35:03,418 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2014-12-20 07:35:04,228 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:35:04,228 INFO repeater task phins3_hcorrstats starting 3.2 seconds late 2014-12-20 07:35:04,965 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:35:14,971 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2014-12-20 07:35:16,776 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:35:16,776 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:35:02 2014-12-20 07:36:11,850 INFO repeater task os75nb_lastensq starting 2.9 seconds late 2014-12-20 07:36:13,311 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:36:13,311 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:35:59 2014-12-20 07:36:23,318 INFO repeater task os150nb_10pings starting 2.3 seconds late 2014-12-20 07:36:23,449 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:37:13,518 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2014-12-20 07:37:15,341 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:37:15,342 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:37:02 2014-12-20 07:38:25,430 INFO repeater task os75nb_10pings starting 4.4 seconds late 2014-12-20 07:38:25,562 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:38:45,590 INFO repeater task os150nb_10pings starting 4.6 seconds late 2014-12-20 07:38:45,719 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:39:10,751 INFO repeater task os75nb_lastensq starting 1.8 seconds late 2014-12-20 07:39:12,220 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:39:12,220 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:38:59 2014-12-20 07:39:17,227 INFO repeater task os150nb_lastensq starting 5.2 seconds late 2014-12-20 07:39:19,135 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:39:19,135 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:39:02 2014-12-20 07:40:04,197 INFO repeater task ashtech_hcorrstats starting 3.2 seconds late 2014-12-20 07:40:04,994 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:40:04,994 INFO repeater task phins3_hcorrstats starting 4.0 seconds late 2014-12-20 07:40:05,781 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 07:41:05,861 INFO repeater task os150nb_10pings starting 4.8 seconds late 2014-12-20 07:41:05,985 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:41:15,995 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2014-12-20 07:41:17,908 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:41:17,908 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:41:02 2014-12-20 07:41:42,941 INFO repeater task os75nb_10pings starting 1.9 seconds late 2014-12-20 07:41:43,065 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:42:13,105 INFO repeater task os75nb_lastensq starting 4.1 seconds late 2014-12-20 07:42:14,493 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:42:14,493 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:41:59 2014-12-20 07:43:14,575 INFO repeater task os150nb_lastensq starting 2.6 seconds late 2014-12-20 07:43:16,462 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:43:16,462 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:43:02 2014-12-20 07:43:21,470 INFO repeater task os150nb_10pings starting 0.4 seconds late 2014-12-20 07:43:21,604 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:45:01,739 INFO repeater task os150nb_quick starting 0.7 seconds late 2014-12-20 07:45:10,860 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:45:10,860 INFO repeater task os150nb_quick ended after 9.1 seconds 2014-12-20 07:45:10,860 INFO repeater task os150nb_3dayplots starting 9.8 seconds late 2014-12-20 07:45:23,471 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:45:23,471 INFO repeater task os150nb_3dayplots ended after 12.6 seconds 2014-12-20 07:45:23,471 INFO repeater task os150nb_tsstats starting 22.4 seconds late 2014-12-20 07:45:24,582 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2014-12-20 07:45:24,583 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 07:45:24,583 INFO repeater task os75nb_lastensq starting 15.6 seconds late 2014-12-20 07:45:25,964 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:45:25,965 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:44:59 2014-12-20 07:45:25,965 INFO repeater task os75nb_10pings starting 24.9 seconds late 2014-12-20 07:45:26,087 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:45:26,088 INFO repeater task os75nb_quick starting 25.1 seconds late 2014-12-20 07:45:32,989 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 07:45:32,990 INFO repeater task os75nb_quick ended after 6.9 seconds 2014-12-20 07:45:32,990 INFO repeater task os75nb_3dayplots starting 32.0 seconds late 2014-12-20 07:45:45,857 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 07:45:45,857 INFO repeater task os75nb_3dayplots ended after 12.9 seconds 2014-12-20 07:45:45,857 INFO repeater task os75nb_tsstats starting 44.8 seconds late 2014-12-20 07:45:46,757 INFO repeater task os75nb_tsstats ended after 0.9 seconds 2014-12-20 07:45:46,757 INFO repeater task os150nb_beamplots starting 45.7 seconds late 2014-12-20 07:45:48,011 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:45:48,011 INFO repeater task os150nb_beamplots ended after 1.3 seconds 2014-12-20 07:45:48,011 INFO repeater task os150nb_beamplots starting 47.0 seconds late 2014-12-20 07:45:49,210 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:45:49,210 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 07:45:49,210 INFO repeater task os75nb_beamplots starting 48.2 seconds late 2014-12-20 07:45:50,316 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:45:50,317 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 07:45:50,317 INFO repeater task os75nb_beamplots starting 49.3 seconds late 2014-12-20 07:45:51,534 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 07:45:51,535 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2014-12-20 07:45:51,535 INFO repeater task ashtech_hcorrstats starting 50.5 seconds late 2014-12-20 07:45:52,364 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:45:52,364 INFO repeater task phins3_hcorrstats starting 51.3 seconds late 2014-12-20 07:45:53,139 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 07:45:58,142 INFO repeater task os150nb_lastensq starting 46.1 seconds late 2014-12-20 07:45:59,878 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2014-12-20 07:45:59,878 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:45:02 2014-12-20 07:45:59,878 INFO repeater task os150nb_10pings starting 18.9 seconds late 2014-12-20 07:46:00,000 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:47:15,102 INFO repeater task os150nb_lastensq starting 3.1 seconds late 2014-12-20 07:47:16,995 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:47:16,996 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:47:02 2014-12-20 07:48:02,061 INFO repeater task os150nb_10pings starting 1.0 seconds late 2014-12-20 07:48:02,187 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:48:12,203 INFO repeater task os75nb_lastensq starting 3.2 seconds late 2014-12-20 07:48:13,582 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:48:13,582 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:47:59 2014-12-20 07:48:23,594 INFO repeater task os75nb_10pings starting 2.6 seconds late 2014-12-20 07:48:23,730 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:49:13,799 INFO repeater task os150nb_lastensq starting 1.8 seconds late 2014-12-20 07:49:15,586 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:49:15,586 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:49:02 2014-12-20 07:50:05,641 INFO repeater task ashtech_hcorrstats starting 4.6 seconds late 2014-12-20 07:50:06,461 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:50:06,462 INFO repeater task phins3_hcorrstats starting 5.4 seconds late 2014-12-20 07:50:07,200 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:50:22,216 INFO repeater task os150nb_10pings starting 1.2 seconds late 2014-12-20 07:50:22,341 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:51:12,403 INFO repeater task os150nb_lastensq starting 0.4 seconds late 2014-12-20 07:51:14,179 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:51:14,179 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:51:02 2014-12-20 07:51:14,180 INFO repeater task os75nb_lastensq starting 5.2 seconds late 2014-12-20 07:51:15,603 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:51:15,603 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:50:59 2014-12-20 07:51:45,646 INFO repeater task os75nb_10pings starting 4.6 seconds late 2014-12-20 07:51:45,777 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:52:45,855 INFO repeater task os150nb_10pings starting 4.8 seconds late 2014-12-20 07:52:45,985 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:53:16,024 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2014-12-20 07:53:17,856 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:53:17,856 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:53:02 2014-12-20 07:54:12,927 INFO repeater task os75nb_lastensq starting 3.9 seconds late 2014-12-20 07:54:14,383 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 07:54:14,384 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:53:59 2014-12-20 07:55:04,448 INFO repeater task os150nb_10pings starting 3.4 seconds late 2014-12-20 07:55:04,586 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:55:04,587 INFO repeater task os75nb_10pings starting 3.6 seconds late 2014-12-20 07:55:04,727 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:55:04,728 INFO repeater task ashtech_hcorrstats starting 3.7 seconds late 2014-12-20 07:55:05,562 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 07:55:05,563 INFO repeater task phins3_hcorrstats starting 4.5 seconds late 2014-12-20 07:55:06,302 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 07:55:16,314 INFO repeater task os150nb_lastensq starting 4.3 seconds late 2014-12-20 07:55:18,181 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:55:18,181 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:55:02 2014-12-20 07:57:13,321 INFO repeater task os150nb_lastensq starting 1.3 seconds late 2014-12-20 07:57:15,184 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 07:57:15,184 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:57:02 2014-12-20 07:57:15,185 INFO repeater task os75nb_lastensq starting 6.2 seconds late 2014-12-20 07:57:16,609 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 07:57:16,609 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:56:59 2014-12-20 07:57:21,616 INFO repeater task os150nb_10pings starting 0.6 seconds late 2014-12-20 07:57:21,745 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 07:58:21,821 INFO repeater task os75nb_10pings starting 0.8 seconds late 2014-12-20 07:58:21,953 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 07:59:12,021 INFO repeater task os150nb_lastensq starting 0.0 seconds late 2014-12-20 07:59:13,801 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 07:59:13,801 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 07:59:02 2014-12-20 07:59:43,842 INFO repeater task os150nb_10pings starting 2.8 seconds late 2014-12-20 07:59:43,968 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 08:00:03,994 INFO repeater task os150nb_quick starting 3.0 seconds late 2014-12-20 08:00:13,479 INFO repeater task os150nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 08:00:13,479 INFO repeater task os150nb_quick ended after 9.5 seconds 2014-12-20 08:00:13,480 INFO repeater task os150nb_3dayplots starting 12.5 seconds late 2014-12-20 08:00:26,059 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 08:00:26,060 INFO repeater task os150nb_3dayplots ended after 12.6 seconds 2014-12-20 08:00:26,060 INFO repeater task os150nb_tsstats starting 25.0 seconds late 2014-12-20 08:00:27,175 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2014-12-20 08:00:27,175 INFO repeater task os150nb_tsstats ended after 1.1 seconds 2014-12-20 08:00:27,176 INFO repeater task os75nb_lastensq starting 18.2 seconds late 2014-12-20 08:00:28,657 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2014-12-20 08:00:28,658 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 07:59:59 2014-12-20 08:00:28,658 INFO repeater task os75nb_quick starting 27.6 seconds late 2014-12-20 08:00:35,832 INFO repeater task os75nb_quick output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:3847: UserWarning: Warning: converting a masked element to nan. warnings.warn("Warning: converting a masked element to nan.") 2014-12-20 08:00:35,833 INFO repeater task os75nb_quick ended after 7.2 seconds 2014-12-20 08:00:35,833 INFO repeater task os75nb_3dayplots starting 34.8 seconds late 2014-12-20 08:00:49,139 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2014-12-20 08:00:49,139 INFO repeater task os75nb_3dayplots ended after 13.3 seconds 2014-12-20 08:00:49,139 INFO repeater task os75nb_tsstats starting 48.1 seconds late 2014-12-20 08:00:50,040 INFO repeater task os75nb_tsstats ended after 0.9 seconds 2014-12-20 08:00:50,040 INFO repeater task os150nb_beamplots starting 49.0 seconds late 2014-12-20 08:00:51,208 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 08:00:51,208 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 08:00:51,208 INFO repeater task os150nb_beamplots starting 50.2 seconds late 2014-12-20 08:00:52,390 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 08:00:52,390 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2014-12-20 08:00:52,390 INFO repeater task os75nb_beamplots starting 51.4 seconds late 2014-12-20 08:00:53,503 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 08:00:53,503 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 08:00:53,504 INFO repeater task os75nb_beamplots starting 52.5 seconds late 2014-12-20 08:00:54,623 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2014-12-20 08:00:54,624 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2014-12-20 08:00:54,624 INFO repeater task ashtech_hcorrstats starting 53.6 seconds late 2014-12-20 08:00:55,468 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 08:00:55,469 INFO repeater task phins3_hcorrstats starting 54.4 seconds late 2014-12-20 08:00:56,219 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2014-12-20 08:01:16,237 INFO repeater task os150nb_lastensq starting 4.2 seconds late 2014-12-20 08:01:18,009 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 08:01:18,009 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 08:01:02 2014-12-20 08:01:43,042 INFO repeater task os75nb_10pings starting 2.0 seconds late 2014-12-20 08:01:43,164 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 08:02:03,193 INFO repeater task os150nb_10pings starting 2.2 seconds late 2014-12-20 08:02:03,326 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 08:03:13,417 INFO repeater task os150nb_lastensq starting 1.4 seconds late 2014-12-20 08:03:15,194 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 08:03:15,194 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 08:03:02 2014-12-20 08:03:15,195 INFO repeater task os75nb_lastensq starting 6.2 seconds late 2014-12-20 08:03:16,573 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 08:03:16,573 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 08:02:59 2014-12-20 08:04:21,650 INFO repeater task os150nb_10pings starting 0.6 seconds late 2014-12-20 08:04:21,783 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 08:05:01,836 INFO repeater task os75nb_10pings starting 0.8 seconds late 2014-12-20 08:05:01,959 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 08:05:01,960 INFO repeater task ashtech_hcorrstats starting 0.9 seconds late 2014-12-20 08:05:02,747 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2014-12-20 08:05:02,748 INFO repeater task phins3_hcorrstats starting 1.7 seconds late 2014-12-20 08:05:03,470 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2014-12-20 08:05:13,474 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2014-12-20 08:05:15,356 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 08:05:15,356 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 08:05:02 2014-12-20 08:06:10,430 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2014-12-20 08:06:11,808 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 08:06:11,809 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 08:05:59 2014-12-20 08:06:41,847 INFO repeater task os150nb_10pings starting 0.8 seconds late 2014-12-20 08:06:41,990 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 08:07:12,022 INFO repeater task os150nb_lastensq starting 0.0 seconds late 2014-12-20 08:07:13,921 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2014-12-20 08:07:13,921 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 08:07:02 2014-12-20 08:08:24,022 INFO repeater task os75nb_10pings starting 3.0 seconds late 2014-12-20 08:08:24,148 INFO repeater task os75nb_10pings ended after 0.1 seconds 2014-12-20 08:09:04,200 INFO repeater task os150nb_10pings starting 3.2 seconds late 2014-12-20 08:09:04,326 INFO repeater task os150nb_10pings ended after 0.1 seconds 2014-12-20 08:09:09,334 INFO repeater task os75nb_lastensq starting 0.3 seconds late 2014-12-20 08:09:10,781 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2014-12-20 08:09:10,781 INFO repeater task os75nb_lastensq: Time from /home/data/RR1413/proc/os75nb/load/wrote_ens_time: 2014/12/20 08:08:59 2014-12-20 08:09:15,784 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2014-12-20 08:09:17,626 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2014-12-20 08:09:17,626 INFO repeater task os150nb_lastensq: Time from /home/data/RR1413/proc/os150nb/load/wrote_ens_time: 2014/12/20 08:09:02 2014-12-20 08:09:57,199 INFO Translate translate exiting, gps /home/data/RR1413/rbin/ashtech 2014-12-20 08:09:57,203 INFO Translate translate exiting, gps /home/data/RR1413/rbin/hydrins 2014-12-20 08:09:57,208 INFO Translate translate exiting, hdg /home/data/RR1413/rbin/phins3 2014-12-20 08:09:57,219 INFO Translate translate exiting, adu /home/data/RR1413/rbin/ashtech 2014-12-20 08:09:57,221 INFO Translate translate exiting, gps /home/data/RR1413/rbin/gpsnav 2014-12-20 08:09:57,238 INFO Translate translate exiting, hdg /home/data/RR1413/rbin/hydrins 2014-12-20 08:09:57,681 INFO root cleanup_figs not_logging 2014-12-20 08:09:57,784 INFO root Exiting DAS_while_logging.py