2015-03-17 19:08:58,798 INFO repeater task os75nb_tsstats starting 22.9 seconds late 2015-03-17 19:08:59,461 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 19:08:59,462 INFO repeater task os150nb_beamplots starting 23.6 seconds late 2015-03-17 19:09:00,641 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:09:00,642 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 19:09:00,642 INFO repeater task os150nb_beamplots starting 24.7 seconds late 2015-03-17 19:09:01,909 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:09:01,909 INFO repeater task os150nb_beamplots ended after 1.3 seconds 2015-03-17 19:09:01,909 INFO repeater task os75nb_beamplots starting 26.0 seconds late 2015-03-17 19:09:02,994 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:09:02,994 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 19:09:02,994 INFO repeater task os75nb_beamplots starting 27.1 seconds late 2015-03-17 19:09:04,152 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:09:04,152 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 19:09:04,152 INFO repeater task ashtech_hcorrstats starting 28.3 seconds late 2015-03-17 19:09:04,936 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:09:04,936 INFO repeater task phins3_hcorrstats starting 29.0 seconds late 2015-03-17 19:09:05,709 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:09:10,714 INFO repeater task os150nb_lastensq starting 24.7 seconds late 2015-03-17 19:09:12,459 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 19:09:12,459 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:08:36 2015-03-17 19:10:17,549 INFO repeater task os150nb_10pings starting 1.7 seconds late 2015-03-17 19:10:17,678 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:10:47,718 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-03-17 19:10:49,608 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:10:49,609 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:10:36 2015-03-17 19:11:49,697 INFO repeater task os75nb_lastensq starting 3.7 seconds late 2015-03-17 19:11:51,083 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:11:51,083 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:11:36 2015-03-17 19:11:56,090 INFO repeater task os75nb_10pings starting 0.2 seconds late 2015-03-17 19:11:56,231 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:12:36,286 INFO repeater task os150nb_10pings starting 0.4 seconds late 2015-03-17 19:12:36,415 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:12:46,431 INFO repeater task os150nb_lastensq starting 0.4 seconds late 2015-03-17 19:12:48,217 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:12:48,217 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:12:36 2015-03-17 19:13:38,292 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2015-03-17 19:13:39,135 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:13:39,135 INFO repeater task phins3_hcorrstats starting 3.2 seconds late 2015-03-17 19:13:39,911 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:14:50,004 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2015-03-17 19:14:51,907 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:14:51,907 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:14:36 2015-03-17 19:14:51,908 INFO repeater task os75nb_lastensq starting 5.9 seconds late 2015-03-17 19:14:53,296 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:14:53,296 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:14:36 2015-03-17 19:14:58,303 INFO repeater task os150nb_10pings starting 2.4 seconds late 2015-03-17 19:14:58,430 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:15:18,459 INFO repeater task os75nb_10pings starting 2.6 seconds late 2015-03-17 19:15:18,588 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:16:48,710 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2015-03-17 19:16:50,537 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:16:50,538 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:16:36 2015-03-17 19:17:20,575 INFO repeater task os150nb_10pings starting 4.7 seconds late 2015-03-17 19:17:20,702 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:17:50,748 INFO repeater task os75nb_lastensq starting 4.7 seconds late 2015-03-17 19:17:52,145 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:17:52,145 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:17:36 2015-03-17 19:18:37,208 INFO repeater task os75nb_10pings starting 1.3 seconds late 2015-03-17 19:18:37,343 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:18:37,345 INFO repeater task ashtech_hcorrstats starting 1.4 seconds late 2015-03-17 19:18:38,138 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:18:38,138 INFO repeater task phins3_hcorrstats starting 2.2 seconds late 2015-03-17 19:18:38,911 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:18:48,924 INFO repeater task os150nb_lastensq starting 2.9 seconds late 2015-03-17 19:18:50,808 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:18:50,808 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:18:36 2015-03-17 19:19:40,878 INFO repeater task os150nb_10pings starting 5.0 seconds late 2015-03-17 19:19:41,004 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:20:46,094 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-03-17 19:20:47,908 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:20:47,908 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:20:36 2015-03-17 19:20:47,909 INFO repeater task os75nb_lastensq starting 1.9 seconds late 2015-03-17 19:20:49,275 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:20:49,276 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:20:36 2015-03-17 19:21:59,372 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-03-17 19:21:59,504 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:21:59,505 INFO repeater task os75nb_10pings starting 3.6 seconds late 2015-03-17 19:21:59,633 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:22:49,702 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-03-17 19:22:51,567 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:22:51,567 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:22:36 2015-03-17 19:23:36,631 INFO repeater task os150nb_quick starting 0.7 seconds late 2015-03-17 19:23:40,663 INFO repeater task os150nb_quick ended after 4.0 seconds 2015-03-17 19:23:40,664 INFO repeater task os150nb_3dayplots starting 4.8 seconds late 2015-03-17 19:23:45,449 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 19:23:45,449 INFO repeater task os150nb_3dayplots ended after 4.8 seconds 2015-03-17 19:23:45,449 INFO repeater task os150nb_tsstats starting 9.6 seconds late 2015-03-17 19:23:46,141 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-17 19:23:46,142 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2015-03-17 19:23:47,534 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:23:47,534 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:23:36 2015-03-17 19:23:47,535 INFO repeater task os75nb_quick starting 11.6 seconds late 2015-03-17 19:23:51,233 INFO repeater task os75nb_quick ended after 3.7 seconds 2015-03-17 19:23:51,233 INFO repeater task os75nb_3dayplots starting 15.3 seconds late 2015-03-17 19:23:56,126 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 19:23:56,126 INFO repeater task os75nb_3dayplots ended after 4.9 seconds 2015-03-17 19:23:56,127 INFO repeater task os75nb_tsstats starting 20.2 seconds late 2015-03-17 19:23:56,832 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 19:23:56,833 INFO repeater task os150nb_beamplots starting 20.9 seconds late 2015-03-17 19:23:58,062 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:23:58,062 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 19:23:58,062 INFO repeater task os150nb_beamplots starting 22.2 seconds late 2015-03-17 19:23:59,310 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:23:59,310 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 19:23:59,311 INFO repeater task os75nb_beamplots starting 23.4 seconds late 2015-03-17 19:24:00,443 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:24:00,443 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 19:24:00,443 INFO repeater task os75nb_beamplots starting 24.5 seconds late 2015-03-17 19:24:01,582 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:24:01,582 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 19:24:01,583 INFO repeater task ashtech_hcorrstats starting 25.7 seconds late 2015-03-17 19:24:02,408 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:24:02,408 INFO repeater task phins3_hcorrstats starting 26.5 seconds late 2015-03-17 19:24:03,185 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:24:18,204 INFO repeater task os150nb_10pings starting 2.3 seconds late 2015-03-17 19:24:18,344 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:24:48,385 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-03-17 19:24:50,147 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:24:50,148 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:24:36 2015-03-17 19:25:20,187 INFO repeater task os75nb_10pings starting 4.3 seconds late 2015-03-17 19:25:20,316 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:26:40,430 INFO repeater task os150nb_10pings starting 4.5 seconds late 2015-03-17 19:26:40,559 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:26:50,569 INFO repeater task os150nb_lastensq starting 4.6 seconds late 2015-03-17 19:26:52,436 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:26:52,436 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:26:36 2015-03-17 19:26:52,437 INFO repeater task os75nb_lastensq starting 6.4 seconds late 2015-03-17 19:26:53,838 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:26:53,838 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:26:36 2015-03-17 19:28:38,990 INFO repeater task os75nb_10pings starting 3.1 seconds late 2015-03-17 19:28:39,119 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:28:39,120 INFO repeater task ashtech_hcorrstats starting 3.2 seconds late 2015-03-17 19:28:39,903 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:28:39,903 INFO repeater task phins3_hcorrstats starting 4.0 seconds late 2015-03-17 19:28:40,693 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:28:50,706 INFO repeater task os150nb_lastensq starting 4.7 seconds late 2015-03-17 19:28:52,494 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:28:52,494 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:28:36 2015-03-17 19:28:57,501 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-03-17 19:28:57,632 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:29:47,705 INFO repeater task os75nb_lastensq starting 1.7 seconds late 2015-03-17 19:29:49,140 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:29:49,140 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:29:36 2015-03-17 19:30:49,221 INFO repeater task os150nb_lastensq starting 3.2 seconds late 2015-03-17 19:30:51,019 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:30:51,019 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:30:36 2015-03-17 19:31:16,060 INFO repeater task os150nb_10pings starting 0.2 seconds late 2015-03-17 19:31:16,189 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:31:56,237 INFO repeater task os75nb_10pings starting 0.3 seconds late 2015-03-17 19:31:56,371 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:32:46,432 INFO repeater task os150nb_lastensq starting 0.4 seconds late 2015-03-17 19:32:48,197 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:32:48,198 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:32:36 2015-03-17 19:32:48,199 INFO repeater task os75nb_lastensq starting 2.2 seconds late 2015-03-17 19:32:49,550 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:32:49,550 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:32:36 2015-03-17 19:33:39,616 INFO repeater task os150nb_10pings starting 3.7 seconds late 2015-03-17 19:33:39,754 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:33:39,758 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2015-03-17 19:33:40,560 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:33:40,561 INFO repeater task phins3_hcorrstats starting 4.7 seconds late 2015-03-17 19:33:41,322 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:34:46,403 INFO repeater task os150nb_lastensq starting 0.4 seconds late 2015-03-17 19:34:48,200 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:34:48,201 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:34:36 2015-03-17 19:35:18,240 INFO repeater task os75nb_10pings starting 2.3 seconds late 2015-03-17 19:35:18,370 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:35:48,406 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2015-03-17 19:35:49,829 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:35:49,829 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:35:36 2015-03-17 19:35:59,843 INFO repeater task os150nb_10pings starting 3.9 seconds late 2015-03-17 19:35:59,978 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:36:50,044 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2015-03-17 19:36:51,849 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:36:51,849 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:36:36 2015-03-17 19:38:16,969 INFO repeater task os150nb_10pings starting 1.1 seconds late 2015-03-17 19:38:17,095 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:38:37,124 INFO repeater task os150nb_quick starting 1.2 seconds late 2015-03-17 19:38:41,307 INFO repeater task os150nb_quick ended after 4.2 seconds 2015-03-17 19:38:41,307 INFO repeater task os150nb_3dayplots starting 5.4 seconds late 2015-03-17 19:38:46,475 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 19:38:46,476 INFO repeater task os150nb_3dayplots ended after 5.2 seconds 2015-03-17 19:38:46,476 INFO repeater task os150nb_tsstats starting 10.6 seconds late 2015-03-17 19:38:47,197 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-17 19:38:47,198 INFO repeater task os75nb_lastensq starting 1.2 seconds late 2015-03-17 19:38:48,580 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:38:48,580 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:38:36 2015-03-17 19:38:48,580 INFO repeater task os75nb_10pings starting 12.7 seconds late 2015-03-17 19:38:48,714 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:38:48,715 INFO repeater task os75nb_quick starting 12.8 seconds late 2015-03-17 19:38:52,430 INFO repeater task os75nb_quick ended after 3.7 seconds 2015-03-17 19:38:52,430 INFO repeater task os75nb_3dayplots starting 16.5 seconds late 2015-03-17 19:38:57,404 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 19:38:57,405 INFO repeater task os75nb_3dayplots ended after 5.0 seconds 2015-03-17 19:38:57,405 INFO repeater task os75nb_tsstats starting 21.5 seconds late 2015-03-17 19:38:58,109 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 19:38:58,109 INFO repeater task os150nb_beamplots starting 22.2 seconds late 2015-03-17 19:38:59,374 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:38:59,374 INFO repeater task os150nb_beamplots ended after 1.3 seconds 2015-03-17 19:38:59,374 INFO repeater task os150nb_beamplots starting 23.5 seconds late 2015-03-17 19:39:00,631 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:39:00,631 INFO repeater task os150nb_beamplots ended after 1.3 seconds 2015-03-17 19:39:00,631 INFO repeater task os75nb_beamplots starting 24.7 seconds late 2015-03-17 19:39:01,833 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:39:01,834 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 19:39:01,834 INFO repeater task os75nb_beamplots starting 25.9 seconds late 2015-03-17 19:39:02,984 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:39:02,984 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 19:39:02,985 INFO repeater task ashtech_hcorrstats starting 27.1 seconds late 2015-03-17 19:39:03,770 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:39:03,771 INFO repeater task phins3_hcorrstats starting 27.9 seconds late 2015-03-17 19:39:04,543 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:39:09,546 INFO repeater task os150nb_lastensq starting 23.5 seconds late 2015-03-17 19:39:11,304 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:39:11,304 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:38:36 2015-03-17 19:40:36,422 INFO repeater task os150nb_10pings starting 0.5 seconds late 2015-03-17 19:40:36,555 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:40:46,570 INFO repeater task os150nb_lastensq starting 0.6 seconds late 2015-03-17 19:40:48,361 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:40:48,361 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:40:36 2015-03-17 19:41:48,442 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2015-03-17 19:41:49,879 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:41:49,879 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:41:36 2015-03-17 19:41:59,894 INFO repeater task os75nb_10pings starting 4.0 seconds late 2015-03-17 19:42:00,024 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:42:50,098 INFO repeater task os150nb_lastensq starting 4.1 seconds late 2015-03-17 19:42:51,959 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:42:51,959 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:42:36 2015-03-17 19:42:56,967 INFO repeater task os150nb_10pings starting 1.1 seconds late 2015-03-17 19:42:57,095 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:43:37,152 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2015-03-17 19:43:37,917 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:43:37,917 INFO repeater task phins3_hcorrstats starting 2.0 seconds late 2015-03-17 19:43:38,712 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:44:48,801 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2015-03-17 19:44:50,621 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:44:50,622 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:44:36 2015-03-17 19:44:50,623 INFO repeater task os75nb_lastensq starting 4.6 seconds late 2015-03-17 19:44:51,965 INFO repeater task os75nb_lastensq ended after 1.3 seconds 2015-03-17 19:44:51,965 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:44:36 2015-03-17 19:45:16,996 INFO repeater task os150nb_10pings starting 1.1 seconds late 2015-03-17 19:45:17,133 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:45:17,134 INFO repeater task os75nb_10pings starting 1.2 seconds late 2015-03-17 19:45:17,262 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:46:47,380 INFO repeater task os150nb_lastensq starting 1.4 seconds late 2015-03-17 19:46:49,186 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:46:49,187 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:46:36 2015-03-17 19:47:39,256 INFO repeater task os150nb_10pings starting 3.4 seconds late 2015-03-17 19:47:39,392 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:47:49,406 INFO repeater task os75nb_lastensq starting 3.4 seconds late 2015-03-17 19:47:50,820 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:47:50,821 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:47:36 2015-03-17 19:48:40,889 INFO repeater task os75nb_10pings starting 5.0 seconds late 2015-03-17 19:48:41,017 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:48:41,018 INFO repeater task ashtech_hcorrstats starting 5.1 seconds late 2015-03-17 19:48:41,773 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:48:41,773 INFO repeater task phins3_hcorrstats starting 5.9 seconds late 2015-03-17 19:48:42,517 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-17 19:48:47,522 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2015-03-17 19:48:49,388 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:48:49,389 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:48:36 2015-03-17 19:49:59,477 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-03-17 19:49:59,610 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:50:49,681 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-03-17 19:50:51,500 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:50:51,500 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:50:36 2015-03-17 19:50:51,501 INFO repeater task os75nb_lastensq starting 5.5 seconds late 2015-03-17 19:50:52,883 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:50:52,884 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:50:36 2015-03-17 19:51:57,966 INFO repeater task os75nb_10pings starting 2.1 seconds late 2015-03-17 19:51:58,088 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:52:18,116 INFO repeater task os150nb_10pings starting 2.2 seconds late 2015-03-17 19:52:18,243 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:52:48,280 INFO repeater task os150nb_lastensq starting 2.3 seconds late 2015-03-17 19:52:50,127 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:52:50,128 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:52:36 2015-03-17 19:53:40,199 INFO repeater task os150nb_quick starting 4.3 seconds late 2015-03-17 19:53:44,477 INFO repeater task os150nb_quick ended after 4.3 seconds 2015-03-17 19:53:44,478 INFO repeater task os150nb_3dayplots starting 8.6 seconds late 2015-03-17 19:53:49,569 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 19:53:49,569 INFO repeater task os150nb_3dayplots ended after 5.1 seconds 2015-03-17 19:53:49,569 INFO repeater task os150nb_tsstats starting 13.7 seconds late 2015-03-17 19:53:50,331 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 19:53:50,332 INFO repeater task os150nb_tsstats ended after 0.8 seconds 2015-03-17 19:53:50,332 INFO repeater task os75nb_lastensq starting 4.3 seconds late 2015-03-17 19:53:51,794 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-17 19:53:51,794 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:53:36 2015-03-17 19:53:51,795 INFO repeater task os75nb_quick starting 15.9 seconds late 2015-03-17 19:53:55,680 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-17 19:53:55,680 INFO repeater task os75nb_3dayplots starting 19.8 seconds late 2015-03-17 19:54:00,767 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 19:54:00,767 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2015-03-17 19:54:00,768 INFO repeater task os75nb_tsstats starting 24.9 seconds late 2015-03-17 19:54:01,509 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 19:54:01,510 INFO repeater task os150nb_beamplots starting 25.6 seconds late 2015-03-17 19:54:02,749 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:54:02,749 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 19:54:02,749 INFO repeater task os150nb_beamplots starting 26.8 seconds late 2015-03-17 19:54:03,999 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:54:03,999 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 19:54:04,000 INFO repeater task os75nb_beamplots starting 28.1 seconds late 2015-03-17 19:54:05,182 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:54:05,183 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 19:54:05,183 INFO repeater task os75nb_beamplots starting 29.3 seconds late 2015-03-17 19:54:06,400 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 19:54:06,400 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 19:54:06,401 INFO repeater task ashtech_hcorrstats starting 30.5 seconds late 2015-03-17 19:54:07,158 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:54:07,159 INFO repeater task phins3_hcorrstats starting 31.3 seconds late 2015-03-17 19:54:07,913 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:54:37,956 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-03-17 19:54:38,084 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:54:48,099 INFO repeater task os150nb_lastensq starting 2.1 seconds late 2015-03-17 19:54:49,952 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 19:54:49,952 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:54:36 2015-03-17 19:55:19,993 INFO repeater task os75nb_10pings starting 4.1 seconds late 2015-03-17 19:55:20,121 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:56:50,246 INFO repeater task os150nb_lastensq starting 4.2 seconds late 2015-03-17 19:56:52,074 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:56:52,074 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:56:36 2015-03-17 19:56:52,075 INFO repeater task os75nb_lastensq starting 6.1 seconds late 2015-03-17 19:56:53,464 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:56:53,464 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:56:36 2015-03-17 19:56:58,470 INFO repeater task os150nb_10pings starting 2.6 seconds late 2015-03-17 19:56:58,603 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:58:38,747 INFO repeater task os75nb_10pings starting 2.8 seconds late 2015-03-17 19:58:38,876 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 19:58:38,877 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2015-03-17 19:58:39,633 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 19:58:39,634 INFO repeater task phins3_hcorrstats starting 3.7 seconds late 2015-03-17 19:58:40,390 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 19:58:50,402 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2015-03-17 19:58:52,192 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 19:58:52,192 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 19:58:36 2015-03-17 19:59:17,230 INFO repeater task os150nb_10pings starting 1.3 seconds late 2015-03-17 19:59:17,349 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 19:59:47,395 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2015-03-17 19:59:48,818 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 19:59:48,818 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 19:59:36 2015-03-17 20:00:48,904 INFO repeater task os150nb_lastensq starting 2.9 seconds late 2015-03-17 20:00:50,790 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:00:50,790 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:00:36 2015-03-17 20:01:40,854 INFO repeater task os150nb_10pings starting 5.0 seconds late 2015-03-17 20:01:40,981 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:01:56,003 INFO repeater task os75nb_10pings starting 0.1 seconds late 2015-03-17 20:01:56,135 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:02:46,203 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-17 20:02:47,962 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:02:47,963 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:02:36 2015-03-17 20:02:47,963 INFO repeater task os75nb_lastensq starting 2.0 seconds late 2015-03-17 20:02:49,379 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:02:49,380 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:02:36 2015-03-17 20:03:39,447 INFO repeater task ashtech_hcorrstats starting 3.5 seconds late 2015-03-17 20:03:40,211 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:03:40,212 INFO repeater task phins3_hcorrstats starting 4.3 seconds late 2015-03-17 20:03:40,970 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:03:55,990 INFO repeater task os150nb_10pings starting 0.1 seconds late 2015-03-17 20:03:56,119 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:04:46,189 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-17 20:04:48,082 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:04:48,082 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:04:36 2015-03-17 20:05:18,123 INFO repeater task os75nb_10pings starting 2.2 seconds late 2015-03-17 20:05:18,251 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:05:48,293 INFO repeater task os75nb_lastensq starting 2.3 seconds late 2015-03-17 20:05:49,656 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:05:49,657 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:05:36 2015-03-17 20:06:19,697 INFO repeater task os150nb_10pings starting 3.8 seconds late 2015-03-17 20:06:19,832 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:06:49,877 INFO repeater task os150nb_lastensq starting 3.9 seconds late 2015-03-17 20:06:51,587 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 20:06:51,587 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:06:36 2015-03-17 20:08:36,726 INFO repeater task os150nb_10pings starting 0.8 seconds late 2015-03-17 20:08:36,862 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:08:36,862 INFO repeater task os150nb_quick starting 1.0 seconds late 2015-03-17 20:08:40,868 INFO repeater task os150nb_quick ended after 4.0 seconds 2015-03-17 20:08:40,868 INFO repeater task os150nb_3dayplots starting 5.0 seconds late 2015-03-17 20:08:45,642 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:08:45,643 INFO repeater task os150nb_3dayplots ended after 4.8 seconds 2015-03-17 20:08:45,643 INFO repeater task os150nb_tsstats starting 9.7 seconds late 2015-03-17 20:08:46,335 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 20:08:46,335 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-17 20:08:46,335 INFO repeater task os75nb_lastensq starting 0.3 seconds late 2015-03-17 20:08:47,698 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:08:47,699 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:08:36 2015-03-17 20:08:47,699 INFO repeater task os75nb_10pings starting 11.8 seconds late 2015-03-17 20:08:47,823 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:08:47,823 INFO repeater task os75nb_quick starting 11.9 seconds late 2015-03-17 20:08:51,637 INFO repeater task os75nb_quick ended after 3.8 seconds 2015-03-17 20:08:51,638 INFO repeater task os75nb_3dayplots starting 15.7 seconds late 2015-03-17 20:08:56,441 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:08:56,442 INFO repeater task os75nb_3dayplots ended after 4.8 seconds 2015-03-17 20:08:56,442 INFO repeater task os75nb_tsstats starting 20.5 seconds late 2015-03-17 20:08:57,132 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 20:08:57,133 INFO repeater task os150nb_beamplots starting 21.2 seconds late 2015-03-17 20:08:58,439 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:08:58,439 INFO repeater task os150nb_beamplots ended after 1.3 seconds 2015-03-17 20:08:58,440 INFO repeater task os150nb_beamplots starting 22.5 seconds late 2015-03-17 20:08:59,614 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:08:59,614 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:08:59,615 INFO repeater task os75nb_beamplots starting 23.7 seconds late 2015-03-17 20:09:00,824 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:09:00,825 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 20:09:00,825 INFO repeater task os75nb_beamplots starting 24.9 seconds late 2015-03-17 20:09:01,943 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:09:01,943 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 20:09:01,944 INFO repeater task ashtech_hcorrstats starting 26.0 seconds late 2015-03-17 20:09:02,706 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:09:02,707 INFO repeater task phins3_hcorrstats starting 26.8 seconds late 2015-03-17 20:09:03,497 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:09:08,503 INFO repeater task os150nb_lastensq starting 22.5 seconds late 2015-03-17 20:09:10,236 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 20:09:10,236 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:08:36 2015-03-17 20:10:50,358 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2015-03-17 20:10:52,110 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:10:52,111 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:10:36 2015-03-17 20:10:57,119 INFO repeater task os150nb_10pings starting 1.2 seconds late 2015-03-17 20:10:57,244 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:11:47,311 INFO repeater task os75nb_lastensq starting 1.3 seconds late 2015-03-17 20:11:48,694 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:11:48,694 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:11:36 2015-03-17 20:11:58,709 INFO repeater task os75nb_10pings starting 2.8 seconds late 2015-03-17 20:11:58,850 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:12:48,915 INFO repeater task os150nb_lastensq starting 2.9 seconds late 2015-03-17 20:12:50,759 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:12:50,759 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:12:36 2015-03-17 20:13:20,795 INFO repeater task os150nb_10pings starting 4.9 seconds late 2015-03-17 20:13:20,926 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:13:35,950 INFO repeater task ashtech_hcorrstats starting 0.0 seconds late 2015-03-17 20:13:36,707 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:13:36,708 INFO repeater task phins3_hcorrstats starting 0.8 seconds late 2015-03-17 20:13:37,439 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-17 20:14:47,526 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2015-03-17 20:14:49,385 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:14:49,386 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:14:36 2015-03-17 20:14:49,386 INFO repeater task os75nb_lastensq starting 3.4 seconds late 2015-03-17 20:14:50,762 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:14:50,763 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:14:36 2015-03-17 20:15:20,802 INFO repeater task os75nb_10pings starting 4.9 seconds late 2015-03-17 20:15:20,935 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:15:35,956 INFO repeater task os150nb_10pings starting 0.1 seconds late 2015-03-17 20:15:36,086 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:16:46,187 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-17 20:16:47,954 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:16:47,954 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:16:36 2015-03-17 20:17:48,038 INFO repeater task os75nb_lastensq starting 2.0 seconds late 2015-03-17 20:17:49,407 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:17:49,408 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:17:36 2015-03-17 20:17:59,423 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-03-17 20:17:59,562 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:18:39,628 INFO repeater task os75nb_10pings starting 3.7 seconds late 2015-03-17 20:18:39,763 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:18:39,764 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2015-03-17 20:18:40,563 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:18:40,563 INFO repeater task phins3_hcorrstats starting 4.7 seconds late 2015-03-17 20:18:41,320 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:18:46,325 INFO repeater task os150nb_lastensq starting 0.3 seconds late 2015-03-17 20:18:48,087 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:18:48,087 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:18:36 2015-03-17 20:20:18,212 INFO repeater task os150nb_10pings starting 2.3 seconds late 2015-03-17 20:20:18,336 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:20:48,376 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-03-17 20:20:50,223 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:20:50,223 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:20:36 2015-03-17 20:20:50,224 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2015-03-17 20:20:51,600 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:20:51,601 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:20:36 2015-03-17 20:21:56,696 INFO repeater task os75nb_10pings starting 0.8 seconds late 2015-03-17 20:21:56,831 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:22:36,886 INFO repeater task os150nb_10pings starting 1.0 seconds late 2015-03-17 20:22:37,012 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:22:47,024 INFO repeater task os150nb_lastensq starting 1.0 seconds late 2015-03-17 20:22:48,860 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:22:48,860 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:22:36 2015-03-17 20:23:38,932 INFO repeater task os150nb_quick starting 3.0 seconds late 2015-03-17 20:23:43,144 INFO repeater task os150nb_quick ended after 4.2 seconds 2015-03-17 20:23:43,144 INFO repeater task os150nb_3dayplots starting 7.2 seconds late 2015-03-17 20:23:48,036 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:23:48,036 INFO repeater task os150nb_3dayplots ended after 4.9 seconds 2015-03-17 20:23:48,036 INFO repeater task os150nb_tsstats starting 12.1 seconds late 2015-03-17 20:23:48,765 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 20:23:48,765 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-17 20:23:48,765 INFO repeater task os75nb_lastensq starting 2.8 seconds late 2015-03-17 20:23:50,124 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:23:50,125 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:23:36 2015-03-17 20:23:50,125 INFO repeater task os75nb_quick starting 14.2 seconds late 2015-03-17 20:23:53,989 INFO repeater task os75nb_quick ended after 3.9 seconds 2015-03-17 20:23:53,989 INFO repeater task os75nb_3dayplots starting 18.1 seconds late 2015-03-17 20:23:58,874 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:23:58,874 INFO repeater task os75nb_3dayplots ended after 4.9 seconds 2015-03-17 20:23:58,874 INFO repeater task os75nb_tsstats starting 23.0 seconds late 2015-03-17 20:23:59,554 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 20:23:59,554 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 20:23:59,554 INFO repeater task os150nb_beamplots starting 23.7 seconds late 2015-03-17 20:24:00,733 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:24:00,733 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:24:00,733 INFO repeater task os150nb_beamplots starting 24.8 seconds late 2015-03-17 20:24:01,890 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:24:01,890 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:24:01,890 INFO repeater task os75nb_beamplots starting 26.0 seconds late 2015-03-17 20:24:03,052 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:24:03,052 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 20:24:03,052 INFO repeater task os75nb_beamplots starting 27.2 seconds late 2015-03-17 20:24:04,186 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:24:04,186 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 20:24:04,187 INFO repeater task ashtech_hcorrstats starting 28.3 seconds late 2015-03-17 20:24:04,945 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:24:04,946 INFO repeater task phins3_hcorrstats starting 29.0 seconds late 2015-03-17 20:24:05,755 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:24:50,814 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-03-17 20:24:52,591 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:24:52,591 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:24:36 2015-03-17 20:24:57,597 INFO repeater task os150nb_10pings starting 1.7 seconds late 2015-03-17 20:24:57,727 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:25:17,755 INFO repeater task os75nb_10pings starting 1.9 seconds late 2015-03-17 20:25:17,883 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:26:48,003 INFO repeater task os150nb_lastensq starting 2.0 seconds late 2015-03-17 20:26:49,737 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 20:26:49,737 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:26:36 2015-03-17 20:26:49,738 INFO repeater task os75nb_lastensq starting 3.7 seconds late 2015-03-17 20:26:51,118 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:26:51,119 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:26:36 2015-03-17 20:27:16,154 INFO repeater task os150nb_10pings starting 0.3 seconds late 2015-03-17 20:27:16,285 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:28:36,406 INFO repeater task os75nb_10pings starting 0.5 seconds late 2015-03-17 20:28:36,541 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:28:36,542 INFO repeater task ashtech_hcorrstats starting 0.6 seconds late 2015-03-17 20:28:37,338 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:28:37,339 INFO repeater task phins3_hcorrstats starting 1.4 seconds late 2015-03-17 20:28:38,100 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:28:48,112 INFO repeater task os150nb_lastensq starting 2.1 seconds late 2015-03-17 20:28:49,879 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:28:49,879 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:28:36 2015-03-17 20:29:39,948 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-03-17 20:29:40,077 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:29:50,089 INFO repeater task os75nb_lastensq starting 4.1 seconds late 2015-03-17 20:29:51,468 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:29:51,469 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:29:36 2015-03-17 20:30:46,542 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2015-03-17 20:30:48,283 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 20:30:48,283 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:30:36 2015-03-17 20:31:58,380 INFO repeater task os150nb_10pings starting 2.5 seconds late 2015-03-17 20:31:58,503 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:31:58,504 INFO repeater task os75nb_10pings starting 2.6 seconds late 2015-03-17 20:31:58,626 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:32:48,691 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2015-03-17 20:32:50,569 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:32:50,569 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:32:36 2015-03-17 20:32:50,570 INFO repeater task os75nb_lastensq starting 4.6 seconds late 2015-03-17 20:32:52,011 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:32:52,011 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:32:36 2015-03-17 20:33:37,077 INFO repeater task ashtech_hcorrstats starting 1.2 seconds late 2015-03-17 20:33:37,844 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:33:37,844 INFO repeater task phins3_hcorrstats starting 1.9 seconds late 2015-03-17 20:33:38,597 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:34:18,654 INFO repeater task os150nb_10pings starting 2.8 seconds late 2015-03-17 20:34:18,782 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:34:48,825 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2015-03-17 20:34:50,624 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:34:50,624 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:34:36 2015-03-17 20:35:20,664 INFO repeater task os75nb_10pings starting 4.8 seconds late 2015-03-17 20:35:20,797 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:35:50,838 INFO repeater task os75nb_lastensq starting 4.8 seconds late 2015-03-17 20:35:52,244 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:35:52,244 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:35:36 2015-03-17 20:36:37,306 INFO repeater task os150nb_10pings starting 1.4 seconds late 2015-03-17 20:36:37,442 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:36:47,457 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2015-03-17 20:36:49,169 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 20:36:49,170 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:36:36 2015-03-17 20:38:39,324 INFO repeater task os150nb_quick starting 3.4 seconds late 2015-03-17 20:38:43,625 INFO repeater task os150nb_quick ended after 4.3 seconds 2015-03-17 20:38:43,625 INFO repeater task os150nb_3dayplots starting 7.7 seconds late 2015-03-17 20:38:48,584 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:38:48,584 INFO repeater task os150nb_3dayplots ended after 5.0 seconds 2015-03-17 20:38:48,585 INFO repeater task os150nb_tsstats starting 12.7 seconds late 2015-03-17 20:38:49,296 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 20:38:49,296 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-17 20:38:49,297 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2015-03-17 20:38:50,705 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:38:50,705 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:38:36 2015-03-17 20:38:50,705 INFO repeater task os75nb_10pings starting 14.8 seconds late 2015-03-17 20:38:50,830 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:38:50,831 INFO repeater task os75nb_quick starting 14.9 seconds late 2015-03-17 20:38:54,604 INFO repeater task os75nb_quick ended after 3.8 seconds 2015-03-17 20:38:54,604 INFO repeater task os75nb_3dayplots starting 18.7 seconds late 2015-03-17 20:38:59,311 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:38:59,311 INFO repeater task os75nb_3dayplots ended after 4.7 seconds 2015-03-17 20:38:59,311 INFO repeater task os75nb_tsstats starting 23.4 seconds late 2015-03-17 20:38:59,966 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 20:38:59,966 INFO repeater task os150nb_beamplots starting 24.1 seconds late 2015-03-17 20:39:01,117 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:39:01,117 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:39:01,118 INFO repeater task os150nb_beamplots starting 25.2 seconds late 2015-03-17 20:39:02,326 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:39:02,326 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:39:02,327 INFO repeater task os75nb_beamplots starting 26.4 seconds late 2015-03-17 20:39:03,476 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:39:03,476 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 20:39:03,476 INFO repeater task os75nb_beamplots starting 27.6 seconds late 2015-03-17 20:39:04,565 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:39:04,566 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 20:39:04,566 INFO repeater task ashtech_hcorrstats starting 28.7 seconds late 2015-03-17 20:39:05,299 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-03-17 20:39:05,299 INFO repeater task phins3_hcorrstats starting 29.4 seconds late 2015-03-17 20:39:06,037 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-17 20:39:11,042 INFO repeater task os150nb_lastensq starting 25.0 seconds late 2015-03-17 20:39:12,813 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:39:12,814 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:38:36 2015-03-17 20:39:12,814 INFO repeater task os150nb_10pings starting 16.9 seconds late 2015-03-17 20:39:12,943 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:40:48,076 INFO repeater task os150nb_lastensq starting 2.1 seconds late 2015-03-17 20:40:49,967 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:40:49,967 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:40:36 2015-03-17 20:41:20,006 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-03-17 20:41:20,135 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:41:50,176 INFO repeater task os75nb_lastensq starting 4.2 seconds late 2015-03-17 20:41:51,631 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-17 20:41:51,632 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:41:36 2015-03-17 20:41:56,639 INFO repeater task os75nb_10pings starting 0.7 seconds late 2015-03-17 20:41:56,770 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:42:46,836 INFO repeater task os150nb_lastensq starting 0.8 seconds late 2015-03-17 20:42:48,707 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:42:48,708 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:42:36 2015-03-17 20:43:38,770 INFO repeater task os150nb_10pings starting 2.9 seconds late 2015-03-17 20:43:38,901 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:43:38,904 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2015-03-17 20:43:39,699 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:43:39,700 INFO repeater task phins3_hcorrstats starting 3.8 seconds late 2015-03-17 20:43:40,491 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:44:50,588 INFO repeater task os150nb_lastensq starting 4.6 seconds late 2015-03-17 20:44:52,457 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:44:52,457 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:44:36 2015-03-17 20:44:52,458 INFO repeater task os75nb_lastensq starting 6.5 seconds late 2015-03-17 20:44:53,814 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:44:53,814 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:44:36 2015-03-17 20:45:18,846 INFO repeater task os75nb_10pings starting 2.9 seconds late 2015-03-17 20:45:18,975 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:45:59,032 INFO repeater task os150nb_10pings starting 3.1 seconds late 2015-03-17 20:45:59,156 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:46:49,223 INFO repeater task os150nb_lastensq starting 3.2 seconds late 2015-03-17 20:46:50,956 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-03-17 20:46:50,956 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:46:36 2015-03-17 20:47:46,013 INFO repeater task os75nb_lastensq starting 0.0 seconds late 2015-03-17 20:47:47,405 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:47:47,406 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:47:36 2015-03-17 20:48:17,448 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-03-17 20:48:17,571 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:48:37,602 INFO repeater task os75nb_10pings starting 1.7 seconds late 2015-03-17 20:48:37,724 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:48:37,725 INFO repeater task ashtech_hcorrstats starting 1.8 seconds late 2015-03-17 20:48:38,498 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:48:38,498 INFO repeater task phins3_hcorrstats starting 2.6 seconds late 2015-03-17 20:48:39,274 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 20:48:49,287 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2015-03-17 20:48:51,229 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:48:51,229 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:48:36 2015-03-17 20:50:36,375 INFO repeater task os150nb_10pings starting 0.5 seconds late 2015-03-17 20:50:36,507 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:50:46,520 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2015-03-17 20:50:48,322 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:50:48,322 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:50:36 2015-03-17 20:50:48,323 INFO repeater task os75nb_lastensq starting 2.3 seconds late 2015-03-17 20:50:49,709 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:50:49,709 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:50:36 2015-03-17 20:51:59,804 INFO repeater task os75nb_10pings starting 3.9 seconds late 2015-03-17 20:51:59,933 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:52:49,995 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2015-03-17 20:52:51,868 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:52:51,868 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:52:36 2015-03-17 20:52:56,876 INFO repeater task os150nb_10pings starting 1.0 seconds late 2015-03-17 20:52:57,004 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:53:37,064 INFO repeater task os150nb_quick starting 1.2 seconds late 2015-03-17 20:53:41,147 INFO repeater task os150nb_quick ended after 4.1 seconds 2015-03-17 20:53:41,147 INFO repeater task os150nb_3dayplots starting 5.2 seconds late 2015-03-17 20:53:46,082 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:53:46,082 INFO repeater task os150nb_3dayplots ended after 4.9 seconds 2015-03-17 20:53:46,083 INFO repeater task os150nb_tsstats starting 10.2 seconds late 2015-03-17 20:53:46,839 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 20:53:46,839 INFO repeater task os150nb_tsstats ended after 0.8 seconds 2015-03-17 20:53:46,840 INFO repeater task os75nb_lastensq starting 0.8 seconds late 2015-03-17 20:53:48,221 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:53:48,221 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:53:36 2015-03-17 20:53:48,222 INFO repeater task os75nb_quick starting 12.3 seconds late 2015-03-17 20:53:52,071 INFO repeater task os75nb_quick ended after 3.8 seconds 2015-03-17 20:53:52,072 INFO repeater task os75nb_3dayplots starting 16.2 seconds late 2015-03-17 20:53:57,000 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 20:53:57,000 INFO repeater task os75nb_3dayplots ended after 4.9 seconds 2015-03-17 20:53:57,001 INFO repeater task os75nb_tsstats starting 21.1 seconds late 2015-03-17 20:53:57,715 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 20:53:57,715 INFO repeater task os150nb_beamplots starting 21.8 seconds late 2015-03-17 20:53:58,900 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:53:58,900 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:53:58,900 INFO repeater task os150nb_beamplots starting 23.0 seconds late 2015-03-17 20:54:00,110 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:54:00,111 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 20:54:00,111 INFO repeater task os75nb_beamplots starting 24.2 seconds late 2015-03-17 20:54:01,281 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:54:01,282 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 20:54:01,282 INFO repeater task os75nb_beamplots starting 25.4 seconds late 2015-03-17 20:54:02,417 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 20:54:02,417 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 20:54:02,418 INFO repeater task ashtech_hcorrstats starting 26.5 seconds late 2015-03-17 20:54:03,166 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-03-17 20:54:03,167 INFO repeater task phins3_hcorrstats starting 27.3 seconds late 2015-03-17 20:54:03,905 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-17 20:54:48,966 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-03-17 20:54:50,734 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:54:50,734 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:54:36 2015-03-17 20:55:20,778 INFO repeater task os150nb_10pings starting 4.9 seconds late 2015-03-17 20:55:20,900 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:55:20,901 INFO repeater task os75nb_10pings starting 5.0 seconds late 2015-03-17 20:55:21,028 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:56:46,147 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-03-17 20:56:47,908 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 20:56:47,909 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:56:36 2015-03-17 20:56:47,909 INFO repeater task os75nb_lastensq starting 1.9 seconds late 2015-03-17 20:56:49,291 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:56:49,291 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:56:36 2015-03-17 20:57:39,350 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-03-17 20:57:39,475 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 20:58:39,560 INFO repeater task os75nb_10pings starting 3.7 seconds late 2015-03-17 20:58:39,693 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 20:58:39,695 INFO repeater task ashtech_hcorrstats starting 3.8 seconds late 2015-03-17 20:58:40,476 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 20:58:40,477 INFO repeater task phins3_hcorrstats starting 4.6 seconds late 2015-03-17 20:58:41,225 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-17 20:58:46,230 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-03-17 20:58:48,121 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 20:58:48,122 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 20:58:36 2015-03-17 20:59:48,197 INFO repeater task os75nb_lastensq starting 2.2 seconds late 2015-03-17 20:59:49,604 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 20:59:49,605 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 20:59:36 2015-03-17 20:59:59,619 INFO repeater task os150nb_10pings starting 3.7 seconds late 2015-03-17 20:59:59,753 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:00:49,820 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2015-03-17 21:00:51,736 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 21:00:51,736 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:00:36 2015-03-17 21:01:56,823 INFO repeater task os75nb_10pings starting 0.9 seconds late 2015-03-17 21:01:56,951 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:02:16,978 INFO repeater task os150nb_10pings starting 1.1 seconds late 2015-03-17 21:02:17,108 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:02:47,147 INFO repeater task os150nb_lastensq starting 1.1 seconds late 2015-03-17 21:02:48,990 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:02:48,990 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:02:36 2015-03-17 21:02:48,991 INFO repeater task os75nb_lastensq starting 3.0 seconds late 2015-03-17 21:02:50,468 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-17 21:02:50,469 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:02:36 2015-03-17 21:03:40,538 INFO repeater task ashtech_hcorrstats starting 4.6 seconds late 2015-03-17 21:03:41,360 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 21:03:41,360 INFO repeater task phins3_hcorrstats starting 5.5 seconds late 2015-03-17 21:03:42,115 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 21:04:37,184 INFO repeater task os150nb_10pings starting 1.3 seconds late 2015-03-17 21:04:37,308 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:04:47,322 INFO repeater task os150nb_lastensq starting 1.3 seconds late 2015-03-17 21:04:49,110 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:04:49,111 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:04:36 2015-03-17 21:05:19,157 INFO repeater task os75nb_10pings starting 3.3 seconds late 2015-03-17 21:05:19,287 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:05:49,326 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2015-03-17 21:05:50,771 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 21:05:50,771 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:05:36 2015-03-17 21:06:50,854 INFO repeater task os150nb_lastensq starting 4.9 seconds late 2015-03-17 21:06:52,704 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 21:06:52,704 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:06:36 2015-03-17 21:06:57,712 INFO repeater task os150nb_10pings starting 1.8 seconds late 2015-03-17 21:06:57,834 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:08:37,970 INFO repeater task os150nb_quick starting 2.1 seconds late 2015-03-17 21:08:42,325 INFO repeater task os150nb_quick ended after 4.4 seconds 2015-03-17 21:08:42,325 INFO repeater task os150nb_3dayplots starting 6.4 seconds late 2015-03-17 21:08:47,402 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 21:08:47,402 INFO repeater task os150nb_3dayplots ended after 5.1 seconds 2015-03-17 21:08:47,403 INFO repeater task os150nb_tsstats starting 11.5 seconds late 2015-03-17 21:08:48,115 INFO repeater task os150nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 21:08:48,115 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-03-17 21:08:48,116 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2015-03-17 21:08:49,561 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 21:08:49,562 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:08:36 2015-03-17 21:08:49,562 INFO repeater task os75nb_10pings starting 13.7 seconds late 2015-03-17 21:08:49,686 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:08:49,686 INFO repeater task os75nb_quick starting 13.8 seconds late 2015-03-17 21:08:53,492 INFO repeater task os75nb_quick ended after 3.8 seconds 2015-03-17 21:08:53,493 INFO repeater task os75nb_3dayplots starting 17.6 seconds late 2015-03-17 21:08:58,634 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 21:08:58,634 INFO repeater task os75nb_3dayplots ended after 5.1 seconds 2015-03-17 21:08:58,634 INFO repeater task os75nb_tsstats starting 22.7 seconds late 2015-03-17 21:08:59,320 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-03-17 21:08:59,320 INFO repeater task os75nb_tsstats ended after 0.7 seconds 2015-03-17 21:08:59,320 INFO repeater task os150nb_beamplots starting 23.4 seconds late 2015-03-17 21:09:00,492 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 21:09:00,493 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 21:09:00,493 INFO repeater task os150nb_beamplots starting 24.6 seconds late 2015-03-17 21:09:01,732 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 21:09:01,732 INFO repeater task os150nb_beamplots ended after 1.2 seconds 2015-03-17 21:09:01,732 INFO repeater task os75nb_beamplots starting 25.8 seconds late 2015-03-17 21:09:02,885 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 21:09:02,885 INFO repeater task os75nb_beamplots ended after 1.2 seconds 2015-03-17 21:09:02,886 INFO repeater task os75nb_beamplots starting 27.0 seconds late 2015-03-17 21:09:03,989 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-03-17 21:09:03,989 INFO repeater task os75nb_beamplots ended after 1.1 seconds 2015-03-17 21:09:03,989 INFO repeater task ashtech_hcorrstats starting 28.1 seconds late 2015-03-17 21:09:04,727 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-03-17 21:09:04,727 INFO repeater task phins3_hcorrstats starting 28.8 seconds late 2015-03-17 21:09:05,474 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-03-17 21:09:10,480 INFO repeater task os150nb_lastensq starting 24.5 seconds late 2015-03-17 21:09:12,276 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:09:12,276 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:08:36 2015-03-17 21:09:17,284 INFO repeater task os150nb_10pings starting 1.4 seconds late 2015-03-17 21:09:17,409 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:10:47,535 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2015-03-17 21:10:49,349 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:10:49,350 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:10:36 2015-03-17 21:11:39,417 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-03-17 21:11:39,557 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:11:49,566 INFO repeater task os75nb_lastensq starting 3.6 seconds late 2015-03-17 21:11:50,950 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 21:11:50,951 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:11:36 2015-03-17 21:11:55,958 INFO repeater task os75nb_10pings starting 0.1 seconds late 2015-03-17 21:11:56,082 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:12:46,149 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-03-17 21:12:47,943 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:12:47,944 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:12:36 2015-03-17 21:13:38,014 INFO repeater task ashtech_hcorrstats starting 2.1 seconds late 2015-03-17 21:13:38,789 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 21:13:38,790 INFO repeater task phins3_hcorrstats starting 2.9 seconds late 2015-03-17 21:13:39,598 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 21:13:59,622 INFO repeater task os150nb_10pings starting 3.7 seconds late 2015-03-17 21:13:59,751 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:14:49,823 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2015-03-17 21:14:51,618 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:14:51,618 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:14:36 2015-03-17 21:14:51,619 INFO repeater task os75nb_lastensq starting 5.6 seconds late 2015-03-17 21:14:53,094 INFO repeater task os75nb_lastensq ended after 1.5 seconds 2015-03-17 21:14:53,094 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:14:36 2015-03-17 21:15:18,127 INFO repeater task os75nb_10pings starting 2.2 seconds late 2015-03-17 21:15:18,260 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:16:18,348 INFO repeater task os150nb_10pings starting 2.5 seconds late 2015-03-17 21:16:18,482 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:16:48,523 INFO repeater task os150nb_lastensq starting 2.5 seconds late 2015-03-17 21:16:50,402 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 21:16:50,402 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:16:36 2015-03-17 21:17:50,485 INFO repeater task os75nb_lastensq starting 4.5 seconds late 2015-03-17 21:17:51,873 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 21:17:51,873 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:17:36 2015-03-17 21:18:36,930 INFO repeater task os150nb_10pings starting 1.0 seconds late 2015-03-17 21:18:37,061 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:18:37,062 INFO repeater task os75nb_10pings starting 1.2 seconds late 2015-03-17 21:18:37,190 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:18:37,192 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2015-03-17 21:18:37,985 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-03-17 21:18:37,986 INFO repeater task phins3_hcorrstats starting 2.1 seconds late 2015-03-17 21:18:38,803 INFO repeater task phins3_hcorrstats ended after 0.8 seconds 2015-03-17 21:18:48,814 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2015-03-17 21:18:50,592 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:18:50,593 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:18:36 2015-03-17 21:20:50,762 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-03-17 21:20:52,576 INFO repeater task os150nb_lastensq ended after 1.8 seconds 2015-03-17 21:20:52,576 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:20:36 2015-03-17 21:20:52,577 INFO repeater task os75nb_lastensq starting 6.6 seconds late 2015-03-17 21:20:54,005 INFO repeater task os75nb_lastensq ended after 1.4 seconds 2015-03-17 21:20:54,005 INFO repeater task os75nb_lastensq: Time from /home/data/RR1504/proc/os75nb/load/wrote_ens_time: 2015/03/17 21:20:36 2015-03-17 21:20:59,012 INFO repeater task os150nb_10pings starting 3.1 seconds late 2015-03-17 21:20:59,149 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:21:59,234 INFO repeater task os75nb_10pings starting 3.3 seconds late 2015-03-17 21:21:59,363 INFO repeater task os75nb_10pings ended after 0.1 seconds 2015-03-17 21:22:49,424 INFO repeater task os150nb_lastensq starting 3.4 seconds late 2015-03-17 21:22:51,342 INFO repeater task os150nb_lastensq ended after 1.9 seconds 2015-03-17 21:22:51,343 INFO repeater task os150nb_lastensq: Time from /home/data/RR1504/proc/os150nb/load/wrote_ens_time: 2015/03/17 21:22:36 2015-03-17 21:23:16,375 INFO repeater task os150nb_10pings starting 0.5 seconds late 2015-03-17 21:23:16,506 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-03-17 21:23:36,536 INFO repeater task os150nb_quick starting 0.6 seconds late 2015-03-17 21:23:40,888 INFO repeater task os150nb_quick ended after 4.4 seconds 2015-03-17 21:23:40,889 INFO repeater task os150nb_3dayplots starting 5.0 seconds late 2015-03-17 21:23:45,878 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-03-17 21:23:45,879 INFO repeater task os150nb_3dayplots ended after 5.0 seconds 2015-03-17 21:23:45,879 INFO repeater task os150nb_tsstats starting 10.0 seconds late