2015-06-29 19:36:40,045 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:36:27 2015-06-29 19:37:10,100 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-06-29 19:37:10,222 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:38:40,375 INFO repeater task os150nb_lastensq starting 3.4 seconds late 2015-06-29 19:38:41,927 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:38:41,927 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:38:27 2015-06-29 19:39:27,000 INFO repeater task os150nb_10pings starting 0.4 seconds late 2015-06-29 19:39:27,127 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:40:37,238 INFO repeater task os150nb_lastensq starting 0.2 seconds late 2015-06-29 19:40:38,880 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:40:38,880 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:40:27 2015-06-29 19:41:28,967 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2015-06-29 19:41:29,671 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 19:41:29,671 INFO repeater task phins3_hcorrstats starting 3.1 seconds late 2015-06-29 19:41:30,331 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 19:41:50,354 INFO repeater task os150nb_10pings starting 3.8 seconds late 2015-06-29 19:41:50,475 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:42:40,568 INFO repeater task os150nb_lastensq starting 3.6 seconds late 2015-06-29 19:42:42,146 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:42:42,146 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:42:27 2015-06-29 19:44:07,307 INFO repeater task os150nb_10pings starting 0.7 seconds late 2015-06-29 19:44:07,431 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:44:37,480 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2015-06-29 19:44:39,041 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:44:39,041 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:44:27 2015-06-29 19:46:29,242 INFO repeater task os150nb_10pings starting 2.7 seconds late 2015-06-29 19:46:29,361 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:46:29,362 INFO repeater task os150nb_quick starting 2.8 seconds late 2015-06-29 19:46:34,134 INFO repeater task os150nb_quick ended after 4.8 seconds 2015-06-29 19:46:34,134 INFO repeater task os150nb_3dayplots starting 7.6 seconds late 2015-06-29 19:46:40,297 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 19:46:40,297 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 19:46:40,297 INFO repeater task os150nb_tsstats starting 13.7 seconds late 2015-06-29 19:46:41,032 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 19:46:41,033 INFO repeater task os150nb_beamplots starting 14.5 seconds late 2015-06-29 19:46:42,054 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 19:46:42,054 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 19:46:42,055 INFO repeater task os150nb_beamplots starting 15.5 seconds late 2015-06-29 19:46:43,096 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 19:46:43,096 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 19:46:43,097 INFO repeater task ashtech_hcorrstats starting 16.5 seconds late 2015-06-29 19:46:43,802 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 19:46:43,802 INFO repeater task phins3_hcorrstats starting 17.2 seconds late 2015-06-29 19:46:44,474 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 19:46:49,480 INFO repeater task os150nb_lastensq starting 12.5 seconds late 2015-06-29 19:46:51,100 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:46:51,101 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:46:27 2015-06-29 19:48:41,296 INFO repeater task os150nb_lastensq starting 4.3 seconds late 2015-06-29 19:48:42,901 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:48:42,901 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:48:27 2015-06-29 19:48:47,908 INFO repeater task os150nb_10pings starting 1.4 seconds late 2015-06-29 19:48:48,033 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:50:38,232 INFO repeater task os150nb_lastensq starting 1.2 seconds late 2015-06-29 19:50:39,800 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:50:39,800 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:50:27 2015-06-29 19:51:09,852 INFO repeater task os150nb_10pings starting 3.3 seconds late 2015-06-29 19:51:09,971 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:51:30,006 INFO repeater task ashtech_hcorrstats starting 3.4 seconds late 2015-06-29 19:51:30,724 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 19:51:30,724 INFO repeater task phins3_hcorrstats starting 4.2 seconds late 2015-06-29 19:51:31,384 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 19:52:41,501 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2015-06-29 19:52:43,078 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:52:43,078 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:52:27 2015-06-29 19:53:28,158 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-06-29 19:53:28,282 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:54:38,406 INFO repeater task os150nb_lastensq starting 1.4 seconds late 2015-06-29 19:54:39,996 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:54:39,996 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:54:27 2015-06-29 19:55:50,118 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-06-29 19:55:50,234 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:56:30,310 INFO repeater task ashtech_hcorrstats starting 3.7 seconds late 2015-06-29 19:56:31,061 INFO repeater task ashtech_hcorrstats ended after 0.8 seconds 2015-06-29 19:56:31,061 INFO repeater task phins3_hcorrstats starting 4.5 seconds late 2015-06-29 19:56:31,749 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 19:56:41,757 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-06-29 19:56:43,385 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:56:43,385 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:56:27 2015-06-29 19:58:08,538 INFO repeater task os150nb_10pings starting 2.0 seconds late 2015-06-29 19:58:08,653 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 19:58:38,702 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-06-29 19:58:40,269 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 19:58:40,269 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 19:58:27 2015-06-29 20:00:30,465 INFO repeater task os150nb_10pings starting 3.9 seconds late 2015-06-29 20:00:30,585 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:00:40,600 INFO repeater task os150nb_lastensq starting 3.6 seconds late 2015-06-29 20:00:42,180 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:00:42,181 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:00:27 2015-06-29 20:01:27,254 INFO repeater task os150nb_quick starting 0.7 seconds late 2015-06-29 20:01:32,108 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 20:01:32,108 INFO repeater task os150nb_3dayplots starting 5.6 seconds late 2015-06-29 20:01:38,329 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 20:01:38,329 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 20:01:38,329 INFO repeater task os150nb_tsstats starting 11.8 seconds late 2015-06-29 20:01:39,064 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 20:01:39,065 INFO repeater task os150nb_beamplots starting 12.5 seconds late 2015-06-29 20:01:40,140 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:01:40,140 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 20:01:40,141 INFO repeater task os150nb_beamplots starting 13.6 seconds late 2015-06-29 20:01:41,179 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:01:41,179 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 20:01:41,179 INFO repeater task ashtech_hcorrstats starting 14.6 seconds late 2015-06-29 20:01:41,907 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:01:41,907 INFO repeater task phins3_hcorrstats starting 15.3 seconds late 2015-06-29 20:01:42,586 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:02:37,685 INFO repeater task os150nb_lastensq starting 0.7 seconds late 2015-06-29 20:02:39,269 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:02:39,270 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:02:27 2015-06-29 20:02:49,287 INFO repeater task os150nb_10pings starting 2.7 seconds late 2015-06-29 20:02:49,410 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:04:39,609 INFO repeater task os150nb_lastensq starting 2.6 seconds late 2015-06-29 20:04:41,179 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:04:41,179 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:04:27 2015-06-29 20:05:11,234 INFO repeater task os150nb_10pings starting 4.7 seconds late 2015-06-29 20:05:11,353 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:06:31,499 INFO repeater task ashtech_hcorrstats starting 4.9 seconds late 2015-06-29 20:06:32,234 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:06:32,235 INFO repeater task phins3_hcorrstats starting 5.7 seconds late 2015-06-29 20:06:32,919 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:06:37,924 INFO repeater task os150nb_lastensq starting 0.9 seconds late 2015-06-29 20:06:39,496 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:06:39,496 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:06:27 2015-06-29 20:07:29,587 INFO repeater task os150nb_10pings starting 3.0 seconds late 2015-06-29 20:07:29,711 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:08:39,840 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2015-06-29 20:08:41,438 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:08:41,438 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:08:27 2015-06-29 20:09:51,564 INFO repeater task os150nb_10pings starting 5.0 seconds late 2015-06-29 20:09:51,684 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:10:41,766 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-06-29 20:10:43,320 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:10:43,321 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:10:27 2015-06-29 20:11:28,395 INFO repeater task ashtech_hcorrstats starting 1.8 seconds late 2015-06-29 20:11:29,108 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:11:29,109 INFO repeater task phins3_hcorrstats starting 2.5 seconds late 2015-06-29 20:11:29,775 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:12:09,846 INFO repeater task os150nb_10pings starting 3.3 seconds late 2015-06-29 20:12:09,960 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:12:40,008 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-06-29 20:12:41,572 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:12:41,572 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:12:27 2015-06-29 20:14:26,757 INFO repeater task os150nb_10pings starting 0.2 seconds late 2015-06-29 20:14:26,882 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:14:41,907 INFO repeater task os150nb_lastensq starting 4.9 seconds late 2015-06-29 20:14:43,498 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:14:43,498 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:14:27 2015-06-29 20:16:28,684 INFO repeater task os150nb_quick starting 2.1 seconds late 2015-06-29 20:16:33,486 INFO repeater task os150nb_quick ended after 4.8 seconds 2015-06-29 20:16:33,486 INFO repeater task os150nb_3dayplots starting 6.9 seconds late 2015-06-29 20:16:39,659 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 20:16:39,660 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 20:16:39,660 INFO repeater task os150nb_tsstats starting 13.1 seconds late 2015-06-29 20:16:40,369 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 20:16:40,369 INFO repeater task os150nb_beamplots starting 13.8 seconds late 2015-06-29 20:16:41,386 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:16:41,387 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 20:16:41,387 INFO repeater task os150nb_beamplots starting 14.8 seconds late 2015-06-29 20:16:42,414 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:16:42,414 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 20:16:42,414 INFO repeater task ashtech_hcorrstats starting 15.9 seconds late 2015-06-29 20:16:43,123 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:16:43,124 INFO repeater task phins3_hcorrstats starting 16.6 seconds late 2015-06-29 20:16:43,789 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:16:48,795 INFO repeater task os150nb_lastensq starting 11.8 seconds late 2015-06-29 20:16:50,343 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 20:16:50,344 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:16:27 2015-06-29 20:16:50,344 INFO repeater task os150nb_10pings starting 3.8 seconds late 2015-06-29 20:16:50,460 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:18:40,653 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-06-29 20:18:42,188 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 20:18:42,188 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:18:27 2015-06-29 20:19:07,222 INFO repeater task os150nb_10pings starting 0.7 seconds late 2015-06-29 20:19:07,344 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:20:37,502 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2015-06-29 20:20:39,042 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 20:20:39,042 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:20:27 2015-06-29 20:21:29,122 INFO repeater task os150nb_10pings starting 2.6 seconds late 2015-06-29 20:21:29,246 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:21:29,247 INFO repeater task ashtech_hcorrstats starting 2.7 seconds late 2015-06-29 20:21:29,991 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:21:29,992 INFO repeater task phins3_hcorrstats starting 3.4 seconds late 2015-06-29 20:21:30,655 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:22:40,783 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2015-06-29 20:22:42,400 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:22:42,401 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:22:27 2015-06-29 20:23:47,515 INFO repeater task os150nb_10pings starting 1.0 seconds late 2015-06-29 20:23:47,643 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:24:37,736 INFO repeater task os150nb_lastensq starting 0.7 seconds late 2015-06-29 20:24:39,288 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:24:39,288 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:24:27 2015-06-29 20:26:09,445 INFO repeater task os150nb_10pings starting 2.9 seconds late 2015-06-29 20:26:09,566 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:26:29,605 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2015-06-29 20:26:30,343 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:26:30,343 INFO repeater task phins3_hcorrstats starting 3.8 seconds late 2015-06-29 20:26:31,018 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:26:41,034 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2015-06-29 20:26:42,591 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:26:42,591 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:26:27 2015-06-29 20:28:27,783 INFO repeater task os150nb_10pings starting 1.2 seconds late 2015-06-29 20:28:27,898 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:28:37,914 INFO repeater task os150nb_lastensq starting 0.9 seconds late 2015-06-29 20:28:39,521 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:28:39,522 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:28:27 2015-06-29 20:30:39,736 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2015-06-29 20:30:41,319 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:30:41,320 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:30:27 2015-06-29 20:30:51,336 INFO repeater task os150nb_10pings starting 4.8 seconds late 2015-06-29 20:30:51,457 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:31:31,528 INFO repeater task os150nb_quick starting 5.0 seconds late 2015-06-29 20:31:36,368 INFO repeater task os150nb_quick ended after 4.8 seconds 2015-06-29 20:31:36,368 INFO repeater task os150nb_3dayplots starting 9.8 seconds late 2015-06-29 20:31:42,530 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 20:31:42,530 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 20:31:42,531 INFO repeater task os150nb_tsstats starting 16.0 seconds late 2015-06-29 20:31:43,239 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 20:31:43,240 INFO repeater task os150nb_beamplots starting 16.7 seconds late 2015-06-29 20:31:44,244 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:31:44,244 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 20:31:44,245 INFO repeater task os150nb_beamplots starting 17.7 seconds late 2015-06-29 20:31:45,262 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:31:45,262 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 20:31:45,263 INFO repeater task ashtech_hcorrstats starting 18.7 seconds late 2015-06-29 20:31:45,963 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:31:45,964 INFO repeater task phins3_hcorrstats starting 19.4 seconds late 2015-06-29 20:31:46,624 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:32:41,723 INFO repeater task os150nb_lastensq starting 4.7 seconds late 2015-06-29 20:32:43,275 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:32:43,275 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:32:27 2015-06-29 20:33:08,321 INFO repeater task os150nb_10pings starting 1.8 seconds late 2015-06-29 20:33:08,445 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:34:38,589 INFO repeater task os150nb_lastensq starting 1.6 seconds late 2015-06-29 20:34:40,149 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:34:40,149 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:34:27 2015-06-29 20:35:30,238 INFO repeater task os150nb_10pings starting 3.7 seconds late 2015-06-29 20:35:30,360 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:36:30,474 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2015-06-29 20:36:31,187 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:36:31,187 INFO repeater task phins3_hcorrstats starting 4.6 seconds late 2015-06-29 20:36:31,863 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:36:41,876 INFO repeater task os150nb_lastensq starting 4.9 seconds late 2015-06-29 20:36:43,486 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:36:43,486 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:36:27 2015-06-29 20:37:48,604 INFO repeater task os150nb_10pings starting 2.0 seconds late 2015-06-29 20:37:48,728 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:38:38,821 INFO repeater task os150nb_lastensq starting 1.8 seconds late 2015-06-29 20:38:40,379 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:38:40,379 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:38:27 2015-06-29 20:40:10,544 INFO repeater task os150nb_10pings starting 4.0 seconds late 2015-06-29 20:40:10,665 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:40:40,719 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-06-29 20:40:42,303 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:40:42,303 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:40:27 2015-06-29 20:41:27,385 INFO repeater task ashtech_hcorrstats starting 0.8 seconds late 2015-06-29 20:41:28,100 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:41:28,100 INFO repeater task phins3_hcorrstats starting 1.5 seconds late 2015-06-29 20:41:28,782 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:42:28,884 INFO repeater task os150nb_10pings starting 2.3 seconds late 2015-06-29 20:42:29,008 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:42:39,022 INFO repeater task os150nb_lastensq starting 2.0 seconds late 2015-06-29 20:42:40,567 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 20:42:40,567 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:42:27 2015-06-29 20:44:40,730 INFO repeater task os150nb_lastensq starting 3.7 seconds late 2015-06-29 20:44:42,309 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:44:42,310 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:44:27 2015-06-29 20:44:47,317 INFO repeater task os150nb_10pings starting 0.8 seconds late 2015-06-29 20:44:47,439 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:46:27,607 INFO repeater task os150nb_quick starting 1.0 seconds late 2015-06-29 20:46:32,492 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 20:46:32,493 INFO repeater task os150nb_3dayplots starting 5.9 seconds late 2015-06-29 20:46:38,820 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 20:46:38,820 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2015-06-29 20:46:38,821 INFO repeater task os150nb_tsstats starting 12.3 seconds late 2015-06-29 20:46:39,570 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 20:46:39,570 INFO repeater task os150nb_beamplots starting 13.0 seconds late 2015-06-29 20:46:40,647 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:46:40,648 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 20:46:40,648 INFO repeater task os150nb_beamplots starting 14.1 seconds late 2015-06-29 20:46:41,729 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 20:46:41,729 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 20:46:41,729 INFO repeater task ashtech_hcorrstats starting 15.2 seconds late 2015-06-29 20:46:42,447 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:46:42,448 INFO repeater task phins3_hcorrstats starting 15.9 seconds late 2015-06-29 20:46:43,158 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:46:48,164 INFO repeater task os150nb_lastensq starting 11.2 seconds late 2015-06-29 20:46:49,726 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:46:49,727 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:46:27 2015-06-29 20:47:09,756 INFO repeater task os150nb_10pings starting 3.2 seconds late 2015-06-29 20:47:09,877 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:48:40,029 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-06-29 20:48:41,578 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 20:48:41,578 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:48:27 2015-06-29 20:49:26,661 INFO repeater task os150nb_10pings starting 0.1 seconds late 2015-06-29 20:49:26,779 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:50:41,913 INFO repeater task os150nb_lastensq starting 4.9 seconds late 2015-06-29 20:50:43,478 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:50:43,478 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:50:27 2015-06-29 20:51:28,560 INFO repeater task ashtech_hcorrstats starting 2.0 seconds late 2015-06-29 20:51:29,257 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:51:29,257 INFO repeater task phins3_hcorrstats starting 2.7 seconds late 2015-06-29 20:51:29,922 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:51:49,957 INFO repeater task os150nb_10pings starting 3.4 seconds late 2015-06-29 20:51:50,080 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:52:40,166 INFO repeater task os150nb_lastensq starting 3.2 seconds late 2015-06-29 20:52:41,745 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:52:41,745 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:52:27 2015-06-29 20:54:06,898 INFO repeater task os150nb_10pings starting 0.3 seconds late 2015-06-29 20:54:07,024 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:54:37,077 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-06-29 20:54:38,627 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:54:38,628 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:54:27 2015-06-29 20:56:28,824 INFO repeater task os150nb_10pings starting 2.3 seconds late 2015-06-29 20:56:28,943 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 20:56:28,944 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2015-06-29 20:56:29,641 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 20:56:29,641 INFO repeater task phins3_hcorrstats starting 3.1 seconds late 2015-06-29 20:56:30,311 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 20:56:40,323 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2015-06-29 20:56:41,925 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 20:56:41,925 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:56:27 2015-06-29 20:58:37,138 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2015-06-29 20:58:38,684 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 20:58:38,684 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 20:58:27 2015-06-29 20:58:48,696 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-06-29 20:58:48,818 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:00:39,022 INFO repeater task os150nb_lastensq starting 2.0 seconds late 2015-06-29 21:00:40,606 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:00:40,606 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:00:27 2015-06-29 21:01:10,660 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-06-29 21:01:10,776 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:01:30,806 INFO repeater task os150nb_quick starting 4.2 seconds late 2015-06-29 21:01:35,689 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 21:01:35,690 INFO repeater task os150nb_3dayplots starting 9.1 seconds late 2015-06-29 21:01:41,884 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 21:01:41,884 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 21:01:41,884 INFO repeater task os150nb_tsstats starting 15.3 seconds late 2015-06-29 21:01:42,604 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 21:01:42,604 INFO repeater task os150nb_beamplots starting 16.0 seconds late 2015-06-29 21:01:43,623 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:01:43,623 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 21:01:43,623 INFO repeater task os150nb_beamplots starting 17.1 seconds late 2015-06-29 21:01:44,659 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:01:44,660 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 21:01:44,660 INFO repeater task ashtech_hcorrstats starting 18.1 seconds late 2015-06-29 21:01:45,376 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:01:45,376 INFO repeater task phins3_hcorrstats starting 18.8 seconds late 2015-06-29 21:01:46,060 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:02:41,161 INFO repeater task os150nb_lastensq starting 4.2 seconds late 2015-06-29 21:02:42,753 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:02:42,753 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:02:27 2015-06-29 21:03:27,837 INFO repeater task os150nb_10pings starting 1.3 seconds late 2015-06-29 21:03:27,963 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:04:38,088 INFO repeater task os150nb_lastensq starting 1.1 seconds late 2015-06-29 21:04:39,671 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:04:39,671 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:04:27 2015-06-29 21:05:49,798 INFO repeater task os150nb_10pings starting 3.2 seconds late 2015-06-29 21:05:49,929 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:06:30,002 INFO repeater task ashtech_hcorrstats starting 3.4 seconds late 2015-06-29 21:06:30,709 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:06:30,709 INFO repeater task phins3_hcorrstats starting 4.1 seconds late 2015-06-29 21:06:31,362 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:06:41,377 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2015-06-29 21:06:42,978 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:06:42,979 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:06:27 2015-06-29 21:08:08,130 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-06-29 21:08:08,253 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:08:38,308 INFO repeater task os150nb_lastensq starting 1.3 seconds late 2015-06-29 21:08:39,922 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:08:39,923 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:08:27 2015-06-29 21:10:30,119 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-06-29 21:10:30,237 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:10:40,253 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2015-06-29 21:10:41,858 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:10:41,859 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:10:27 2015-06-29 21:11:26,941 INFO repeater task ashtech_hcorrstats starting 0.4 seconds late 2015-06-29 21:11:27,646 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:11:27,647 INFO repeater task phins3_hcorrstats starting 1.1 seconds late 2015-06-29 21:11:28,309 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:12:38,429 INFO repeater task os150nb_lastensq starting 1.4 seconds late 2015-06-29 21:12:40,139 INFO repeater task os150nb_lastensq ended after 1.7 seconds 2015-06-29 21:12:40,140 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:12:27 2015-06-29 21:12:50,156 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-06-29 21:12:50,280 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:14:40,478 INFO repeater task os150nb_lastensq starting 3.5 seconds late 2015-06-29 21:14:42,063 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:14:42,063 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:14:27 2015-06-29 21:15:07,101 INFO repeater task os150nb_10pings starting 0.5 seconds late 2015-06-29 21:15:07,221 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:16:27,363 INFO repeater task os150nb_quick starting 0.8 seconds late 2015-06-29 21:16:32,241 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 21:16:32,241 INFO repeater task os150nb_3dayplots starting 5.7 seconds late 2015-06-29 21:16:38,567 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 21:16:38,567 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2015-06-29 21:16:38,568 INFO repeater task os150nb_tsstats starting 12.0 seconds late 2015-06-29 21:16:39,285 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 21:16:39,285 INFO repeater task os150nb_beamplots starting 12.7 seconds late 2015-06-29 21:16:40,368 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:16:40,368 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 21:16:40,368 INFO repeater task os150nb_beamplots starting 13.8 seconds late 2015-06-29 21:16:41,450 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:16:41,450 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 21:16:41,450 INFO repeater task ashtech_hcorrstats starting 14.9 seconds late 2015-06-29 21:16:42,165 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:16:42,165 INFO repeater task phins3_hcorrstats starting 15.6 seconds late 2015-06-29 21:16:42,832 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:16:47,838 INFO repeater task os150nb_lastensq starting 10.8 seconds late 2015-06-29 21:16:49,404 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:16:49,404 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:16:27 2015-06-29 21:17:29,476 INFO repeater task os150nb_10pings starting 2.9 seconds late 2015-06-29 21:17:29,593 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:18:39,708 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2015-06-29 21:18:41,301 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:18:41,301 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:18:27 2015-06-29 21:19:51,419 INFO repeater task os150nb_10pings starting 4.9 seconds late 2015-06-29 21:19:51,541 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:20:41,626 INFO repeater task os150nb_lastensq starting 4.6 seconds late 2015-06-29 21:20:43,275 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:20:43,275 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:20:27 2015-06-29 21:21:28,360 INFO repeater task ashtech_hcorrstats starting 1.8 seconds late 2015-06-29 21:21:29,078 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:21:29,078 INFO repeater task phins3_hcorrstats starting 2.5 seconds late 2015-06-29 21:21:29,742 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:22:09,812 INFO repeater task os150nb_10pings starting 3.3 seconds late 2015-06-29 21:22:09,930 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:22:39,984 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-06-29 21:22:41,558 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:22:41,558 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:22:27 2015-06-29 21:24:26,743 INFO repeater task os150nb_10pings starting 0.2 seconds late 2015-06-29 21:24:26,869 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:24:41,895 INFO repeater task os150nb_lastensq starting 4.9 seconds late 2015-06-29 21:24:43,502 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:24:43,502 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:24:27 2015-06-29 21:26:28,693 INFO repeater task ashtech_hcorrstats starting 2.1 seconds late 2015-06-29 21:26:29,418 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:26:29,418 INFO repeater task phins3_hcorrstats starting 2.9 seconds late 2015-06-29 21:26:30,075 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:26:40,091 INFO repeater task os150nb_lastensq starting 3.1 seconds late 2015-06-29 21:26:41,673 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:26:41,673 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:26:27 2015-06-29 21:26:46,681 INFO repeater task os150nb_10pings starting 0.1 seconds late 2015-06-29 21:26:46,802 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:28:42,003 INFO repeater task os150nb_lastensq starting 5.0 seconds late 2015-06-29 21:28:43,577 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:28:43,577 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:28:27 2015-06-29 21:29:08,619 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-06-29 21:29:08,732 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:30:38,888 INFO repeater task os150nb_lastensq starting 1.9 seconds late 2015-06-29 21:30:40,439 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:30:40,439 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:30:27 2015-06-29 21:31:30,524 INFO repeater task os150nb_10pings starting 4.0 seconds late 2015-06-29 21:31:30,651 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:31:30,651 INFO repeater task os150nb_quick starting 4.1 seconds late 2015-06-29 21:31:35,575 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 21:31:35,576 INFO repeater task os150nb_3dayplots starting 9.0 seconds late 2015-06-29 21:31:41,922 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 21:31:41,922 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2015-06-29 21:31:41,922 INFO repeater task os150nb_tsstats starting 15.4 seconds late 2015-06-29 21:31:42,664 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 21:31:42,664 INFO repeater task os150nb_beamplots starting 16.1 seconds late 2015-06-29 21:31:43,736 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:31:43,736 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 21:31:43,737 INFO repeater task os150nb_beamplots starting 17.2 seconds late 2015-06-29 21:31:44,826 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:31:44,826 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 21:31:44,827 INFO repeater task ashtech_hcorrstats starting 18.3 seconds late 2015-06-29 21:31:45,545 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:31:45,546 INFO repeater task phins3_hcorrstats starting 19.0 seconds late 2015-06-29 21:31:46,230 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:32:41,309 INFO repeater task os150nb_lastensq starting 4.3 seconds late 2015-06-29 21:32:42,894 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:32:42,895 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:32:27 2015-06-29 21:33:48,014 INFO repeater task os150nb_10pings starting 1.5 seconds late 2015-06-29 21:33:48,133 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:34:38,213 INFO repeater task os150nb_lastensq starting 1.2 seconds late 2015-06-29 21:34:39,795 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:34:39,795 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:34:27 2015-06-29 21:36:09,955 INFO repeater task os150nb_10pings starting 3.4 seconds late 2015-06-29 21:36:10,078 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:36:30,116 INFO repeater task ashtech_hcorrstats starting 3.6 seconds late 2015-06-29 21:36:30,795 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:36:30,796 INFO repeater task phins3_hcorrstats starting 4.2 seconds late 2015-06-29 21:36:31,445 INFO repeater task phins3_hcorrstats ended after 0.6 seconds 2015-06-29 21:36:41,461 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2015-06-29 21:36:43,018 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:36:43,018 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:36:27 2015-06-29 21:38:28,202 INFO repeater task os150nb_10pings starting 1.6 seconds late 2015-06-29 21:38:28,321 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:38:38,337 INFO repeater task os150nb_lastensq starting 1.3 seconds late 2015-06-29 21:38:39,940 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:38:39,940 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:38:27 2015-06-29 21:40:40,155 INFO repeater task os150nb_lastensq starting 3.2 seconds late 2015-06-29 21:40:41,747 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:40:41,748 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:40:27 2015-06-29 21:40:46,755 INFO repeater task os150nb_10pings starting 0.2 seconds late 2015-06-29 21:40:46,874 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:41:26,948 INFO repeater task ashtech_hcorrstats starting 0.4 seconds late 2015-06-29 21:41:27,644 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:41:27,645 INFO repeater task phins3_hcorrstats starting 1.1 seconds late 2015-06-29 21:41:28,309 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:42:38,408 INFO repeater task os150nb_lastensq starting 1.4 seconds late 2015-06-29 21:42:39,993 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:42:39,993 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:42:27 2015-06-29 21:43:10,043 INFO repeater task os150nb_10pings starting 3.5 seconds late 2015-06-29 21:43:10,164 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:44:40,319 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2015-06-29 21:44:41,920 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:44:41,920 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:44:27 2015-06-29 21:45:26,992 INFO repeater task os150nb_10pings starting 0.4 seconds late 2015-06-29 21:45:27,109 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:46:27,209 INFO repeater task os150nb_quick starting 0.7 seconds late 2015-06-29 21:46:32,068 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 21:46:32,069 INFO repeater task os150nb_3dayplots starting 5.5 seconds late 2015-06-29 21:46:38,308 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 21:46:38,308 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 21:46:38,308 INFO repeater task os150nb_tsstats starting 11.8 seconds late 2015-06-29 21:46:39,023 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 21:46:39,023 INFO repeater task os150nb_beamplots starting 12.5 seconds late 2015-06-29 21:46:40,060 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:46:40,061 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 21:46:40,061 INFO repeater task os150nb_beamplots starting 13.5 seconds late 2015-06-29 21:46:41,092 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 21:46:41,092 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 21:46:41,092 INFO repeater task ashtech_hcorrstats starting 14.5 seconds late 2015-06-29 21:46:41,784 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:46:41,784 INFO repeater task phins3_hcorrstats starting 15.2 seconds late 2015-06-29 21:46:42,474 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:46:47,480 INFO repeater task os150nb_lastensq starting 10.5 seconds late 2015-06-29 21:46:49,080 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:46:49,080 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:46:27 2015-06-29 21:47:49,184 INFO repeater task os150nb_10pings starting 2.6 seconds late 2015-06-29 21:47:49,307 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:48:39,392 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2015-06-29 21:48:41,015 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:48:41,015 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:48:27 2015-06-29 21:50:11,179 INFO repeater task os150nb_10pings starting 4.6 seconds late 2015-06-29 21:50:11,304 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:50:41,354 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2015-06-29 21:50:42,944 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:50:42,945 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:50:27 2015-06-29 21:51:28,030 INFO repeater task ashtech_hcorrstats starting 1.5 seconds late 2015-06-29 21:51:28,729 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:51:28,729 INFO repeater task phins3_hcorrstats starting 2.2 seconds late 2015-06-29 21:51:29,382 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:52:29,484 INFO repeater task os150nb_10pings starting 2.9 seconds late 2015-06-29 21:52:29,604 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:52:39,620 INFO repeater task os150nb_lastensq starting 2.6 seconds late 2015-06-29 21:52:41,184 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:52:41,184 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:52:27 2015-06-29 21:54:41,399 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2015-06-29 21:54:43,032 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:54:43,032 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:54:27 2015-06-29 21:54:48,040 INFO repeater task os150nb_10pings starting 1.5 seconds late 2015-06-29 21:54:48,158 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:56:28,330 INFO repeater task ashtech_hcorrstats starting 1.8 seconds late 2015-06-29 21:56:29,053 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 21:56:29,053 INFO repeater task phins3_hcorrstats starting 2.5 seconds late 2015-06-29 21:56:29,724 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 21:56:39,740 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2015-06-29 21:56:41,314 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:56:41,315 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:56:27 2015-06-29 21:57:11,355 INFO repeater task os150nb_10pings starting 4.8 seconds late 2015-06-29 21:57:11,474 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 21:58:41,588 INFO repeater task os150nb_lastensq starting 4.6 seconds late 2015-06-29 21:58:43,141 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 21:58:43,141 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 21:58:27 2015-06-29 21:59:28,220 INFO repeater task os150nb_10pings starting 1.7 seconds late 2015-06-29 21:59:28,345 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:00:38,474 INFO repeater task os150nb_lastensq starting 1.5 seconds late 2015-06-29 22:00:40,093 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:00:40,093 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:00:27 2015-06-29 22:01:30,183 INFO repeater task os150nb_quick starting 3.6 seconds late 2015-06-29 22:01:35,074 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 22:01:35,074 INFO repeater task os150nb_3dayplots starting 8.5 seconds late 2015-06-29 22:01:41,403 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 22:01:41,403 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2015-06-29 22:01:41,404 INFO repeater task os150nb_tsstats starting 14.8 seconds late 2015-06-29 22:01:42,118 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 22:01:42,118 INFO repeater task os150nb_beamplots starting 15.6 seconds late 2015-06-29 22:01:43,184 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:01:43,184 INFO repeater task os150nb_beamplots ended after 1.1 seconds 2015-06-29 22:01:43,184 INFO repeater task os150nb_beamplots starting 16.6 seconds late 2015-06-29 22:01:44,218 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:01:44,219 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:01:44,219 INFO repeater task ashtech_hcorrstats starting 17.7 seconds late 2015-06-29 22:01:44,909 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:01:44,910 INFO repeater task phins3_hcorrstats starting 18.3 seconds late 2015-06-29 22:01:45,563 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:01:50,569 INFO repeater task os150nb_10pings starting 4.0 seconds late 2015-06-29 22:01:50,698 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:02:40,788 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2015-06-29 22:02:42,333 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:02:42,333 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:02:27 2015-06-29 22:04:07,480 INFO repeater task os150nb_10pings starting 0.9 seconds late 2015-06-29 22:04:07,598 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:04:37,650 INFO repeater task os150nb_lastensq starting 0.7 seconds late 2015-06-29 22:04:39,176 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:04:39,176 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:04:27 2015-06-29 22:06:29,367 INFO repeater task os150nb_10pings starting 2.8 seconds late 2015-06-29 22:06:29,490 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:06:29,491 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2015-06-29 22:06:30,190 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:06:30,191 INFO repeater task phins3_hcorrstats starting 3.6 seconds late 2015-06-29 22:06:30,850 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:06:40,865 INFO repeater task os150nb_lastensq starting 3.9 seconds late 2015-06-29 22:06:42,389 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:06:42,389 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:06:27 2015-06-29 22:08:37,596 INFO repeater task os150nb_lastensq starting 0.6 seconds late 2015-06-29 22:08:39,176 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:08:39,176 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:08:27 2015-06-29 22:08:49,192 INFO repeater task os150nb_10pings starting 2.6 seconds late 2015-06-29 22:08:49,311 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:10:39,494 INFO repeater task os150nb_lastensq starting 2.5 seconds late 2015-06-29 22:10:41,049 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:10:41,050 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:10:27 2015-06-29 22:11:11,093 INFO repeater task os150nb_10pings starting 4.5 seconds late 2015-06-29 22:11:11,222 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:11:31,256 INFO repeater task ashtech_hcorrstats starting 4.7 seconds late 2015-06-29 22:11:31,986 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:11:31,986 INFO repeater task phins3_hcorrstats starting 5.4 seconds late 2015-06-29 22:11:32,668 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:12:37,782 INFO repeater task os150nb_lastensq starting 0.8 seconds late 2015-06-29 22:12:39,290 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:12:39,290 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:12:27 2015-06-29 22:13:29,368 INFO repeater task os150nb_10pings starting 2.8 seconds late 2015-06-29 22:13:29,485 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:14:39,609 INFO repeater task os150nb_lastensq starting 2.6 seconds late 2015-06-29 22:14:41,179 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:14:41,179 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:14:27 2015-06-29 22:15:51,304 INFO repeater task os150nb_10pings starting 4.7 seconds late 2015-06-29 22:15:51,422 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:16:31,490 INFO repeater task os150nb_quick starting 4.9 seconds late 2015-06-29 22:16:36,300 INFO repeater task os150nb_quick ended after 4.8 seconds 2015-06-29 22:16:36,301 INFO repeater task os150nb_3dayplots starting 9.7 seconds late 2015-06-29 22:16:42,571 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 22:16:42,571 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2015-06-29 22:16:42,571 INFO repeater task os150nb_tsstats starting 16.0 seconds late 2015-06-29 22:16:43,283 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 22:16:43,283 INFO repeater task os150nb_beamplots starting 16.7 seconds late 2015-06-29 22:16:44,303 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:16:44,303 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:16:44,304 INFO repeater task os150nb_beamplots starting 17.7 seconds late 2015-06-29 22:16:45,331 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:16:45,331 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:16:45,331 INFO repeater task ashtech_hcorrstats starting 18.8 seconds late 2015-06-29 22:16:46,031 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:16:46,032 INFO repeater task phins3_hcorrstats starting 19.5 seconds late 2015-06-29 22:16:46,711 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:16:51,717 INFO repeater task os150nb_lastensq starting 14.7 seconds late 2015-06-29 22:16:53,274 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:16:53,274 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:16:27 2015-06-29 22:18:08,408 INFO repeater task os150nb_10pings starting 1.9 seconds late 2015-06-29 22:18:08,531 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:18:38,579 INFO repeater task os150nb_lastensq starting 1.6 seconds late 2015-06-29 22:18:40,112 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:18:40,113 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:18:27 2015-06-29 22:20:30,319 INFO repeater task os150nb_10pings starting 3.8 seconds late 2015-06-29 22:20:30,435 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:20:40,451 INFO repeater task os150nb_lastensq starting 3.5 seconds late 2015-06-29 22:20:42,017 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:20:42,017 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:20:27 2015-06-29 22:21:27,100 INFO repeater task ashtech_hcorrstats starting 0.5 seconds late 2015-06-29 22:21:27,800 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:21:27,801 INFO repeater task phins3_hcorrstats starting 1.2 seconds late 2015-06-29 22:21:28,464 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:22:38,591 INFO repeater task os150nb_lastensq starting 1.6 seconds late 2015-06-29 22:22:40,133 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:22:40,133 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:22:27 2015-06-29 22:22:50,150 INFO repeater task os150nb_10pings starting 3.6 seconds late 2015-06-29 22:22:50,276 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:24:40,477 INFO repeater task os150nb_lastensq starting 3.5 seconds late 2015-06-29 22:24:42,016 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:24:42,016 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:24:27 2015-06-29 22:25:07,061 INFO repeater task os150nb_10pings starting 0.5 seconds late 2015-06-29 22:25:07,184 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:26:27,313 INFO repeater task ashtech_hcorrstats starting 0.8 seconds late 2015-06-29 22:26:28,030 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:26:28,030 INFO repeater task phins3_hcorrstats starting 1.5 seconds late 2015-06-29 22:26:28,699 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:26:38,714 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-06-29 22:26:40,268 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:26:40,268 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:26:27 2015-06-29 22:27:30,356 INFO repeater task os150nb_10pings starting 3.8 seconds late 2015-06-29 22:27:30,477 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:28:40,600 INFO repeater task os150nb_lastensq starting 3.6 seconds late 2015-06-29 22:28:42,187 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:28:42,187 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:28:27 2015-06-29 22:29:47,297 INFO repeater task os150nb_10pings starting 0.7 seconds late 2015-06-29 22:29:47,415 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:30:37,503 INFO repeater task os150nb_lastensq starting 0.5 seconds late 2015-06-29 22:30:39,043 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:30:39,044 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:30:27 2015-06-29 22:31:29,133 INFO repeater task os150nb_quick starting 2.6 seconds late 2015-06-29 22:31:34,048 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 22:31:34,049 INFO repeater task os150nb_3dayplots starting 7.5 seconds late 2015-06-29 22:31:40,439 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 22:31:40,440 INFO repeater task os150nb_3dayplots ended after 6.4 seconds 2015-06-29 22:31:40,440 INFO repeater task os150nb_tsstats starting 13.9 seconds late 2015-06-29 22:31:41,164 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 22:31:41,164 INFO repeater task os150nb_beamplots starting 14.6 seconds late 2015-06-29 22:31:42,197 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:31:42,197 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:31:42,197 INFO repeater task os150nb_beamplots starting 15.6 seconds late 2015-06-29 22:31:43,221 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:31:43,221 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:31:43,222 INFO repeater task ashtech_hcorrstats starting 16.7 seconds late 2015-06-29 22:31:43,925 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:31:43,925 INFO repeater task phins3_hcorrstats starting 17.4 seconds late 2015-06-29 22:31:44,599 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:32:09,642 INFO repeater task os150nb_10pings starting 3.1 seconds late 2015-06-29 22:32:09,769 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:32:39,816 INFO repeater task os150nb_lastensq starting 2.8 seconds late 2015-06-29 22:32:41,443 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:32:41,444 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:32:27 2015-06-29 22:34:26,638 INFO repeater task os150nb_10pings starting 0.1 seconds late 2015-06-29 22:34:26,762 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:34:41,788 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-06-29 22:34:43,355 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:34:43,355 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:34:27 2015-06-29 22:36:28,550 INFO repeater task ashtech_hcorrstats starting 2.0 seconds late 2015-06-29 22:36:29,275 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:36:29,276 INFO repeater task phins3_hcorrstats starting 2.7 seconds late 2015-06-29 22:36:29,943 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:36:39,954 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2015-06-29 22:36:41,512 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:36:41,513 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:36:27 2015-06-29 22:36:51,527 INFO repeater task os150nb_10pings starting 5.0 seconds late 2015-06-29 22:36:51,650 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:38:41,847 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2015-06-29 22:38:43,404 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:38:43,404 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:38:27 2015-06-29 22:39:08,449 INFO repeater task os150nb_10pings starting 1.9 seconds late 2015-06-29 22:39:08,570 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:40:38,726 INFO repeater task os150nb_lastensq starting 1.7 seconds late 2015-06-29 22:40:40,304 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:40:40,304 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:40:27 2015-06-29 22:41:30,391 INFO repeater task os150nb_10pings starting 3.8 seconds late 2015-06-29 22:41:30,508 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:41:30,509 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2015-06-29 22:41:31,206 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:41:31,207 INFO repeater task phins3_hcorrstats starting 4.6 seconds late 2015-06-29 22:41:31,889 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:42:42,000 INFO repeater task os150nb_lastensq starting 5.0 seconds late 2015-06-29 22:42:43,580 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:42:43,580 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:42:27 2015-06-29 22:43:48,700 INFO repeater task os150nb_10pings starting 2.1 seconds late 2015-06-29 22:43:48,825 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:44:38,907 INFO repeater task os150nb_lastensq starting 1.9 seconds late 2015-06-29 22:44:40,471 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:44:40,472 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:44:27 2015-06-29 22:46:10,618 INFO repeater task os150nb_10pings starting 4.1 seconds late 2015-06-29 22:46:10,746 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:46:30,777 INFO repeater task os150nb_quick starting 4.2 seconds late 2015-06-29 22:46:35,657 INFO repeater task os150nb_quick ended after 4.9 seconds 2015-06-29 22:46:35,658 INFO repeater task os150nb_3dayplots starting 9.1 seconds late 2015-06-29 22:46:41,882 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-06-29 22:46:41,882 INFO repeater task os150nb_3dayplots ended after 6.2 seconds 2015-06-29 22:46:41,882 INFO repeater task os150nb_tsstats starting 15.3 seconds late 2015-06-29 22:46:42,599 INFO repeater task os150nb_tsstats ended after 0.7 seconds 2015-06-29 22:46:42,599 INFO repeater task os150nb_beamplots starting 16.0 seconds late 2015-06-29 22:46:43,625 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:46:43,626 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:46:43,626 INFO repeater task os150nb_beamplots starting 17.1 seconds late 2015-06-29 22:46:44,662 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-06-29 22:46:44,662 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2015-06-29 22:46:44,662 INFO repeater task ashtech_hcorrstats starting 18.1 seconds late 2015-06-29 22:46:45,363 INFO repeater task ashtech_hcorrstats ended after 0.7 seconds 2015-06-29 22:46:45,363 INFO repeater task phins3_hcorrstats starting 18.8 seconds late 2015-06-29 22:46:46,027 INFO repeater task phins3_hcorrstats ended after 0.7 seconds 2015-06-29 22:46:51,033 INFO repeater task os150nb_lastensq starting 14.0 seconds late 2015-06-29 22:46:52,597 INFO repeater task os150nb_lastensq ended after 1.6 seconds 2015-06-29 22:46:52,597 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:46:27 2015-06-29 22:48:27,772 INFO repeater task os150nb_10pings starting 1.2 seconds late 2015-06-29 22:48:27,892 INFO repeater task os150nb_10pings ended after 0.1 seconds 2015-06-29 22:48:37,907 INFO repeater task os150nb_lastensq starting 0.9 seconds late 2015-06-29 22:48:39,446 INFO repeater task os150nb_lastensq ended after 1.5 seconds 2015-06-29 22:48:39,446 INFO repeater task os150nb_lastensq: Time from /home/data/RR1509/proc/os150nb/load/wrote_ens_time: 2015/06/29 22:48:27 2015-06-29 22:49:38,688 INFO Translate translate exiting, hdg /home/data/RR1509/rbin/phins3 2015-06-29 22:49:38,692 INFO Translate translate exiting, adu /home/data/RR1509/rbin/ashtech 2015-06-29 22:49:38,715 INFO Translate translate exiting, gps /home/data/RR1509/rbin/gpsnav 2015-06-29 22:49:38,728 INFO Translate translate exiting, gps /home/data/RR1509/rbin/ashtech 2015-06-29 22:49:38,728 INFO Translate translate exiting, gps /home/data/RR1509/rbin/hydrins 2015-06-29 22:49:38,737 INFO Translate translate exiting, hdg /home/data/RR1509/rbin/hydrins 2015-06-29 22:49:39,553 INFO root cleanup_figs not_logging 2015-06-29 22:49:39,565 INFO root Exiting DAS_while_logging.py