2011-10-04 01:00:43,744 INFO     repeater     task rsync_/disk2/home starting 1.4 seconds late
2011-10-04 01:00:43,935 INFO     repeater     task rsync_/disk2/home ended after 0.2 seconds
2011-10-04 01:00:43,936 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.6 seconds late
2011-10-04 01:00:45,143 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.2 seconds
2011-10-04 02:00:45,699 INFO     repeater     task rsync_/disk2/home starting 3.3 seconds late
2011-10-04 02:00:45,951 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 02:00:45,952 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.6 seconds late
2011-10-04 02:00:47,288 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.3 seconds
2011-10-04 03:00:42,712 INFO     repeater     task rsync_/disk2/home starting 0.3 seconds late
2011-10-04 03:00:42,919 INFO     repeater     task rsync_/disk2/home ended after 0.2 seconds
2011-10-04 03:00:42,919 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 0.5 seconds late
2011-10-04 03:00:46,607 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 3.7 seconds
2011-10-04 04:00:47,079 INFO     repeater     task rsync_/disk2/home starting 4.7 seconds late
2011-10-04 04:00:47,336 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 04:00:47,337 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.0 seconds late
2011-10-04 04:00:48,763 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.4 seconds
2011-10-04 05:00:44,376 INFO     repeater     task rsync_/disk2/home starting 2.0 seconds late
2011-10-04 05:00:44,615 INFO     repeater     task rsync_/disk2/home ended after 0.2 seconds
2011-10-04 05:00:44,616 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.2 seconds late
2011-10-04 05:00:45,863 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.2 seconds
2011-10-04 06:00:46,375 INFO     repeater     task rsync_/disk2/home starting 4.0 seconds late
2011-10-04 06:00:46,659 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 06:00:46,660 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.3 seconds late
2011-10-04 06:00:48,260 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-04 07:00:43,328 INFO     repeater     task rsync_/disk2/home starting 0.9 seconds late
2011-10-04 07:00:43,368 INFO     repeater     task rsync_/disk2/home ended after 0.0 seconds
2011-10-04 07:00:43,369 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.0 seconds late
2011-10-04 07:00:43,919 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 0.6 seconds
2011-10-04 08:00:44,132 INFO     repeater     task rsync_/disk2/home starting 1.7 seconds late
2011-10-04 08:00:44,199 INFO     repeater     task rsync_/disk2/home ended after 0.1 seconds
2011-10-04 08:00:44,200 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.8 seconds late
2011-10-04 08:00:44,579 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 0.4 seconds
2011-10-04 09:00:44,919 INFO     repeater     task rsync_/disk2/home starting 2.5 seconds late
2011-10-04 09:00:45,143 INFO     repeater     task rsync_/disk2/home ended after 0.2 seconds
2011-10-04 09:00:45,144 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.8 seconds late
2011-10-04 09:00:46,468 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.3 seconds
2011-10-04 10:00:46,851 INFO     repeater     task rsync_/disk2/home starting 4.5 seconds late
2011-10-04 10:00:47,095 INFO     repeater     task rsync_/disk2/home ended after 0.2 seconds
2011-10-04 10:00:47,096 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.7 seconds late
2011-10-04 10:00:51,412 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 4.3 seconds
2011-10-04 11:00:47,011 INFO     repeater     task rsync_/disk2/home starting 4.6 seconds late
2011-10-04 11:00:47,376 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-04 11:00:47,377 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.0 seconds late
2011-10-04 11:00:48,956 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-04 12:00:44,544 INFO     repeater     task rsync_/disk2/home starting 2.2 seconds late
2011-10-04 12:00:44,992 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-04 12:00:44,993 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.6 seconds late
2011-10-04 12:00:50,071 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 5.1 seconds
2011-10-04 13:00:45,592 INFO     repeater     task rsync_/disk2/home starting 3.2 seconds late
2011-10-04 13:00:45,851 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 13:00:45,852 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.5 seconds late
2011-10-04 13:00:50,739 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 4.9 seconds
2011-10-04 14:00:46,204 INFO     repeater     task rsync_/disk2/home starting 3.8 seconds late
2011-10-04 14:00:46,476 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 14:00:46,476 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.1 seconds late
2011-10-04 14:00:48,500 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-04 15:00:43,835 INFO     repeater     task rsync_/disk2/home starting 1.5 seconds late
2011-10-04 15:00:44,099 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 15:00:44,100 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.7 seconds late
2011-10-04 15:00:45,539 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.4 seconds
2011-10-04 16:00:45,932 INFO     repeater     task rsync_/disk2/home starting 3.5 seconds late
2011-10-04 16:00:46,239 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 16:00:46,240 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.9 seconds late
2011-10-04 16:00:47,767 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.5 seconds
2011-10-04 17:00:43,103 INFO     repeater     task rsync_/disk2/home starting 0.7 seconds late
2011-10-04 17:00:43,504 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-04 17:00:43,519 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.1 seconds late
2011-10-04 17:00:50,344 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 6.8 seconds
2011-10-04 18:00:45,854 INFO     repeater     task rsync_/disk2/home starting 3.5 seconds late
2011-10-04 18:00:46,267 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-04 18:00:46,268 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.9 seconds late
2011-10-04 18:00:50,760 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 4.5 seconds
2011-10-04 19:00:46,389 INFO     repeater     task rsync_/disk2/home starting 4.0 seconds late
2011-10-04 19:00:46,648 INFO     repeater     task rsync_/disk2/home ended after 0.3 seconds
2011-10-04 19:00:46,649 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.3 seconds late
2011-10-04 19:00:48,191 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.5 seconds
2011-10-04 20:00:43,737 INFO     repeater     task rsync_/disk2/home starting 1.4 seconds late
2011-10-04 20:00:44,352 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-04 20:00:44,353 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.0 seconds late
2011-10-04 20:00:45,907 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-04 21:00:46,309 INFO     repeater     task rsync_/disk2/home starting 3.9 seconds late
2011-10-04 21:00:46,704 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-04 21:00:46,704 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.3 seconds late
2011-10-04 21:00:48,271 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-04 22:00:43,908 INFO     repeater     task rsync_/disk2/home starting 1.5 seconds late
2011-10-04 22:00:44,402 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-04 22:00:44,403 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.0 seconds late
2011-10-04 22:00:46,059 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-04 23:00:46,668 INFO     repeater     task rsync_/disk2/home starting 4.3 seconds late
2011-10-04 23:00:47,295 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-04 23:00:47,296 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.9 seconds late
2011-10-04 23:00:48,911 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-05 00:00:44,513 INFO     repeater     task rsync_/disk2/home starting 2.1 seconds late
2011-10-05 00:00:45,175 INFO     repeater     task rsync_/disk2/home ended after 0.7 seconds
2011-10-05 00:00:45,193 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.8 seconds late
2011-10-05 00:00:46,944 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 01:00:42,627 INFO     repeater     task rsync_/disk2/home starting 0.2 seconds late
2011-10-05 01:00:43,023 DEBUG    repeater     task rsync_/disk2/home returncode 24
2011-10-05 01:00:43,023 INFO     repeater     task rsync_/disk2/home output:
file has vanished: "/home/data/TN269_leg2/proc/os75bb/edit/abadbin_tmp.asc"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(977) [sender=2.6.9]

2011-10-05 01:00:43,024 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-05 01:00:43,024 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 0.6 seconds late
2011-10-05 01:00:44,612 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-05 02:00:45,053 INFO     repeater     task rsync_/disk2/home starting 2.7 seconds late
2011-10-05 02:00:45,560 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-05 02:00:45,561 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.2 seconds late
2011-10-05 02:00:47,284 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 03:00:42,571 INFO     repeater     task rsync_/disk2/home starting 0.2 seconds late
2011-10-05 03:00:43,211 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 03:00:43,227 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 0.8 seconds late
2011-10-05 03:00:44,891 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 04:00:45,609 INFO     repeater     task rsync_/disk2/home starting 3.2 seconds late
2011-10-05 04:00:46,632 INFO     repeater     task rsync_/disk2/home ended after 1.0 seconds
2011-10-05 04:00:46,648 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.3 seconds late
2011-10-05 04:00:48,435 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 05:00:44,043 INFO     repeater     task rsync_/disk2/home starting 1.7 seconds late
2011-10-05 05:00:44,563 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-05 05:00:44,578 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.2 seconds late
2011-10-05 05:00:46,408 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 06:00:46,816 INFO     repeater     task rsync_/disk2/home starting 4.4 seconds late
2011-10-05 06:00:47,376 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 06:00:47,377 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.0 seconds late
2011-10-05 06:00:49,123 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 07:00:44,721 INFO     repeater     task rsync_/disk2/home starting 2.3 seconds late
2011-10-05 07:00:45,124 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-05 07:00:45,125 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.7 seconds late
2011-10-05 07:00:46,681 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-05 08:00:47,085 INFO     repeater     task rsync_/disk2/home starting 4.7 seconds late
2011-10-05 08:00:48,087 INFO     repeater     task rsync_/disk2/home ended after 1.0 seconds
2011-10-05 08:00:48,088 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.7 seconds late
2011-10-05 08:00:49,727 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.6 seconds
2011-10-05 09:00:45,224 INFO     repeater     task rsync_/disk2/home starting 2.8 seconds late
2011-10-05 09:00:45,867 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 09:00:45,895 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.5 seconds late
2011-10-05 09:00:47,549 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 10:00:43,075 INFO     repeater     task rsync_/disk2/home starting 0.7 seconds late
2011-10-05 10:00:43,692 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 10:00:43,708 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.3 seconds late
2011-10-05 10:00:45,500 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 11:00:46,108 INFO     repeater     task rsync_/disk2/home starting 3.7 seconds late
2011-10-05 11:00:46,756 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 11:00:46,774 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.4 seconds late
2011-10-05 11:00:48,587 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 12:00:44,277 INFO     repeater     task rsync_/disk2/home starting 1.9 seconds late
2011-10-05 12:00:44,772 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-05 12:00:45,216 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.8 seconds late
2011-10-05 12:00:47,112 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-05 13:00:42,637 INFO     repeater     task rsync_/disk2/home starting 0.3 seconds late
2011-10-05 13:00:43,047 INFO     repeater     task rsync_/disk2/home ended after 0.4 seconds
2011-10-05 13:00:43,048 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 0.7 seconds late
2011-10-05 13:00:44,759 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 14:00:45,236 INFO     repeater     task rsync_/disk2/home starting 2.9 seconds late
2011-10-05 14:00:45,784 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-05 14:00:45,785 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.4 seconds late
2011-10-05 14:00:47,675 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-05 15:00:43,147 INFO     repeater     task rsync_/disk2/home starting 0.8 seconds late
2011-10-05 15:00:43,648 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-05 15:00:43,649 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.3 seconds late
2011-10-05 15:00:45,371 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 16:00:45,791 INFO     repeater     task rsync_/disk2/home starting 3.4 seconds late
2011-10-05 16:00:46,451 INFO     repeater     task rsync_/disk2/home ended after 0.7 seconds
2011-10-05 16:00:46,452 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.1 seconds late
2011-10-05 16:00:48,264 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 17:00:43,800 INFO     repeater     task rsync_/disk2/home starting 1.4 seconds late
2011-10-05 17:00:44,419 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 17:00:44,420 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.0 seconds late
2011-10-05 17:00:46,211 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 18:00:46,635 INFO     repeater     task rsync_/disk2/home starting 4.3 seconds late
2011-10-05 18:00:47,472 INFO     repeater     task rsync_/disk2/home ended after 0.8 seconds
2011-10-05 18:00:47,673 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.3 seconds late
2011-10-05 18:00:49,480 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 19:00:44,888 INFO     repeater     task rsync_/disk2/home starting 2.5 seconds late
2011-10-05 19:00:45,667 INFO     repeater     task rsync_/disk2/home ended after 0.8 seconds
2011-10-05 19:00:45,668 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.3 seconds late
2011-10-05 19:00:47,344 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 20:00:42,776 INFO     repeater     task rsync_/disk2/home starting 0.4 seconds late
2011-10-05 20:00:43,635 INFO     repeater     task rsync_/disk2/home ended after 0.9 seconds
2011-10-05 20:00:43,636 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.3 seconds late
2011-10-05 20:00:45,404 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-05 21:00:45,855 INFO     repeater     task rsync_/disk2/home starting 3.5 seconds late
2011-10-05 21:00:46,463 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 21:00:46,464 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.1 seconds late
2011-10-05 21:00:48,135 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.7 seconds
2011-10-05 22:00:43,596 INFO     repeater     task rsync_/disk2/home starting 1.2 seconds late
2011-10-05 22:00:44,207 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-05 22:00:44,232 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.8 seconds late
2011-10-05 22:00:46,203 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-05 23:00:46,696 INFO     repeater     task rsync_/disk2/home starting 4.3 seconds late
2011-10-05 23:00:47,409 INFO     repeater     task rsync_/disk2/home ended after 0.7 seconds
2011-10-05 23:00:47,423 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.0 seconds late
2011-10-05 23:00:49,275 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-06 00:00:44,904 INFO     repeater     task rsync_/disk2/home starting 2.5 seconds late
2011-10-06 00:00:46,035 INFO     repeater     task rsync_/disk2/home ended after 1.1 seconds
2011-10-06 00:00:46,048 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.7 seconds late
2011-10-06 00:00:48,015 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 01:00:43,492 INFO     repeater     task rsync_/disk2/home starting 1.1 seconds late
2011-10-06 01:00:44,011 INFO     repeater     task rsync_/disk2/home ended after 0.5 seconds
2011-10-06 01:00:44,012 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.6 seconds late
2011-10-06 01:00:45,899 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-06 02:00:46,524 INFO     repeater     task rsync_/disk2/home starting 4.1 seconds late
2011-10-06 02:00:47,699 ERROR    repeater     task rsync_/disk2/home returncode 23
2011-10-06 02:00:47,701 INFO     repeater     task rsync_/disk2/home output:
rsync: read errors mapping "/home/data/TN269_leg2/proc/os75bb/contour/allbins_resid_stats.mat": No data available (61)
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

2011-10-06 02:00:47,701 INFO     repeater     task rsync_/disk2/home ended after 1.2 seconds
2011-10-06 02:00:47,716 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.3 seconds late
2011-10-06 02:00:49,753 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 03:00:45,223 INFO     repeater     task rsync_/disk2/home starting 2.8 seconds late
2011-10-06 03:00:46,048 INFO     repeater     task rsync_/disk2/home ended after 0.8 seconds
2011-10-06 03:00:46,049 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.7 seconds late
2011-10-06 03:00:48,019 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 04:00:43,507 INFO     repeater     task rsync_/disk2/home starting 1.1 seconds late
2011-10-06 04:00:44,419 INFO     repeater     task rsync_/disk2/home ended after 0.9 seconds
2011-10-06 04:00:44,419 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.0 seconds late
2011-10-06 04:00:46,407 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 05:00:46,977 INFO     repeater     task rsync_/disk2/home starting 4.6 seconds late
2011-10-06 05:00:48,043 INFO     repeater     task rsync_/disk2/home ended after 1.1 seconds
2011-10-06 05:00:48,207 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.8 seconds late
2011-10-06 05:00:50,079 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-06 06:00:45,487 INFO     repeater     task rsync_/disk2/home starting 3.1 seconds late
2011-10-06 06:00:46,287 INFO     repeater     task rsync_/disk2/home ended after 0.8 seconds
2011-10-06 06:00:46,288 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.9 seconds late
2011-10-06 06:00:48,355 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-06 07:00:43,852 INFO     repeater     task rsync_/disk2/home starting 1.5 seconds late
2011-10-06 07:00:44,467 INFO     repeater     task rsync_/disk2/home ended after 0.6 seconds
2011-10-06 07:00:44,483 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.1 seconds late
2011-10-06 07:00:46,352 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-06 08:00:46,735 INFO     repeater     task rsync_/disk2/home starting 4.4 seconds late
2011-10-06 08:00:47,959 INFO     repeater     task rsync_/disk2/home ended after 1.2 seconds
2011-10-06 08:00:47,979 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.6 seconds late
2011-10-06 08:00:49,984 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 09:00:45,535 INFO     repeater     task rsync_/disk2/home starting 3.2 seconds late
2011-10-06 09:00:46,760 INFO     repeater     task rsync_/disk2/home ended after 1.2 seconds
2011-10-06 09:00:46,779 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.4 seconds late
2011-10-06 09:00:48,607 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.8 seconds
2011-10-06 10:00:44,126 INFO     repeater     task rsync_/disk2/home starting 1.7 seconds late
2011-10-06 10:00:45,052 INFO     repeater     task rsync_/disk2/home ended after 0.9 seconds
2011-10-06 10:00:45,071 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.7 seconds late
2011-10-06 10:00:47,148 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-06 11:00:42,670 INFO     repeater     task rsync_/disk2/home starting 0.3 seconds late
2011-10-06 11:00:43,464 INFO     repeater     task rsync_/disk2/home ended after 0.8 seconds
2011-10-06 11:00:43,487 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.1 seconds late
2011-10-06 11:00:45,444 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 12:00:45,900 INFO     repeater     task rsync_/disk2/home starting 3.5 seconds late
2011-10-06 12:00:46,691 INFO     repeater     task rsync_/disk2/home ended after 0.8 seconds
2011-10-06 12:00:46,708 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.3 seconds late
2011-10-06 12:00:48,816 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-06 13:00:44,393 INFO     repeater     task rsync_/disk2/home starting 2.0 seconds late
2011-10-06 13:00:47,111 INFO     repeater     task rsync_/disk2/home ended after 2.7 seconds
2011-10-06 13:00:47,127 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.7 seconds late
2011-10-06 13:00:49,076 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 1.9 seconds
2011-10-06 14:00:44,640 INFO     repeater     task rsync_/disk2/home starting 2.3 seconds late
2011-10-06 14:00:46,307 INFO     repeater     task rsync_/disk2/home ended after 1.7 seconds
2011-10-06 14:00:46,307 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.9 seconds late
2011-10-06 14:00:48,327 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 15:00:43,723 INFO     repeater     task rsync_/disk2/home starting 1.3 seconds late
2011-10-06 15:00:47,140 INFO     repeater     task rsync_/disk2/home ended after 3.4 seconds
2011-10-06 15:00:47,141 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.8 seconds late
2011-10-06 15:00:49,155 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 16:00:44,586 INFO     repeater     task rsync_/disk2/home starting 2.2 seconds late
2011-10-06 16:00:48,187 INFO     repeater     task rsync_/disk2/home ended after 3.6 seconds
2011-10-06 16:00:48,203 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.8 seconds late
2011-10-06 16:00:50,376 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-06 17:00:46,007 INFO     repeater     task rsync_/disk2/home starting 3.6 seconds late
2011-10-06 17:00:50,149 INFO     repeater     task rsync_/disk2/home ended after 4.1 seconds
2011-10-06 17:00:50,150 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 7.8 seconds late
2011-10-06 17:00:52,211 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-06 18:00:42,552 INFO     repeater     task rsync_/disk2/home starting 0.2 seconds late
2011-10-06 18:00:45,656 INFO     repeater     task rsync_/disk2/home ended after 3.1 seconds
2011-10-06 18:00:45,690 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.3 seconds late
2011-10-06 18:00:48,003 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.3 seconds
2011-10-06 19:00:43,759 INFO     repeater     task rsync_/disk2/home starting 1.4 seconds late
2011-10-06 19:00:45,724 INFO     repeater     task rsync_/disk2/home ended after 2.0 seconds
2011-10-06 19:00:45,727 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.3 seconds late
2011-10-06 19:00:47,731 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 20:00:43,183 INFO     repeater     task rsync_/disk2/home starting 0.8 seconds late
2011-10-06 20:00:46,291 INFO     repeater     task rsync_/disk2/home ended after 3.1 seconds
2011-10-06 20:00:46,292 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.9 seconds late
2011-10-06 20:00:48,467 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-06 21:00:43,920 INFO     repeater     task rsync_/disk2/home starting 1.5 seconds late
2011-10-06 21:00:45,818 INFO     repeater     task rsync_/disk2/home ended after 1.9 seconds
2011-10-06 21:00:45,818 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.4 seconds late
2011-10-06 21:00:47,855 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-06 22:00:43,328 INFO     repeater     task rsync_/disk2/home starting 0.9 seconds late
2011-10-06 22:00:46,320 INFO     repeater     task rsync_/disk2/home ended after 3.0 seconds
2011-10-06 22:00:46,407 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.0 seconds late
2011-10-06 22:00:48,584 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-06 23:00:44,043 INFO     repeater     task rsync_/disk2/home starting 1.7 seconds late
2011-10-06 23:00:47,260 INFO     repeater     task rsync_/disk2/home ended after 3.2 seconds
2011-10-06 23:00:47,261 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.9 seconds late
2011-10-06 23:00:49,259 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.0 seconds
2011-10-07 00:00:44,672 INFO     repeater     task rsync_/disk2/home starting 2.3 seconds late
2011-10-07 00:00:47,447 INFO     repeater     task rsync_/disk2/home ended after 2.8 seconds
2011-10-07 00:00:47,448 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.1 seconds late
2011-10-07 00:00:49,659 ERROR    repeater     task rsync_/mnt/snapserv1/adcp returncode 23
2011-10-07 00:00:49,660 INFO     repeater     task rsync_/mnt/snapserv1/adcp output:
rsync: read errors mapping "/home/data/TN269_leg2/proc/os75bb/contour/allbins_u.mat": No data available (61)
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]

2011-10-07 00:00:49,660 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 01:00:45,167 INFO     repeater     task rsync_/disk2/home starting 2.8 seconds late
2011-10-07 01:00:47,979 INFO     repeater     task rsync_/disk2/home ended after 2.8 seconds
2011-10-07 01:00:47,980 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.6 seconds late
2011-10-07 01:00:50,115 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-07 02:00:45,784 INFO     repeater     task rsync_/disk2/home starting 3.4 seconds late
2011-10-07 02:00:49,695 INFO     repeater     task rsync_/disk2/home ended after 3.9 seconds
2011-10-07 02:00:49,696 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 7.3 seconds late
2011-10-07 02:00:51,831 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-07 03:00:47,163 INFO     repeater     task rsync_/disk2/home starting 4.8 seconds late
2011-10-07 03:00:51,595 INFO     repeater     task rsync_/disk2/home ended after 4.4 seconds
2011-10-07 03:00:51,602 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 9.2 seconds late
2011-10-07 03:00:53,803 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 04:00:44,419 INFO     repeater     task rsync_/disk2/home starting 2.0 seconds late
2011-10-07 04:00:47,760 INFO     repeater     task rsync_/disk2/home ended after 3.3 seconds
2011-10-07 04:00:47,761 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.4 seconds late
2011-10-07 04:00:49,959 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 05:00:45,548 INFO     repeater     task rsync_/disk2/home starting 3.2 seconds late
2011-10-07 05:00:49,591 INFO     repeater     task rsync_/disk2/home ended after 4.0 seconds
2011-10-07 05:00:49,594 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 7.2 seconds late
2011-10-07 05:00:51,967 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 06:00:42,675 INFO     repeater     task rsync_/disk2/home starting 0.3 seconds late
2011-10-07 06:00:47,340 INFO     repeater     task rsync_/disk2/home ended after 4.7 seconds
2011-10-07 06:00:47,341 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.0 seconds late
2011-10-07 06:00:49,651 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.3 seconds
2011-10-07 07:00:45,024 INFO     repeater     task rsync_/disk2/home starting 2.6 seconds late
2011-10-07 07:00:48,463 INFO     repeater     task rsync_/disk2/home ended after 3.4 seconds
2011-10-07 07:00:48,483 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 6.1 seconds late
2011-10-07 07:00:50,684 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 08:00:46,309 INFO     repeater     task rsync_/disk2/home starting 3.9 seconds late
2011-10-07 08:00:51,007 INFO     repeater     task rsync_/disk2/home ended after 4.7 seconds
2011-10-07 08:00:51,008 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 8.6 seconds late
2011-10-07 08:00:53,212 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 09:00:43,660 INFO     repeater     task rsync_/disk2/home starting 1.3 seconds late
2011-10-07 09:00:47,992 INFO     repeater     task rsync_/disk2/home ended after 4.3 seconds
2011-10-07 09:00:47,993 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.6 seconds late
2011-10-07 09:00:50,092 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-07 10:00:45,559 INFO     repeater     task rsync_/disk2/home starting 3.2 seconds late
2011-10-07 10:00:49,924 INFO     repeater     task rsync_/disk2/home ended after 4.4 seconds
2011-10-07 10:00:49,947 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 7.6 seconds late
2011-10-07 10:00:52,356 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 11:00:43,024 INFO     repeater     task rsync_/disk2/home starting 0.6 seconds late
2011-10-07 11:00:46,359 INFO     repeater     task rsync_/disk2/home ended after 3.3 seconds
2011-10-07 11:00:46,360 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.0 seconds late
2011-10-07 11:00:48,463 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.1 seconds
2011-10-07 12:00:43,840 INFO     repeater     task rsync_/disk2/home starting 1.5 seconds late
2011-10-07 12:00:47,679 INFO     repeater     task rsync_/disk2/home ended after 3.8 seconds
2011-10-07 12:00:47,699 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.3 seconds late
2011-10-07 12:00:50,128 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 13:00:45,710 INFO     repeater     task rsync_/disk2/home starting 3.3 seconds late
2011-10-07 13:00:50,075 INFO     repeater     task rsync_/disk2/home ended after 4.4 seconds
2011-10-07 13:00:50,076 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 7.7 seconds late
2011-10-07 13:00:52,327 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.3 seconds
2011-10-07 14:00:42,760 INFO     repeater     task rsync_/disk2/home starting 0.4 seconds late
2011-10-07 14:00:50,763 INFO     repeater     task rsync_/disk2/home ended after 8.0 seconds
2011-10-07 14:00:50,764 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 8.4 seconds late
2011-10-07 14:00:53,119 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 15:00:43,820 INFO     repeater     task rsync_/disk2/home starting 1.4 seconds late
2011-10-07 15:00:47,521 INFO     repeater     task rsync_/disk2/home ended after 3.7 seconds
2011-10-07 15:00:47,521 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.1 seconds late
2011-10-07 15:00:49,715 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 16:00:45,148 INFO     repeater     task rsync_/disk2/home starting 2.8 seconds late
2011-10-07 16:00:49,399 INFO     repeater     task rsync_/disk2/home ended after 4.3 seconds
2011-10-07 16:00:49,415 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 7.0 seconds late
2011-10-07 16:00:51,799 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 17:00:42,452 INFO     repeater     task rsync_/disk2/home starting 0.1 seconds late
2011-10-07 17:00:43,672 INFO     repeater     task rsync_/disk2/home ended after 1.2 seconds
2011-10-07 17:00:43,675 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 1.3 seconds late
2011-10-07 17:00:46,040 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 18:00:46,614 INFO     repeater     task rsync_/disk2/home starting 4.2 seconds late
2011-10-07 18:00:48,151 INFO     repeater     task rsync_/disk2/home ended after 1.5 seconds
2011-10-07 18:00:48,167 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.8 seconds late
2011-10-07 18:00:50,560 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 19:00:46,191 INFO     repeater     task rsync_/disk2/home starting 3.8 seconds late
2011-10-07 19:00:47,451 INFO     repeater     task rsync_/disk2/home ended after 1.3 seconds
2011-10-07 19:00:47,467 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 5.1 seconds late
2011-10-07 19:00:49,848 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds
2011-10-07 20:00:45,456 INFO     repeater     task rsync_/disk2/home starting 3.1 seconds late
2011-10-07 20:00:47,003 INFO     repeater     task rsync_/disk2/home ended after 1.5 seconds
2011-10-07 20:00:47,016 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.6 seconds late
2011-10-07 20:00:49,340 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.3 seconds
2011-10-07 21:00:44,815 INFO     repeater     task rsync_/disk2/home starting 2.4 seconds late
2011-10-07 21:00:46,776 INFO     repeater     task rsync_/disk2/home ended after 2.0 seconds
2011-10-07 21:00:46,777 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 4.4 seconds late
2011-10-07 21:00:48,995 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.2 seconds
2011-10-07 22:00:44,473 INFO     repeater     task rsync_/disk2/home starting 2.1 seconds late
2011-10-07 22:00:46,056 INFO     repeater     task rsync_/disk2/home ended after 1.6 seconds
2011-10-07 22:00:46,057 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.7 seconds late
2011-10-07 22:00:48,332 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.3 seconds
2011-10-07 23:00:43,710 INFO     repeater     task rsync_/disk2/home starting 1.3 seconds late
2011-10-07 23:00:45,467 INFO     repeater     task rsync_/disk2/home ended after 1.8 seconds
2011-10-07 23:00:45,468 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 3.1 seconds late
2011-10-07 23:00:47,723 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.3 seconds
2011-10-08 00:00:43,357 INFO     repeater     task rsync_/disk2/home starting 1.0 seconds late
2011-10-08 00:00:45,079 INFO     repeater     task rsync_/disk2/home ended after 1.7 seconds
2011-10-08 00:00:45,080 INFO     repeater     task rsync_/mnt/snapserv1/adcp starting 2.7 seconds late
2011-10-08 00:00:47,476 INFO     repeater     task rsync_/mnt/snapserv1/adcp ended after 2.4 seconds