[2013-02-25 19:44:43.533569] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-7: server 172.16.5.3:24012 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:44:43.542444] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:41.544123 (xid=0x326599x)
[2013-02-25 19:44:43.542468] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:43.542544] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:42.454724 (xid=0x326609x)
[2013-02-25 19:44:43.542556] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:43.542599] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:44.313093 (xid=0x326629x)
[2013-02-25 19:44:43.542610] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:43.542651] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:46.391951 (xid=0x326677x)
[2013-02-25 19:44:43.542672] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:43.542716] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:49.825564 (xid=0x326778x)
[2013-02-25 19:44:43.542727] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:43.542768] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:52.009537 (xid=0x326808x)
[2013-02-25 19:44:43.542779] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:43.542820] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:39.940308 (xid=0x326575x)
[2013-02-25 19:44:43.542840] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:43.542912] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:39.969168 (xid=0x326576x)
[2013-02-25 19:44:43.542929] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:43.542979] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:40.075152 (xid=0x326577x)
[2013-02-25 19:44:43.542991] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:43.543035] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:40.085799 (xid=0x326578x)
[2013-02-25 19:44:43.559421] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:43.559466] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.281236 (xid=0x326897x)
[2013-02-25 19:44:43.559480] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:43.559534] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:44:01.527788 (xid=0x326898x)
[2013-02-25 19:44:43.560540] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-7: timer must have expired
[2013-02-25 19:44:43.560591] I [client.c:2090:client_rpc_notify] 0-adata-client-7: disconnected
[2013-02-25 19:44:43.568372] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (b468c831-5c04-4bbd-8b03-b94d7f937e55) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568405] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (b468c831-5c04-4bbd-8b03-b94d7f937e55) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568424] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (6283b736-96c5-4e26-af53-8dd78496db18) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568446] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (6283b736-96c5-4e26-af53-8dd78496db18) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568466] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (841636a6-ceb2-438a-a5a5-9826cdb3e484) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568493] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (841636a6-ceb2-438a-a5a5-9826cdb3e484) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568540] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (a845cc17-4d1f-4e43-855d-572b3b432137) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.568566] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (a845cc17-4d1f-4e43-855d-572b3b432137) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.577017] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (77e633d9-10fb-4663-849d-c7470f67a1f0) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.577054] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (77e633d9-10fb-4663-849d-c7470f67a1f0) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.577081] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (fe59a55c-5592-4898-a879-5d3dd9be1a9a) remote_fd is -1. EBADFD
[2013-02-25 19:44:43.577110] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (fe59a55c-5592-4898-a879-5d3dd9be1a9a) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.534877] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-17: server 172.16.5.3:24017 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:44:53.535046] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-3: server 172.16.5.3:24010 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:44:53.535105] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:48.770366 (xid=0x298464x)
[2013-02-25 19:44:53.535694] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.535771] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:48.828780 (xid=0x298466x)
[2013-02-25 19:44:53.535785] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.535987] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:51.537487 (xid=0x298498x)
[2013-02-25 19:44:53.536001] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.536052] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:51.747156 (xid=0x298501x)
[2013-02-25 19:44:53.536075] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.536132] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:51.790686 (xid=0x298504x)
[2013-02-25 19:44:53.536146] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.536192] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:44:43.717886 (xid=0x298604x)
[2013-02-25 19:44:53.536204] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.536259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:48.614291 (xid=0x298462x)
[2013-02-25 19:44:53.536274] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536372] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:48.618533 (xid=0x298463x)
[2013-02-25 19:44:53.536387] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536436] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:48.811049 (xid=0x298465x)
[2013-02-25 19:44:53.536448] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536493] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:48.829615 (xid=0x298467x)
[2013-02-25 19:44:53.536504] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536554] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:48.934115 (xid=0x298468x)
[2013-02-25 19:44:53.536579] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536625] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:49.312895 (xid=0x298469x)
[2013-02-25 19:44:53.536637] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536680] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:49.368297 (xid=0x298470x)
[2013-02-25 19:44:53.536691] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536733] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:49.442365 (xid=0x298471x)
[2013-02-25 19:44:53.536745] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536787] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:49.464161 (xid=0x298472x)
[2013-02-25 19:44:53.536798] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536840] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:49.582742 (xid=0x298473x)
[2013-02-25 19:44:53.536851] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.536893] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:43:49.628951 (xid=0x298474x)
[2013-02-25 19:44:53.536910] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.543197] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.281432 (xid=0x298597x)
[2013-02-25 19:44:53.543209] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.543247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:43:56.411115 (xid=0x298598x)
[2013-02-25 19:44:53.543257] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.543302] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:43:59.697714 (xid=0x298599x)
[2013-02-25 19:44:53.543316] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566028] I [socket.c:2315:socket_submit_request] 0-adata-client-17: not connected (priv->connected = 0)
[2013-02-25 19:44:53.566052] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-17: failed to submit rpc-request (XID: 0x298605x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-17)
[2013-02-25 19:44:53.566062] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566092] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:44:11.528951 (xid=0x298600x)
[2013-02-25 19:44:53.566105] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-17: timer must have expired
[2013-02-25 19:44:53.566129] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:44:43.633061 (xid=0x298601x)
[2013-02-25 19:44:53.566146] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566176] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:44:43.653948 (xid=0x298602x)
[2013-02-25 19:44:53.566186] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:44:43.665306 (xid=0x298603x)
[2013-02-25 19:44:53.566247] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566285] I [client.c:2090:client_rpc_notify] 0-adata-client-17: disconnected
[2013-02-25 19:44:53.566367] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:38.476455 (xid=0x298238x)
[2013-02-25 19:44:53.566378] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566422] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:41.247096 (xid=0x298274x)
[2013-02-25 19:44:53.566434] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566479] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:43.845617 (xid=0x298305x)
[2013-02-25 19:44:53.566491] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566534] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:45.933219 (xid=0x298354x)
[2013-02-25 19:44:53.566545] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566585] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:46.494402 (xid=0x298386x)
[2013-02-25 19:44:53.566597] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566637] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:47.321554 (xid=0x298394x)
[2013-02-25 19:44:53.566650] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566698] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:52.337389 (xid=0x298475x)
[2013-02-25 19:44:53.566721] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566768] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:55.805521 (xid=0x298556x)
[2013-02-25 19:44:53.566784] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566832] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:44:43.631615 (xid=0x298566x)
[2013-02-25 19:44:53.566847] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.566887] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.292601 (xid=0x298138x)
[2013-02-25 19:44:53.566900] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:53.567054] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.292980 (xid=0x298139x)
[2013-02-25 19:44:53.572662] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:53.572700] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 19:43:38.429664 (xid=0x298232x)
[2013-02-25 19:44:53.572712] W [client3_1-fops.c:418:client3_1_open_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (97156e3e-e623-4786-b33c-38413b77a8a3)
[2013-02-25 19:44:53.572761] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.440263 (xid=0x298233x)
[2013-02-25 19:44:53.572774] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (03093467-1e3d-4554-9f0d-7f56e29ff948)
[2013-02-25 19:44:53.572832] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:43:38.475218 (xid=0x298234x)
[2013-02-25 19:44:53.572850] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573235] I [socket.c:2315:socket_submit_request] 0-adata-client-3: not connected (priv->connected = 0)
[2013-02-25 19:44:53.573251] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-3: failed to submit rpc-request (XID: 0x298569x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-3)
[2013-02-25 19:44:53.573261] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573291] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:43:38.475246 (xid=0x298235x)
[2013-02-25 19:44:53.573301] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573332] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-3: failed to submit rpc-request (XID: 0x298570x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-3)
[2013-02-25 19:44:53.573344] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573368] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:43:38.475412 (xid=0x298236x)
[2013-02-25 19:44:53.573378] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573401] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-3: failed to submit rpc-request (XID: 0x298571x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-3)
[2013-02-25 19:44:53.573412] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573435] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:43:38.475443 (xid=0x298237x)
[2013-02-25 19:44:53.573445] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573468] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-3: failed to submit rpc-request (XID: 0x298572x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-3)
[2013-02-25 19:44:53.573479] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.573502] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:39.272264 (xid=0x298239x)
[2013-02-25 19:44:53.573521] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.584582] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:50.168653 (xid=0x298450x)
[2013-02-25 19:44:53.584593] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.590487] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.274167 (xid=0x298563x)
[2013-02-25 19:44:53.590498] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.590538] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.281154 (xid=0x298564x)
[2013-02-25 19:44:53.590549] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:53.590589] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:44:11.528962 (xid=0x298565x)
[2013-02-25 19:44:53.590600] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-3: timer must have expired
[2013-02-25 19:44:53.590627] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:44:43.632897 (xid=0x298567x)
[2013-02-25 19:44:53.590637] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.590659] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:44:43.988658 (xid=0x298568x)
[2013-02-25 19:44:53.590668] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:53.590702] I [client.c:2090:client_rpc_notify] 0-adata-client-3: disconnected
[2013-02-25 19:44:53.590927] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.590955] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594489] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (d3d2b36b-9f9c-4940-85b2-8e8f028c793f) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594538] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (d3d2b36b-9f9c-4940-85b2-8e8f028c793f) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594709] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (77cde4fe-900a-4bca-9820-f52c21e8b0a9) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594742] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (77cde4fe-900a-4bca-9820-f52c21e8b0a9) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594777] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (59914386-d621-4a84-899b-b906de455edc) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594802] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (59914386-d621-4a84-899b-b906de455edc) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594823] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (4c88ad1d-e185-4317-bd2f-1ff08c6edb6e) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594850] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (4c88ad1d-e185-4317-bd2f-1ff08c6edb6e) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594873] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (0174c42c-b3c3-40d6-95ff-0d59c939471d) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594895] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (0174c42c-b3c3-40d6-95ff-0d59c939471d) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594948] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (fe09089b-298b-4a7e-89d7-78a59b56b192) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.594978] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (fe09089b-298b-4a7e-89d7-78a59b56b192) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.595003] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (ac1bdf3c-4049-48eb-810d-5c8b97111b93) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.595027] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (ac1bdf3c-4049-48eb-810d-5c8b97111b93) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.595049] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (ef9ece03-6163-4cf7-b5d3-40f38dec346d) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.595074] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (ef9ece03-6163-4cf7-b5d3-40f38dec346d) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.595094] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (a1143267-6095-4d03-b13e-4199da1c4e8b) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.595132] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (a1143267-6095-4d03-b13e-4199da1c4e8b) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.599745] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (b4990331-39ab-4f22-9ce9-54c03df30774) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.599790] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (b4990331-39ab-4f22-9ce9-54c03df30774) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.599931] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (2586eece-6112-4b3f-933d-42d0a3c42830) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.599970] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (2586eece-6112-4b3f-933d-42d0a3c42830) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.600011] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (d991f95e-c35b-4227-a1ee-8e3d708e41ac) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.600041] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (d991f95e-c35b-4227-a1ee-8e3d708e41ac) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.600222] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (e4ecc455-7ed8-4e3a-92ae-6b3784495987) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.600257] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (e4ecc455-7ed8-4e3a-92ae-6b3784495987) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.627212] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:44:53.627272] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.536240] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-9: server 172.16.5.3:24013 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:44:54.536406] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:38.884285 (xid=0x290305x)
[2013-02-25 19:44:54.536424] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.536474] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:39.865542 (xid=0x290325x)
[2013-02-25 19:44:54.536487] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.536586] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:43.345368 (xid=0x290359x)
[2013-02-25 19:44:54.536599] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.536644] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:48.209535 (xid=0x290490x)
[2013-02-25 19:44:54.536669] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.536714] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:50.167852 (xid=0x290515x)
[2013-02-25 19:44:54.536727] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.536829] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:56.346299 (xid=0x290629x)
[2013-02-25 19:44:54.536843] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.536887] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.307532 (xid=0x290224x)
[2013-02-25 19:44:54.536902] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:54.536985] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.308037 (xid=0x290225x)
[2013-02-25 19:44:54.537000] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:54.537144] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.309387 (xid=0x290226x)
[2013-02-25 19:44:54.537160] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:54.537318] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.309823 (xid=0x290227x)
[2013-02-25 19:44:54.537339] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:54.537410] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.310209 (xid=0x290228x)
[2013-02-25 19:44:54.537431] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:44:54.537537] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:38.310590 (xid=0x290229x)
[2013-02-25 19:44:54.562184] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562223] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.634859 (xid=0x290615x)
[2013-02-25 19:44:54.562234] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562274] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.646025 (xid=0x290616x)
[2013-02-25 19:44:54.562303] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562350] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.679489 (xid=0x290617x)
[2013-02-25 19:44:54.562363] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562402] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.685916 (xid=0x290618x)
[2013-02-25 19:44:54.562413] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562452] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.731217 (xid=0x290619x)
[2013-02-25 19:44:54.562463] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562516] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.735381 (xid=0x290620x)
[2013-02-25 19:44:54.562530] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562572] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.841938 (xid=0x290621x)
[2013-02-25 19:44:54.562584] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562624] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:55.979886 (xid=0x290622x)
[2013-02-25 19:44:54.562635] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562675] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.029783 (xid=0x290623x)
[2013-02-25 19:44:54.562695] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562736] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.098196 (xid=0x290624x)
[2013-02-25 19:44:54.562748] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562788] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.125099 (xid=0x290625x)
[2013-02-25 19:44:54.562799] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562838] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.199635 (xid=0x290626x)
[2013-02-25 19:44:54.562850] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562889] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.274289 (xid=0x290627x)
[2013-02-25 19:44:54.562900] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562940] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:43:56.281270 (xid=0x290628x)
[2013-02-25 19:44:54.562951] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:44:54.562991] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:44:12.529137 (xid=0x290630x)
[2013-02-25 19:44:54.563002] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-9: timer must have expired
[2013-02-25 19:44:54.563029] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:44:43.600696 (xid=0x290631x)
[2013-02-25 19:44:54.563039] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.563086] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:44:43.632950 (xid=0x290632x)
[2013-02-25 19:44:54.563099] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.563123] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:44:53.603592 (xid=0x290633x)
[2013-02-25 19:44:54.563132] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.563172] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:53.618623 (xid=0x290634x)
[2013-02-25 19:44:54.563185] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 19:44:54.563299] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:53.687320 (xid=0x290635x)
[2013-02-25 19:44:54.563313] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (4c1dcc30-d294-46cb-be65-5565a88f1e5d)
[2013-02-25 19:44:54.563390] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:53.696719 (xid=0x290636x)
[2013-02-25 19:44:54.563403] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (4c1dcc30-d294-46cb-be65-5565a88f1e5d)
[2013-02-25 19:44:54.563468] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:44:53.710346 (xid=0x290637x)
[2013-02-25 19:44:54.563487] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:44:54.563531] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:53.871362 (xid=0x290638x)
[2013-02-25 19:44:54.563545] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (4c1dcc30-d294-46cb-be65-5565a88f1e5d)
[2013-02-25 19:44:54.563630] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:53.883391 (xid=0x290639x)
[2013-02-25 19:44:54.563644] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (4c1dcc30-d294-46cb-be65-5565a88f1e5d)
[2013-02-25 19:44:54.563702] I [client.c:2090:client_rpc_notify] 0-adata-client-9: disconnected
[2013-02-25 19:44:54.566800] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-9:  (60ae29e0-8677-4293-b82c-c396d588d1fb) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.566838] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-9:  (60ae29e0-8677-4293-b82c-c396d588d1fb) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.568865] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-9:  (89383e5e-137f-43ff-a2be-2746b99ddafc) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.568958] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-9:  (89383e5e-137f-43ff-a2be-2746b99ddafc) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569022] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-9:  (12150b53-3486-4cbd-a83e-9da9274f9b69) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569052] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-9:  (12150b53-3486-4cbd-a83e-9da9274f9b69) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569088] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-9:  (5ce6563a-cf0c-4883-ae25-874088e6123a) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569114] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-9:  (5ce6563a-cf0c-4883-ae25-874088e6123a) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569149] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-9:  (3aa7e80c-0163-43e7-bc55-36fefecb653c) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569207] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-9:  (3aa7e80c-0163-43e7-bc55-36fefecb653c) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569251] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-9:  (9eb1a9b1-8966-4737-b84d-b99392b98c64) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.569279] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-9:  (9eb1a9b1-8966-4737-b84d-b99392b98c64) remote_fd is -1. EBADFD
[2013-02-25 19:44:54.887889] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-17: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:44:54.905501] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-17: Connected to 172.16.5.3:24017, attached to remote volume '/var/gfs/sdj/adata-s003-9'.
[2013-02-25 19:44:54.905528] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-17: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:44:54.905556] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-17: 84 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:44:54.956362] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-3: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:44:54.956975] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-3: Connected to 172.16.5.3:24010, attached to remote volume '/var/gfs/sdc/adata-s003-2'.
[2013-02-25 19:44:54.957003] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:44:54.957015] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-3: 35 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:44:54.974151] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:44:54.975715] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-3: Server lk version = 1
[2013-02-25 19:44:55.008815] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-17: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:44:55.009406] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-17: Server lk version = 1
[2013-02-25 19:44:55.423638] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:55.535054] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:56.973935] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:57.415732] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:58.319001] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:58.904415] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:59.103569] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-8: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:44:59.816525] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:44:59.816554] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:44:59.816574] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:44:59.816583] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:44:59.816605] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:44:59.816615] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:45:00.537143] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-1: server 172.16.5.3:24009 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:45:00.537819] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:39.287640 (xid=0x372978x)
[2013-02-25 19:45:00.537849] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:45:00.537915] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:43:52.733493 (xid=0x373222x)
[2013-02-25 19:45:00.537930] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:45:00.537980] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:44:43.636876 (xid=0x373358x)
[2013-02-25 19:45:00.537992] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:45:00.538045] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:44:43.978824 (xid=0x373454x)
[2013-02-25 19:45:01.015957] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:45:01.016652] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:58.740914 (xid=0x373860x)
[2013-02-25 19:45:01.016684] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:45:01.016766] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x374423x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 19:45:01.016786] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:45:01.017413] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:44:59.370510 (xid=0x373861x)
[2013-02-25 19:45:01.017434] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:45:01.017505] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x374424x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 19:45:01.017552] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:45:01.018173] I [client.c:2090:client_rpc_notify] 0-adata-client-1: disconnected
[2013-02-25 19:45:01.039201] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-1:  (e57b53c7-f51c-4d9a-87c3-5628c62392bf) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.039294] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-1:  (e57b53c7-f51c-4d9a-87c3-5628c62392bf) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.301622] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-1:  (05331ce3-63d1-416e-84e8-3b7f0c0b7cd6) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.301673] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-1:  (05331ce3-63d1-416e-84e8-3b7f0c0b7cd6) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.302272] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-1:  (8ca70a35-32e3-42ef-8305-8db99f73c00c) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.302377] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-1:  (8ca70a35-32e3-42ef-8305-8db99f73c00c) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.302671] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-1:  (65efd052-c74c-4cef-9dbe-4dbbe2ebce26) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.302703] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-1:  (65efd052-c74c-4cef-9dbe-4dbbe2ebce26) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.303021] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-1:  (eda8fbe9-3de5-496f-b331-c114e1f61c49) remote_fd is -1. EBADFD
[2013-02-25 19:45:01.303134] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-1:  (eda8fbe9-3de5-496f-b331-c114e1f61c49) remote_fd is -1. EBADFD
[2013-02-25 19:45:04.858203] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:45:04.858235] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:45:04.858763] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:45:04.858779] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:45:04.859347] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:45:04.859363] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.321898] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.321947] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.322027] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.322041] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.322084] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.322097] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.322122] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.322135] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.322236] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.322258] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.322911] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.322933] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.322972] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323008] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.323040] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323052] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.323070] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323101] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.323122] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323131] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.323147] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323156] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.323172] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323180] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:00.323201] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:46:00.323215] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:46:08.542863] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-5: server 172.16.5.3:24011 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:46:08.543453] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:04.849367 (xid=0x302082x)
[2013-02-25 19:46:08.543535] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:08.543693] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:07.595938 (xid=0x302313x)
[2013-02-25 19:46:08.543716] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:08.543771] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:09.235849 (xid=0x302321x)
[2013-02-25 19:46:08.543787] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:08.543929] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:09.396035 (xid=0x302325x)
[2013-02-25 19:46:08.543955] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:08.585356] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585409] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:46:01.156936 (xid=0x302451x)
[2013-02-25 19:46:08.585420] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:08.585442] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:01.639083 (xid=0x302452x)
[2013-02-25 19:46:08.585453] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585498] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:02.432675 (xid=0x302453x)
[2013-02-25 19:46:08.585510] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585553] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:03.474872 (xid=0x302454x)
[2013-02-25 19:46:08.585565] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585608] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:03.486259 (xid=0x302455x)
[2013-02-25 19:46:08.585620] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585663] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:05.428967 (xid=0x302456x)
[2013-02-25 19:46:08.585675] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585719] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:06.708308 (xid=0x302457x)
[2013-02-25 19:46:08.585736] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585782] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:06.847112 (xid=0x302458x)
[2013-02-25 19:46:08.585794] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:08.585826] I [client.c:2090:client_rpc_notify] 0-adata-client-5: disconnected
[2013-02-25 19:46:08.586345] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (36c7c8bc-52e6-413e-a4b3-7d9d47c0d3de) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.586379] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (36c7c8bc-52e6-413e-a4b3-7d9d47c0d3de) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589073] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (f57e7896-1847-4397-a652-9129d25010cb) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589128] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (f57e7896-1847-4397-a652-9129d25010cb) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589170] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (fdaefe7d-b2c2-4769-858e-944d2cc9c568) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589211] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (fdaefe7d-b2c2-4769-858e-944d2cc9c568) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589278] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (613ce112-885f-47e9-9a92-172afc212777) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589355] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (613ce112-885f-47e9-9a92-172afc212777) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589437] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (ee69ff98-53b1-40ca-afa9-03f41f071326) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589485] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (ee69ff98-53b1-40ca-afa9-03f41f071326) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589534] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (66d45cf0-5397-4bb1-97da-0d1a8d7095bd) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589580] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (66d45cf0-5397-4bb1-97da-0d1a8d7095bd) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589621] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (486b08db-d856-4390-8e4e-14f75c341924) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589665] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (486b08db-d856-4390-8e4e-14f75c341924) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589691] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (5e73ce83-7282-4556-b010-1f6f78619485) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589745] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (5e73ce83-7282-4556-b010-1f6f78619485) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589784] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (13e5a53d-9c98-4b1b-aa8d-c72b9480f7b4) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.589836] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (13e5a53d-9c98-4b1b-aa8d-c72b9480f7b4) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.610138] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (36c7c8bc-52e6-413e-a4b3-7d9d47c0d3de) remote_fd is -1. EBADFD
[2013-02-25 19:46:08.610185] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (36c7c8bc-52e6-413e-a4b3-7d9d47c0d3de) remote_fd is -1. EBADFD
[2013-02-25 19:46:42.286573] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (5453ea55-4d66-4b80-affa-808f067d8991) remote_fd is -1. EBADFD
[2013-02-25 19:46:42.286647] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (5453ea55-4d66-4b80-affa-808f067d8991) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.546344] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-19: server 172.16.5.3:24018 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:46:50.546512] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:42.942561 (xid=0x302870x)
[2013-02-25 19:46:50.546528] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.546578] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:45.326016 (xid=0x302872x)
[2013-02-25 19:46:50.546590] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.546633] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:54.325420 (xid=0x302887x)
[2013-02-25 19:46:50.546646] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.546700] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:03.485436 (xid=0x302911x)
[2013-02-25 19:46:50.546714] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.546779] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:17.977450 (xid=0x302934x)
[2013-02-25 19:46:50.546794] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.546845] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:19.834107 (xid=0x302819x)
[2013-02-25 19:46:50.546861] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.546937] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:45:20.454983 (xid=0x302820x)
[2013-02-25 19:46:50.546961] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.547007] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:20.553868 (xid=0x302821x)
[2013-02-25 19:46:50.547021] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547086] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:20.838039 (xid=0x302822x)
[2013-02-25 19:46:50.547099] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547147] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:21.456354 (xid=0x302823x)
[2013-02-25 19:46:50.547159] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:21.826572 (xid=0x302824x)
[2013-02-25 19:46:50.547267] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547358] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:45:22.171466 (xid=0x302825x)
[2013-02-25 19:46:50.547376] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.547454] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:22.414677 (xid=0x302826x)
[2013-02-25 19:46:50.547480] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547575] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:22.454948 (xid=0x302827x)
[2013-02-25 19:46:50.547608] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547704] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:23.869394 (xid=0x302828x)
[2013-02-25 19:46:50.547723] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547811] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:23.885799 (xid=0x302829x)
[2013-02-25 19:46:50.547827] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.547917] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:24.074418 (xid=0x302830x)
[2013-02-25 19:46:50.547950] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548026] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:24.361480 (xid=0x302831x)
[2013-02-25 19:46:50.548058] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548147] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:25.463109 (xid=0x302832x)
[2013-02-25 19:46:50.548165] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548252] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:25.560302 (xid=0x302833x)
[2013-02-25 19:46:50.548277] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548366] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:25.597482 (xid=0x302834x)
[2013-02-25 19:46:50.548398] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548472] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:26.198941 (xid=0x302835x)
[2013-02-25 19:46:50.548506] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548594] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:26.330503 (xid=0x302836x)
[2013-02-25 19:46:50.548612] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548702] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:45:26.539566 (xid=0x302837x)
[2013-02-25 19:46:50.548718] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-19: timer must have expired
[2013-02-25 19:46:50.548757] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:26.903737 (xid=0x302838x)
[2013-02-25 19:46:50.548773] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548864] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:27.088069 (xid=0x302839x)
[2013-02-25 19:46:50.548897] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.548977] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:45:27.129852 (xid=0x302840x)
[2013-02-25 19:46:50.549007] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:46:50.549069] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:28.166886 (xid=0x302841x)
[2013-02-25 19:46:50.549104] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.549199] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:29.178968 (xid=0x302842x)
[2013-02-25 19:46:50.556735] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.556781] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:42.512432 (xid=0x302964x)
[2013-02-25 19:46:50.556800] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.556848] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:43.706365 (xid=0x302965x)
[2013-02-25 19:46:50.556860] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.556906] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:44.820018 (xid=0x302966x)
[2013-02-25 19:46:50.556917] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.556963] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:46.243231 (xid=0x302967x)
[2013-02-25 19:46:50.556974] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.557020] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:48.632351 (xid=0x302968x)
[2013-02-25 19:46:50.557031] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.557076] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:49.696363 (xid=0x302969x)
[2013-02-25 19:46:50.557087] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:46:50.557121] I [client.c:2090:client_rpc_notify] 0-adata-client-19: disconnected
[2013-02-25 19:46:50.558545] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-19:  (71e61f6f-e3a8-4c03-8994-703bee2a5649) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.558581] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-19:  (71e61f6f-e3a8-4c03-8994-703bee2a5649) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.560346] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-19:  (d3e19b10-0d9a-499d-b555-69312a650990) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.560396] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-19:  (d3e19b10-0d9a-499d-b555-69312a650990) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.560450] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-19:  (346b33ca-ce3b-47f6-86f0-0df1ecca0f93) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.560494] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-19:  (346b33ca-ce3b-47f6-86f0-0df1ecca0f93) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.560519] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-19:  (251b2b8b-c58a-4a54-8f02-eab3bcf988c8) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.560566] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-19:  (251b2b8b-c58a-4a54-8f02-eab3bcf988c8) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.565486] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-19:  (71e61f6f-e3a8-4c03-8994-703bee2a5649) remote_fd is -1. EBADFD
[2013-02-25 19:46:50.565519] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-19:  (71e61f6f-e3a8-4c03-8994-703bee2a5649) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.547808] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-17: server 172.16.5.3:24017 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:47:01.548449] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:04.613452 (xid=0x299639x)
[2013-02-25 19:47:01.548467] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548476] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.548485] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.548517] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:04.613470 (xid=0x299640x)
[2013-02-25 19:47:01.548527] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548570] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:05.301395 (xid=0x300098x)
[2013-02-25 19:47:01.548581] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548621] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:05.549248 (xid=0x300099x)
[2013-02-25 19:47:01.548633] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548640] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.548648] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.548692] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:07.064929 (xid=0x300101x)
[2013-02-25 19:47:01.548703] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548754] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:08.595736 (xid=0x300108x)
[2013-02-25 19:47:01.548767] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548830] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:08.982739 (xid=0x300110x)
[2013-02-25 19:47:01.548843] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548882] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:10.290441 (xid=0x300119x)
[2013-02-25 19:47:01.548893] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548901] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.548922] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.548960] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:10.293147 (xid=0x300121x)
[2013-02-25 19:47:01.548980] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.548989] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.548996] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549038] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:11.681574 (xid=0x300129x)
[2013-02-25 19:47:01.549057] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549065] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549072] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549100] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:11.681592 (xid=0x300130x)
[2013-02-25 19:47:01.549109] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549152] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:12.719141 (xid=0x300136x)
[2013-02-25 19:47:01.549165] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549172] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549180] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:15.854435 (xid=0x300148x)
[2013-02-25 19:47:01.549339] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549349] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549357] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549402] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:18.089418 (xid=0x300155x)
[2013-02-25 19:47:01.549415] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549423] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549430] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549458] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:21.414683 (xid=0x300162x)
[2013-02-25 19:47:01.549473] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549481] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549488] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549513] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:21.454966 (xid=0x300163x)
[2013-02-25 19:47:01.549523] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549563] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:21.825433 (xid=0x300165x)
[2013-02-25 19:47:01.549579] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549587] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549595] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549620] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:21.825449 (xid=0x300166x)
[2013-02-25 19:47:01.549629] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549669] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:26.895600 (xid=0x300179x)
[2013-02-25 19:47:01.549680] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549687] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549696] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549737] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:26.902397 (xid=0x300180x)
[2013-02-25 19:47:01.549750] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549798] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:29.016593 (xid=0x300184x)
[2013-02-25 19:47:01.549810] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549818] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549840] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549868] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:41.329434 (xid=0x300210x)
[2013-02-25 19:47:01.549878] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549885] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549892] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.549916] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:41.329456 (xid=0x300211x)
[2013-02-25 19:47:01.549925] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549965] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:42.671290 (xid=0x300214x)
[2013-02-25 19:47:01.549976] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.549983] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.549994] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550020] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:42.671306 (xid=0x300215x)
[2013-02-25 19:47:01.550030] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550079] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:52.197509 (xid=0x300226x)
[2013-02-25 19:47:01.550098] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550141] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:53.301389 (xid=0x300230x)
[2013-02-25 19:47:01.550152] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550191] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:53.495781 (xid=0x300232x)
[2013-02-25 19:47:01.550203] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550210] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550218] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550247] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:53.502859 (xid=0x300233x)
[2013-02-25 19:47:01.550256] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550306] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:53.756873 (xid=0x300235x)
[2013-02-25 19:47:01.550318] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550326] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550346] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550374] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:54.317481 (xid=0x300236x)
[2013-02-25 19:47:01.550384] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550401] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550415] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550441] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:45:54.317498 (xid=0x300237x)
[2013-02-25 19:47:01.550451] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550492] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:45:59.220550 (xid=0x300243x)
[2013-02-25 19:47:01.550505] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550513] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550520] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550544] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:02.431777 (xid=0x300265x)
[2013-02-25 19:47:01.550553] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550560] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550567] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550593] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:02.431795 (xid=0x300266x)
[2013-02-25 19:47:01.550603] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550644] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:03.473820 (xid=0x300268x)
[2013-02-25 19:47:01.550656] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550663] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550672] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550737] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:03.473838 (xid=0x300269x)
[2013-02-25 19:47:01.550748] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550789] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:05.428053 (xid=0x300272x)
[2013-02-25 19:47:01.550800] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550808] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550818] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550851] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:05.428071 (xid=0x300273x)
[2013-02-25 19:47:01.550860] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550899] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:06.846197 (xid=0x300276x)
[2013-02-25 19:47:01.550910] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.550917] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.550925] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.550960] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:06.846217 (xid=0x300277x)
[2013-02-25 19:47:01.550969] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551020] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:11.236518 (xid=0x300285x)
[2013-02-25 19:47:01.551032] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551047] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551056] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551086] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:11.236537 (xid=0x300286x)
[2013-02-25 19:47:01.551096] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551135] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:14.170986 (xid=0x300294x)
[2013-02-25 19:47:01.551146] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551153] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551161] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551189] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:14.689508 (xid=0x300296x)
[2013-02-25 19:47:01.551199] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551206] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551228] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:14.689528 (xid=0x300297x)
[2013-02-25 19:47:01.551264] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551314] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:22.479643 (xid=0x300305x)
[2013-02-25 19:47:01.551327] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551334] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551348] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551373] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:25.736648 (xid=0x300311x)
[2013-02-25 19:47:01.551383] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551422] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:28.805574 (xid=0x300317x)
[2013-02-25 19:47:01.551433] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551440] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551450] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551477] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:28.805593 (xid=0x300318x)
[2013-02-25 19:47:01.551486] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551535] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:29.401392 (xid=0x300320x)
[2013-02-25 19:47:01.551547] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551597] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:29.784210 (xid=0x300322x)
[2013-02-25 19:47:01.551609] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551650] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:42.362510 (xid=0x300335x)
[2013-02-25 19:47:01.551662] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551669] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551684] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551726] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:42.362528 (xid=0x300336x)
[2013-02-25 19:47:01.551737] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551790] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:50.503267 (xid=0x300345x)
[2013-02-25 19:47:01.551804] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551811] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551819] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551846] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:46:59.269439 (xid=0x300378x)
[2013-02-25 19:47:01.551855] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551862] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551868] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.551894] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:46:59.269457 (xid=0x300379x)
[2013-02-25 19:47:01.551903] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551943] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:47:01.505431 (xid=0x300383x)
[2013-02-25 19:47:01.551954] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.551961] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:01.551971] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:01.552015] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:47:01.505449 (xid=0x300384x)
[2013-02-25 19:47:01.552028] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:47:01.552068] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:45:04.409303 (xid=0x299476x)
[2013-02-25 19:47:01.675571] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:50.905659 (xid=0x300364x)
[2013-02-25 19:47:01.675587] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.675696] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:50.925725 (xid=0x300365x)
[2013-02-25 19:47:01.675726] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.675820] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:50.969652 (xid=0x300366x)
[2013-02-25 19:47:01.675835] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.675953] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:50.993058 (xid=0x300367x)
[2013-02-25 19:47:01.675977] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.676114] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:50.994189 (xid=0x300368x)
[2013-02-25 19:47:01.676130] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.676369] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:51.012363 (xid=0x300369x)
[2013-02-25 19:47:01.676397] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.676505] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:51.036951 (xid=0x300370x)
[2013-02-25 19:47:01.676521] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.676632] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:51.083914 (xid=0x300371x)
[2013-02-25 19:47:01.676648] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.676766] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:51.123644 (xid=0x300372x)
[2013-02-25 19:47:01.676793] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 19:47:01.676890] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:51.124212 (xid=0x300373x)
[2013-02-25 19:47:01.676941] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677086] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:52.251779 (xid=0x300374x)
[2013-02-25 19:47:01.677121] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677291] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-17: failed to submit rpc-request (XID: 0x300415x Program: GlusterFS 3.1, ProgVers: 330, Proc: 11) to rpc-transport (adata-client-17)
[2013-02-25 19:47:01.677294] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:52.388061 (xid=0x300375x)
[2013-02-25 19:47:01.677319] W [client3_1-fops.c:418:client3_1_open_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (08b253f5-b9bf-404c-8b40-e2e9be443029)
[2013-02-25 19:47:01.677348] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677459] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:53.407431 (xid=0x300376x)
[2013-02-25 19:47:01.677477] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677605] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:54.807026 (xid=0x300377x)
[2013-02-25 19:47:01.677624] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677727] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:46:59.271042 (xid=0x300380x)
[2013-02-25 19:47:01.677744] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677839] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:47:00.117704 (xid=0x300381x)
[2013-02-25 19:47:01.677864] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.677960] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:47:01.287063 (xid=0x300382x)
[2013-02-25 19:47:01.677998] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.678186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:47:01.506894 (xid=0x300385x)
[2013-02-25 19:47:01.678204] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:47:01.678302] I [client.c:2090:client_rpc_notify] 0-adata-client-17: disconnected
[2013-02-25 19:47:01.680365] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.680421] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.739127] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (1ddfc38e-9ada-4478-b208-27699b019bd7) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.739175] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (1ddfc38e-9ada-4478-b208-27699b019bd7) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.739658] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (7740cdb8-14a9-4aea-8c4c-687654cbbb5e) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.739747] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (7740cdb8-14a9-4aea-8c4c-687654cbbb5e) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.741010] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (14ce2bec-7614-43fc-8c75-a5f55e9bfe5e) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.741051] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (14ce2bec-7614-43fc-8c75-a5f55e9bfe5e) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.741348] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (c051069d-cb30-4820-8949-0425e55d8693) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.741392] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (c051069d-cb30-4820-8949-0425e55d8693) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.741667] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (3439aaa3-6076-4e67-a71d-6ae2ed665c0a) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.741815] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (3439aaa3-6076-4e67-a71d-6ae2ed665c0a) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.742391] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (2c315734-ae8a-4623-b73a-d5f57eb7beed) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.742424] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (2c315734-ae8a-4623-b73a-d5f57eb7beed) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.742944] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (823a95ae-2e12-4d63-bc5c-740216a2a819) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.742982] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (823a95ae-2e12-4d63-bc5c-740216a2a819) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.743204] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (96ca89a4-8d3b-4412-8d92-8949a29c7655) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.743247] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (96ca89a4-8d3b-4412-8d92-8949a29c7655) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.743470] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (9dd30ba6-e5fe-44f5-b8a9-2585b0dbf740) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.743503] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (9dd30ba6-e5fe-44f5-b8a9-2585b0dbf740) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.993098] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:01.993217] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.389803] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.389847] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.701873] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.701957] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.872901] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.872942] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.964953] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.965000] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.976007] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.976043] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.984515] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.984576] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.992030] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.992075] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.997345] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:02.997403] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.005547] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.005591] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.011741] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.011779] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.018787] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.018836] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.024099] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.024145] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.028866] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.028899] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.036707] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.036744] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.040265] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.040333] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.055641] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.055681] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.056887] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:03.056932] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-17:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:47:05.017470] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:47:05.017497] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:08.160561] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:08.160590] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:10.316219] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:47:10.316266] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:17.296545] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:17.296575] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:24.785676] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:24.785719] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:28.483220] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:28.483258] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:29.759447] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:29.759485] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:34.855152] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:34.855186] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:44.114127] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:44.114151] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:55.721781] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:47:55.721809] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:47:59.723426] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-17: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:47:59.723935] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-17: Connected to 172.16.5.3:24017, attached to remote volume '/var/gfs/sdj/adata-s003-9'.
[2013-02-25 19:47:59.723959] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-17: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:47:59.723988] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-17: 100 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:47:59.732509] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-5: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:47:59.733327] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-5: Connected to 172.16.5.3:24011, attached to remote volume '/var/gfs/sdd/adata-s003-3'.
[2013-02-25 19:47:59.733341] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-5: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:47:59.733351] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-5: 24 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:47:59.734096] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-7: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:47:59.734478] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-17: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:47:59.734706] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-7: Connected to 172.16.5.3:24012, attached to remote volume '/var/gfs/sde/adata-s003-4'.
[2013-02-25 19:47:59.734719] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-7: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:47:59.734730] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-7: 37 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:47:59.740939] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-17: Server lk version = 1
[2013-02-25 19:47:59.752935] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-9: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:47:59.753335] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-9: Connected to 172.16.5.3:24013, attached to remote volume '/var/gfs/sdf/adata-s003-5'.
[2013-02-25 19:47:59.753348] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:47:59.753370] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-9: 22 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:47:59.755683] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-19: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:47:59.755976] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-19: Connected to 172.16.5.3:24018, attached to remote volume '/var/gfs/sdk/adata-s003-10'.
[2013-02-25 19:47:59.755999] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-19: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:47:59.756014] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-19: 20 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:47:59.757181] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:47:59.759043] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-9: Server lk version = 1
[2013-02-25 19:47:59.774738] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-19: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:47:59.775641] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-19: Server lk version = 1
[2013-02-25 19:47:59.802407] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-5: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:47:59.803703] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-5: Server lk version = 1
[2013-02-25 19:47:59.809854] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-7: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:47:59.812972] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-7: Server lk version = 1
[2013-02-25 19:47:59.849432] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-1: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:47:59.849893] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-1: Connected to 172.16.5.3:24009, attached to remote volume '/var/gfs/sdb/adata-s003-1'.
[2013-02-25 19:47:59.849916] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:47:59.849933] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-1: 36 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:47:59.855318] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:47:59.858945] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-1: Server lk version = 1
[2013-02-25 19:48:00.042328] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc009218
[2013-02-25 19:48:00.314863] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:48:00.342337] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:48:00.342372] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:48:00.912954] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0127b4
[2013-02-25 19:48:00.913194] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:48:00.916063] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-0: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:48:01.829204] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:48:01.867540] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-0: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:48:03.116911] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-0: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:48:05.748693] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00cc4c
[2013-02-25 19:48:06.217814] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc00bf04
[2013-02-25 19:48:06.218181] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:48:06.230754] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:48:09.209778] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00878c
[2013-02-25 19:48:23.762690] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc01287c
[2013-02-25 19:48:25.100019] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc012818
[2013-02-25 19:48:26.749558] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc00b8c4
[2013-02-25 19:48:38.247498] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc003c28
[2013-02-25 19:48:51.560722] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-3: server 172.16.5.3:24010 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:48:51.560948] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:48:21.759621 (xid=0x301133x)
[2013-02-25 19:48:51.560966] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561015] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:48:22.837196 (xid=0x301134x)
[2013-02-25 19:48:51.561027] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561071] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:48:28.896161 (xid=0x301135x)
[2013-02-25 19:48:51.561087] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561142] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:01.114752 (xid=0x301117x)
[2013-02-25 19:48:51.561158] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561287] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:01.348749 (xid=0x301118x)
[2013-02-25 19:48:51.561304] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561355] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:01.944059 (xid=0x301119x)
[2013-02-25 19:48:51.561366] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561389] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:02.348805 (xid=0x301120x)
[2013-02-25 19:48:51.561399] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561447] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:04.688476 (xid=0x301121x)
[2013-02-25 19:48:51.561458] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561504] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:04.801741 (xid=0x301122x)
[2013-02-25 19:48:51.561516] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561561] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:05.070382 (xid=0x301123x)
[2013-02-25 19:48:51.561572] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561618] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:05.340747 (xid=0x301124x)
[2013-02-25 19:48:51.561636] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561682] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:07.897061 (xid=0x301125x)
[2013-02-25 19:48:51.561694] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561739] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:08.647711 (xid=0x301126x)
[2013-02-25 19:48:51.561750] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:51.561794] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:09.349568 (xid=0x301127x)
[2013-02-25 19:48:51.561803] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561825] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:09.416073 (xid=0x301128x)
[2013-02-25 19:48:51.561834] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561855] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:09.476289 (xid=0x301129x)
[2013-02-25 19:48:51.561863] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561884] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:48:09.556062 (xid=0x301130x)
[2013-02-25 19:48:51.561893] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-3: timer must have expired
[2013-02-25 19:48:51.561912] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:10.034734 (xid=0x301131x)
[2013-02-25 19:48:51.561927] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561948] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:10.105545 (xid=0x301132x)
[2013-02-25 19:48:51.561957] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.561978] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:48:46.516091 (xid=0x301136x)
[2013-02-25 19:48:51.561987] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:51.562017] I [client.c:2090:client_rpc_notify] 0-adata-client-3: disconnected
[2013-02-25 19:48:51.562214] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (a6478483-30ab-4a28-99e0-a5b88906539c) remote_fd is -1. EBADFD
[2013-02-25 19:48:51.562252] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (a6478483-30ab-4a28-99e0-a5b88906539c) remote_fd is -1. EBADFD
[2013-02-25 19:48:51.562279] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (c826cde9-6fca-4c33-857b-31375fe8597d) remote_fd is -1. EBADFD
[2013-02-25 19:48:51.562304] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (c826cde9-6fca-4c33-857b-31375fe8597d) remote_fd is -1. EBADFD
[2013-02-25 19:48:51.562322] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-3:  (f3981a67-e242-494a-ae87-ecb2130ea3cd) remote_fd is -1. EBADFD
[2013-02-25 19:48:51.562346] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-3:  (f3981a67-e242-494a-ae87-ecb2130ea3cd) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.561900] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-21: server 172.16.5.3:24019 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:48:59.562039] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:47:26.847626 (xid=0x377658x)
[2013-02-25 19:48:59.562064] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.593183] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593228] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.269728 (xid=0x377852x)
[2013-02-25 19:48:59.593241] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593280] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.270482 (xid=0x377853x)
[2013-02-25 19:48:59.593292] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593330] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.271247 (xid=0x377854x)
[2013-02-25 19:48:59.593341] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593380] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.272024 (xid=0x377855x)
[2013-02-25 19:48:59.593391] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593430] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.272865 (xid=0x377856x)
[2013-02-25 19:48:59.593441] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593480] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.273725 (xid=0x377857x)
[2013-02-25 19:48:59.593492] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593539] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.274490 (xid=0x377858x)
[2013-02-25 19:48:59.593551] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593589] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.275268 (xid=0x377859x)
[2013-02-25 19:48:59.593600] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593639] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.276013 (xid=0x377860x)
[2013-02-25 19:48:59.593650] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593689] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.276760 (xid=0x377861x)
[2013-02-25 19:48:59.593701] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593739] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.277601 (xid=0x377862x)
[2013-02-25 19:48:59.593751] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593790] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.278466 (xid=0x377863x)
[2013-02-25 19:48:59.593801] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593840] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.279233 (xid=0x377864x)
[2013-02-25 19:48:59.593858] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593897] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:00.280006 (xid=0x377865x)
[2013-02-25 19:48:59.593909] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:48:59.593947] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:48:00.432125 (xid=0x377866x)
[2013-02-25 19:48:59.593958] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.593987] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x377891x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-21)
[2013-02-25 19:48:59.593999] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594022] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:48:00.434644 (xid=0x377867x)
[2013-02-25 19:48:59.594032] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594056] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x377892x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-21)
[2013-02-25 19:48:59.594066] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594089] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:48:00.434881 (xid=0x377868x)
[2013-02-25 19:48:59.594099] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594122] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x377893x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-21)
[2013-02-25 19:48:59.594133] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594156] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:01.115286 (xid=0x377869x)
[2013-02-25 19:48:59.594174] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594222] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:01.349384 (xid=0x377870x)
[2013-02-25 19:48:59.594235] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594278] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:01.944407 (xid=0x377871x)
[2013-02-25 19:48:59.594289] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594312] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:02.349395 (xid=0x377873x)
[2013-02-25 19:48:59.594323] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594367] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:04.689064 (xid=0x377874x)
[2013-02-25 19:48:59.594379] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594423] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:04.802321 (xid=0x377875x)
[2013-02-25 19:48:59.594435] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594478] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:05.070893 (xid=0x377876x)
[2013-02-25 19:48:59.594490] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594533] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:05.341287 (xid=0x377877x)
[2013-02-25 19:48:59.594553] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594597] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:07.897676 (xid=0x377878x)
[2013-02-25 19:48:59.594609] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594653] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:48:08.648301 (xid=0x377879x)
[2013-02-25 19:48:59.594664] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:48:59.594707] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:09.349753 (xid=0x377880x)
[2013-02-25 19:48:59.594717] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594740] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:09.416233 (xid=0x377881x)
[2013-02-25 19:48:59.594749] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594771] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:09.476452 (xid=0x377882x)
[2013-02-25 19:48:59.594780] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594801] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:10.034912 (xid=0x377883x)
[2013-02-25 19:48:59.594810] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594832] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:48:10.105710 (xid=0x377884x)
[2013-02-25 19:48:59.594847] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:48:59.594861] I [client.c:2090:client_rpc_notify] 0-adata-client-21: disconnected
[2013-02-25 19:48:59.601445] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601485] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601679] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (6c4980e1-5e5f-406e-9ed3-fe5f92fcb9a0) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601712] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (6c4980e1-5e5f-406e-9ed3-fe5f92fcb9a0) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601734] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (788f9a7d-8996-4298-bd43-b6d40843dbb2) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601761] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (788f9a7d-8996-4298-bd43-b6d40843dbb2) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601781] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (7ca6727d-ac57-4aec-acd8-abdcf2bfd737) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601807] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (7ca6727d-ac57-4aec-acd8-abdcf2bfd737) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601827] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (253f02b1-1f03-4a22-8a17-f08a96211dbb) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601852] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (253f02b1-1f03-4a22-8a17-f08a96211dbb) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601932] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (676aeee6-ef3d-46e3-b67f-0556afde139d) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601961] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (676aeee6-ef3d-46e3-b67f-0556afde139d) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.601982] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (c0810890-082d-423f-856f-9ca8149b180f) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.602008] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (c0810890-082d-423f-856f-9ca8149b180f) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.603595] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.603627] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.604437] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.604487] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.604928] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.604989] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.605396] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.605428] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-21:  (f5ac35d8-92c9-413e-8a71-3f3c906e0d70) remote_fd is -1. EBADFD
[2013-02-25 19:48:59.874704] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:48:59.946791] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-2: Non blocking entrylks failed.
[2013-02-25 19:48:59.946822] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-2: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:48:59.970879] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-2: Non blocking entrylks failed.
[2013-02-25 19:48:59.970911] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-2: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:00.127281] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:49:00.165275] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:49:00.271839] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:49:00.271870] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:00.271927] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:49:00.271940] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:00.272039] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:49:00.272062] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:00.272125] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:49:00.272138] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:00.345171] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-21: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:49:00.346206] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-21: Connected to 172.16.5.3:24019, attached to remote volume '/var/gfs/sdl/adata-s003-11'.
[2013-02-25 19:49:00.346231] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-21: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:49:00.346248] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-21: 34 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:49:00.650078] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-21: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:49:00.800728] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-21: Server lk version = 1
[2013-02-25 19:49:01.073476] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:49:01.076178] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:49:01.352763] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:49:01.352825] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.352873] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:49:01.352884] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.352979] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:49:01.352995] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.353066] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:49:01.353079] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.353157] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:49:01.353172] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.461200] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.461220] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.465975] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.466003] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.466381] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.466407] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.466704] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.466727] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.467543] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.467569] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.467900] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.467924] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.468282] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.468307] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.468609] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:49:01.469334] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.469359] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.470345] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.470370] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.470697] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.470722] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.471411] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:49:01.471435] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:01.563434] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-3: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:49:01.567008] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-3: Connected to 172.16.5.3:24010, attached to remote volume '/var/gfs/sdc/adata-s003-2'.
[2013-02-25 19:49:01.567023] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:49:01.567032] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-3: 30 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:49:01.575081] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:49:01.575649] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-3: Server lk version = 1
[2013-02-25 19:49:02.132280] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:49:02.132332] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:02.178439] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-10: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:49:03.644411] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:49:03.644461] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:03.644697] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:49:03.644714] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:03.644922] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:49:03.644962] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:03.645091] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:49:03.645106] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:08.422518] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc008d68
[2013-02-25 19:49:08.443672] W [client-lk.c:370:delete_granted_locks_owner] 0-adata-client-5: fdctx not valid
[2013-02-25 19:49:08.443746] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:49:08.444045] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:49:09.606908] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:49:09.606935] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:11.692051] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:49:11.692097] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:17.602789] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:49:17.602819] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:17.613689] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-10: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:49:17.756291] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00a3ac
[2013-02-25 19:49:17.780410] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:49:18.350363] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:49:18.350416] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:49:54.245867] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00bf68
[2013-02-25 19:49:54.246355] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:49:54.667047] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc010cc0
[2013-02-25 19:49:54.667428] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:50:13.825834] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:50:13.826749] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0075f8
[2013-02-25 19:50:13.826961] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:50:13.880160] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00b798
[2013-02-25 19:50:13.881058] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:50:13.885350] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00a988
[2013-02-25 19:50:13.885813] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:50:13.886358] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00a474
[2013-02-25 19:50:13.886905] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:50:13.962097] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00d930
[2013-02-25 19:50:13.962649] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:50:14.204481] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:50:14.204519] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.284566] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:50:14.284615] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.288067] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:50:14.288097] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.290484] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:50:14.290501] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.300322] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:50:14.300353] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.919658] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:50:14.919685] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  meta-data data entry missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.920188] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:50:14.920207] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:14.934855] W [client3_1-fops.c:418:client3_1_open_cbk] 0-adata-client-9: remote operation failed: No such file or directory. Path: /files/random-file.dat (ba539de4-b0cd-46f0-a50c-e93bef9eb532)
[2013-02-25 19:50:14.935138] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00fe4c
[2013-02-25 19:50:14.935529] W [client3_1-fops.c:418:client3_1_open_cbk] 0-adata-client-9: remote operation failed: No such file or directory. Path: /files/random-file.dat (ba539de4-b0cd-46f0-a50c-e93bef9eb532)
[2013-02-25 19:50:14.946643] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00fe4c
[2013-02-25 19:50:14.947019] W [client3_1-fops.c:418:client3_1_open_cbk] 0-adata-client-9: remote operation failed: No such file or directory. Path: /files/random-file.dat (ba539de4-b0cd-46f0-a50c-e93bef9eb532)
[2013-02-25 19:50:15.096925] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:50:15.096972] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:15.097343] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:50:15.097369] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:15.156388] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:50:15.156418] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:15.157088] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:50:15.157113] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:15.157140] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:50:15.157151] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:15.157167] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:50:15.157176] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:50:17.567895] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc009fc4
[2013-02-25 19:50:17.568289] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:50:23.516453] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc011878
[2013-02-25 19:50:23.531267] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:50:50.223193] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00eb28
[2013-02-25 19:50:50.223677] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:51:14.165121] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:51:14.165146] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.226220] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc0116e8
[2013-02-25 19:51:14.235470] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:51:14.246806] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:51:14.246825] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.247018] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:51:14.247033] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.247284] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:51:14.247299] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.524020] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-0: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:51:14.754265] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:51:14.754293] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.754314] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:51:14.754323] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.754339] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:51:14.754348] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:14.754364] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:51:14.754373] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:51:15.765964] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:51:15.766005] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:01.803426] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0100a4
[2013-02-25 19:52:01.803695] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:01.818330] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc00c094
[2013-02-25 19:52:01.818921] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:01.832229] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc003cf0
[2013-02-25 19:52:01.832729] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:01.944443] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc012d2c
[2013-02-25 19:52:01.944828] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:01.957422] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc0045ec
[2013-02-25 19:52:01.957786] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:01.978409] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00526c
[2013-02-25 19:52:01.978885] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:02.065199] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00c2ec
[2013-02-25 19:52:02.065400] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00733c
[2013-02-25 19:52:02.065544] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00bb1c
[2013-02-25 19:52:02.065726] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.065759] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.066056] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.073591] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00c92c
[2013-02-25 19:52:02.073913] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.105553] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00d868
[2013-02-25 19:52:02.105903] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:02.150977] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00d41c
[2013-02-25 19:52:02.151373] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.161042] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc011a08
[2013-02-25 19:52:02.161452] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:02.166178] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00d034
[2013-02-25 19:52:02.166606] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:02.185486] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc012430
[2013-02-25 19:52:02.186015] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.265561] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0069dc
[2013-02-25 19:52:02.265957] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:02.329517] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc00f80c
[2013-02-25 19:52:02.329944] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:52:02.335606] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:52:02.363974] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00701c
[2013-02-25 19:52:02.364108] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc006b08
[2013-02-25 19:52:02.364266] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.364461] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:02.401492] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc008ca0
[2013-02-25 19:52:02.401945] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.477495] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0061a8
[2013-02-25 19:52:02.477822] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:02.478836] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00f35c
[2013-02-25 19:52:02.479360] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.489888] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc00d098
[2013-02-25 19:52:02.490287] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:02.511984] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00a8c0
[2013-02-25 19:52:02.512515] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:02.567901] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc00db24
[2013-02-25 19:52:02.568210] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc011cc4
[2013-02-25 19:52:02.568406] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:02.568616] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:02.783183] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc005334
[2013-02-25 19:52:02.783619] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:02.809052] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc011b98
[2013-02-25 19:52:02.809513] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:02.881407] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00e7a4
[2013-02-25 19:52:02.881967] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:02.882802] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc011c60
[2013-02-25 19:52:02.883258] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:03.059957] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00a0f0
[2013-02-25 19:52:03.060091] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc008728
[2013-02-25 19:52:03.060219] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc01142c
[2013-02-25 19:52:03.060410] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:03.060456] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:03.060511] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:03.107264] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc008a48
[2013-02-25 19:52:03.107713] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:03.133529] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc00ce40
[2013-02-25 19:52:03.133866] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:03.194273] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00ad0c
[2013-02-25 19:52:03.225797] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc00b4dc
[2013-02-25 19:52:03.226297] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:03.233548] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc011eb8
[2013-02-25 19:52:03.234007] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:03.235661] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:03.269451] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc006018
[2013-02-25 19:52:03.269865] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.099232] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc009ca4
[2013-02-25 19:52:14.099562] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc004524
[2013-02-25 19:52:14.099695] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc0119a4
[2013-02-25 19:52:14.100002] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc006338
[2013-02-25 19:52:14.100248] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:14.100675] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc006c98
[2013-02-25 19:52:14.100778] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.100910] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:14.101033] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00ad70
[2013-02-25 19:52:14.101591] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:14.101639] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc0046b4
[2013-02-25 19:52:14.101773] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-9: Opening fd 0x7f63fc009344
[2013-02-25 19:52:14.101902] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc005dc0
[2013-02-25 19:52:14.102383] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00efd8
[2013-02-25 19:52:14.102468] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:14.102682] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc012d90
[2013-02-25 19:52:14.102730] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:14.102925] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc012df4
[2013-02-25 19:52:14.103142] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc004cf4
[2013-02-25 19:52:14.103191] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:14.103244] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:14.103356] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:14.103471] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc011a6c
[2013-02-25 19:52:14.103478] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:52:14.103788] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.103916] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc00a410
[2013-02-25 19:52:14.104187] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.104244] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0049d4
[2013-02-25 19:52:14.104351] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:52:14.104374] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00fa64
[2013-02-25 19:52:14.104515] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc009d6c
[2013-02-25 19:52:14.104644] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc00dbec
[2013-02-25 19:52:14.104780] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00d4e4
[2013-02-25 19:52:14.104807] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:14.105196] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:14.105213] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-2: Opening fd 0x7f63fc010108
[2013-02-25 19:52:14.105426] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:52:14.105461] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc00cea4
[2013-02-25 19:52:14.105725] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:14.106038] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-7
[2013-02-25 19:52:14.106094] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-2: Non blocking entrylks failed.
[2013-02-25 19:52:14.106111] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-2: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.108796] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:52:14.109081] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc00e038
[2013-02-25 19:52:14.109218] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc005654
[2013-02-25 19:52:14.109458] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc009538
[2013-02-25 19:52:14.110856] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.111460] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:14.111568] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.118134] W [client-lk.c:370:delete_granted_locks_owner] 0-adata-client-9: fdctx not valid
[2013-02-25 19:52:14.118194] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-9
[2013-02-25 19:52:14.125879] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-17
[2013-02-25 19:52:14.128863] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-1
[2013-02-25 19:52:14.135518] W [client-lk.c:370:delete_granted_locks_owner] 0-adata-client-19: fdctx not valid
[2013-02-25 19:52:14.135573] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-9: fd for /files/random-file.dat opened successfully on subvolume adata-client-19
[2013-02-25 19:52:14.148798] W [client-lk.c:370:delete_granted_locks_owner] 0-adata-client-5: fdctx not valid
[2013-02-25 19:52:14.148882] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-2: fd for /files/random-file.dat opened successfully on subvolume adata-client-5
[2013-02-25 19:52:14.232687] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232720] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.232783] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232798] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.232829] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232842] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.232859] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232868] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.232882] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.232904] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232913] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.232947] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.232963] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.233063] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.233092] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.233113] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.233125] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.233150] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.233159] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.233208] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.233221] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.233255] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.233267] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.233356] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 19:52:14.233370] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498368] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498420] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498475] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498490] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498535] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498566] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498612] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498626] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498676] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498715] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498838] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498867] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498889] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498909] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498946] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498958] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.498976] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.498985] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499000] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499008] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499023] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499031] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499045] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499053] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499067] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499076] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499090] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499098] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499112] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499121] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499135] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499143] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499157] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499182] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499206] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499235] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499294] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499333] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.499363] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:52:14.499391] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884090] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884150] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884174] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884194] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884218] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884232] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884260] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884270] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884338] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884352] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884376] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884389] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884455] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884480] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884500] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884509] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884534] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884545] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884561] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884570] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884607] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884618] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:14.884639] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 19:52:14.884652] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.148768] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.148803] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.148846] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.148861] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.148900] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.148914] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.148969] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.148986] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.149024] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.149039] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.149080] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.149095] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.149179] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:52:15.149196] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.196746] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.196769] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.196791] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.196801] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.196816] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.196825] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.196902] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.196916] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200255] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200272] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200288] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200297] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200311] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200320] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200334] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200398] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200419] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200430] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200447] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200456] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.200471] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 19:52:15.200480] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678134] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678168] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678197] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678225] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678379] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678408] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678462] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678483] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678511] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678548] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678574] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678586] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:52:15.678603] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 19:52:15.678611] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.225540] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 19:53:16.225586] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.371721] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.371761] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.371876] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.371901] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.371921] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.371930] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.371947] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.371969] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.371988] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.371997] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372027] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372037] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372053] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372062] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372076] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372085] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372114] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372123] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372144] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372157] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372263] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372284] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:16.372409] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:53:16.372434] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:53:26.401057] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-1: Non blocking entrylks failed.
[2013-02-25 19:53:26.401100] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-1: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:54:39.605477] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-10: server 172.16.5.2:24014 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:54:39.605635] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:03.470202 (xid=0x308803x)
[2013-02-25 19:54:39.605653] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.605709] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:05.434752 (xid=0x308804x)
[2013-02-25 19:54:39.605721] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.605768] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:05.752981 (xid=0x308805x)
[2013-02-25 19:54:39.605780] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.605824] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:07.085707 (xid=0x308807x)
[2013-02-25 19:54:39.605849] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.615772] I [socket.c:2315:socket_submit_request] 0-adata-client-10: not connected (priv->connected = 0)
[2013-02-25 19:54:39.615786] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308840x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.615800] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.615852] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:07.085898 (xid=0x308808x)
[2013-02-25 19:54:39.615868] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.615913] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:07.118382 (xid=0x308809x)
[2013-02-25 19:54:39.615924] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.615969] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:07.903241 (xid=0x308810x)
[2013-02-25 19:54:39.615980] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.615993] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308841x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616004] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616040] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:07.903415 (xid=0x308811x)
[2013-02-25 19:54:39.616051] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616097] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:08.334651 (xid=0x308812x)
[2013-02-25 19:54:39.616110] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616157] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:08.444947 (xid=0x308813x)
[2013-02-25 19:54:39.616179] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616193] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308842x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616206] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:08.665553 (xid=0x308814x)
[2013-02-25 19:54:39.616259] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616300] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:09.975672 (xid=0x308815x)
[2013-02-25 19:54:39.616311] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616354] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:10.393502 (xid=0x308818x)
[2013-02-25 19:54:39.616367] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616383] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308843x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616392] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616432] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:11.205471 (xid=0x308819x)
[2013-02-25 19:54:39.616446] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616463] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308844x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616477] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616523] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:11.565160 (xid=0x308820x)
[2013-02-25 19:54:39.616543] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616556] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308845x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616565] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616605] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:12.305492 (xid=0x308823x)
[2013-02-25 19:54:39.616622] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616665] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:16.738309 (xid=0x308825x)
[2013-02-25 19:54:39.616676] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616725] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:17.125970 (xid=0x308826x)
[2013-02-25 19:54:39.616737] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616754] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308846x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616768] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616806] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:19.410791 (xid=0x308829x)
[2013-02-25 19:54:39.616817] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616863] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:19.917845 (xid=0x308831x)
[2013-02-25 19:54:39.616875] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616892] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308847x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.616912] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.616951] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:19.072324 (xid=0x308741x)
[2013-02-25 19:54:39.616965] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.617160] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:53:21.949571 (xid=0x308742x)
[2013-02-25 19:54:39.617174] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.617345] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:53:21.985118 (xid=0x308743x)
[2013-02-25 19:54:39.617361] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.617418] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:22.135504 (xid=0x308744x)
[2013-02-25 19:54:39.617439] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.617424] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308848x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.617473] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.617530] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:53:23.426466 (xid=0x308745x)
[2013-02-25 19:54:39.617543] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.617598] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:25.232168 (xid=0x308746x)
[2013-02-25 19:54:39.617626] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.618219] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.360103 (xid=0x308747x)
[2013-02-25 19:54:39.618243] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.618331] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.361855 (xid=0x308748x)
[2013-02-25 19:54:39.618350] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.618378] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-10: failed to submit rpc-request (XID: 0x308849x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-10)
[2013-02-25 19:54:39.618395] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.618784] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.374918 (xid=0x308749x)
[2013-02-25 19:54:39.618800] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.618889] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.376251 (xid=0x308750x)
[2013-02-25 19:54:39.618908] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.618998] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.377789 (xid=0x308751x)
[2013-02-25 19:54:39.619020] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.619101] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.379240 (xid=0x308752x)
[2013-02-25 19:54:39.619116] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.619542] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.381014 (xid=0x308753x)
[2013-02-25 19:54:39.619561] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.619634] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.383463 (xid=0x308754x)
[2013-02-25 19:54:39.619648] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.620046] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.385282 (xid=0x308755x)
[2013-02-25 19:54:39.620066] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.620463] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.387567 (xid=0x308756x)
[2013-02-25 19:54:39.620480] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.620611] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.389126 (xid=0x308757x)
[2013-02-25 19:54:39.620627] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.621048] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.390987 (xid=0x308758x)
[2013-02-25 19:54:39.621068] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.621185] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.392438 (xid=0x308759x)
[2013-02-25 19:54:39.621219] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.621683] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.394838 (xid=0x308760x)
[2013-02-25 19:54:39.621706] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.622097] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.396354 (xid=0x308761x)
[2013-02-25 19:54:39.622117] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.622191] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.398195 (xid=0x308762x)
[2013-02-25 19:54:39.622205] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.622283] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.399542 (xid=0x308763x)
[2013-02-25 19:54:39.622339] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.622425] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.403386 (xid=0x308764x)
[2013-02-25 19:54:39.622445] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.622865] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.406460 (xid=0x308765x)
[2013-02-25 19:54:39.622882] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.622970] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.409533 (xid=0x308766x)
[2013-02-25 19:54:39.622985] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.623316] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.413008 (xid=0x308767x)
[2013-02-25 19:54:39.623333] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.623407] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.415081 (xid=0x308768x)
[2013-02-25 19:54:39.623431] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.623910] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.416516 (xid=0x308769x)
[2013-02-25 19:54:39.623930] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.624267] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.418105 (xid=0x308770x)
[2013-02-25 19:54:39.624286] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.624413] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.419484 (xid=0x308771x)
[2013-02-25 19:54:39.624432] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.624741] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.420767 (xid=0x308772x)
[2013-02-25 19:54:39.624764] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:54:39.624848] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.449916 (xid=0x308773x)
[2013-02-25 19:54:39.624871] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.625336] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.452199 (xid=0x308774x)
[2013-02-25 19:54:39.625352] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.625670] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:26.453798 (xid=0x308775x)
[2013-02-25 19:54:39.625686] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.625776] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:29.992731 (xid=0x308776x)
[2013-02-25 19:54:39.625795] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.625890] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:30.358343 (xid=0x308777x)
[2013-02-25 19:54:39.625905] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.626351] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:31.746821 (xid=0x308778x)
[2013-02-25 19:54:39.626368] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.626699] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:33.792334 (xid=0x308779x)
[2013-02-25 19:54:39.626719] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.627029] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:34.316745 (xid=0x308780x)
[2013-02-25 19:54:39.627046] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.627136] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:34.725294 (xid=0x308781x)
[2013-02-25 19:54:39.627151] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.627553] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:35.692128 (xid=0x308782x)
[2013-02-25 19:54:39.627576] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.627616] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:35.892957 (xid=0x308783x)
[2013-02-25 19:54:39.627630] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.627730] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:36.002445 (xid=0x308784x)
[2013-02-25 19:54:39.627752] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.627806] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:36.657584 (xid=0x308785x)
[2013-02-25 19:54:39.627825] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.627884] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:36.797095 (xid=0x308786x)
[2013-02-25 19:54:39.627903] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.627953] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:37.153977 (xid=0x308787x)
[2013-02-25 19:54:39.627974] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.628024] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:37.217987 (xid=0x308788x)
[2013-02-25 19:54:39.628044] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.628100] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:37.296555 (xid=0x308789x)
[2013-02-25 19:54:39.628125] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.628297] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:53:37.499908 (xid=0x308790x)
[2013-02-25 19:54:39.628324] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.628386] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:40.004840 (xid=0x308791x)
[2013-02-25 19:54:39.628409] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.628552] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:41.631549 (xid=0x308792x)
[2013-02-25 19:54:39.628580] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.628737] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:47.452073 (xid=0x308793x)
[2013-02-25 19:54:39.628761] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.628893] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:49.372412 (xid=0x308794x)
[2013-02-25 19:54:39.628914] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.629100] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:53.104558 (xid=0x308795x)
[2013-02-25 19:54:39.629125] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.629260] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:55.152521 (xid=0x308796x)
[2013-02-25 19:54:39.629283] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.629706] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:53:57.599889 (xid=0x308797x)
[2013-02-25 19:54:39.629727] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-10: timer must have expired
[2013-02-25 19:54:39.629758] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:57.753123 (xid=0x308798x)
[2013-02-25 19:54:39.629776] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.630512] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:53:59.979480 (xid=0x308799x)
[2013-02-25 19:54:39.630830] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.630996] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:00.211648 (xid=0x308800x)
[2013-02-25 19:54:39.631023] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.631152] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:54:01.576727 (xid=0x308801x)
[2013-02-25 19:54:39.631175] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:54:39.631234] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:03.273478 (xid=0x308802x)
[2013-02-25 19:54:39.631260] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.631381] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:06.775046 (xid=0x308806x)
[2013-02-25 19:54:39.631405] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.631521] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:10.034309 (xid=0x308816x)
[2013-02-25 19:54:39.631545] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.631665] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:10.182698 (xid=0x308817x)
[2013-02-25 19:54:39.631691] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.632125] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:12.232245 (xid=0x308821x)
[2013-02-25 19:54:39.632168] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.632282] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:12.252113 (xid=0x308822x)
[2013-02-25 19:54:39.632305] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.632746] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:12.647384 (xid=0x308824x)
[2013-02-25 19:54:39.632773] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.632891] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:18.763475 (xid=0x308827x)
[2013-02-25 19:54:39.632914] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633039] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:18.772811 (xid=0x308828x)
[2013-02-25 19:54:39.633064] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633182] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:19.706280 (xid=0x308830x)
[2013-02-25 19:54:39.633208] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633336] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:24.865611 (xid=0x308832x)
[2013-02-25 19:54:39.633361] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633451] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:27.923374 (xid=0x308833x)
[2013-02-25 19:54:39.633480] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633549] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:28.642078 (xid=0x308834x)
[2013-02-25 19:54:39.633574] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633646] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:30.055406 (xid=0x308835x)
[2013-02-25 19:54:39.633671] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633746] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:31.025416 (xid=0x308836x)
[2013-02-25 19:54:39.633768] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633842] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:32.799808 (xid=0x308837x)
[2013-02-25 19:54:39.633867] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.633949] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:34.203343 (xid=0x308838x)
[2013-02-25 19:54:39.633973] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.634051] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:54:37.477453 (xid=0x308839x)
[2013-02-25 19:54:39.634082] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:54:39.634132] I [client.c:2090:client_rpc_notify] 0-adata-client-10: disconnected
[2013-02-25 19:54:39.634201] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (291dd2c6-4573-4f97-a417-99cc3cb4dc27) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.634523] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (2df1c602-45e4-4b07-bc5f-2612761b495e) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.634774] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (920f4a2c-4719-4456-a662-deaea5be3c1c) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.635029] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (65eae854-9718-4aaa-9e90-f875ed056c12) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.635249] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (d55ff87e-1c6b-43af-bf7d-81d6e3ced19e) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.635493] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (80b5ad19-ad98-4bfe-b446-4b751a5ff738) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.635658] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (307d8b58-af2a-4d30-b371-4b02c19906f7) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.635826] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (920f4a2c-4719-4456-a662-deaea5be3c1c) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.635985] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (e929a5b1-41f6-4718-91e1-55f16ab61f7c) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.636130] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (9d31d14d-e40e-462f-abb0-a3a21b074412) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.636225] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (d39e425e-29f7-4ee3-985e-2353d24d529c) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.636328] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (7d49e4f0-a6b8-4fa8-8127-17cb23c936d0) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.636423] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-10:  (1d70e96a-db83-4c57-9388-7ce327d78900) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.637502] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (291dd2c6-4573-4f97-a417-99cc3cb4dc27) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.637612] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (2df1c602-45e4-4b07-bc5f-2612761b495e) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.637681] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (65eae854-9718-4aaa-9e90-f875ed056c12) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.637761] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (d55ff87e-1c6b-43af-bf7d-81d6e3ced19e) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.637827] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (80b5ad19-ad98-4bfe-b446-4b751a5ff738) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.637964] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (307d8b58-af2a-4d30-b371-4b02c19906f7) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.638073] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (e929a5b1-41f6-4718-91e1-55f16ab61f7c) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.638134] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (9d31d14d-e40e-462f-abb0-a3a21b074412) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.638188] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (d39e425e-29f7-4ee3-985e-2353d24d529c) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.638244] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (7d49e4f0-a6b8-4fa8-8127-17cb23c936d0) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.638421] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (1d70e96a-db83-4c57-9388-7ce327d78900) remote_fd is -1. EBADFD
[2013-02-25 19:54:39.641998] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (920f4a2c-4719-4456-a662-deaea5be3c1c) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.141353] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (131573d2-55bf-4bad-8d33-212a4ff0ea7c) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.141444] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (21c966b8-0c90-406b-996a-1718175428e5) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.142378] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (21c966b8-0c90-406b-996a-1718175428e5) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.143005] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (21c966b8-0c90-406b-996a-1718175428e5) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.151496] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (7bb33489-e4f8-4893-add1-50dc0859c212) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.151578] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (31310c77-0a9f-430d-96a2-81f0601d4bca) remote_fd is -1. EBADFD
[2013-02-25 19:54:40.151697] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-10:  (bdcaf414-7493-479c-93cf-e1d832d22f87) remote_fd is -1. EBADFD
[2013-02-25 19:55:15.045568] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc00fe4c
[2013-02-25 19:55:15.046119] W [client3_1-fops.c:418:client3_1_open_cbk] 0-adata-client-9: remote operation failed: No such file or directory. Path: /files/random-file.dat (ba539de4-b0cd-46f0-a50c-e93bef9eb532)
[2013-02-25 19:55:16.229723] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-10: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 19:55:16.230165] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-10: Connected to 172.16.5.2:24014, attached to remote volume '/var/gfs/sdg/adata-s002-6'.
[2013-02-25 19:55:16.230192] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-10: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 19:55:16.230209] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-10: 24 fds open - Delaying child_up until they are re-opened
[2013-02-25 19:55:16.237307] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-10: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 19:55:16.248272] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-10: Server lk version = 1
[2013-02-25 19:55:17.025072] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-9: remote operation failed: No such file or directory
[2013-02-25 19:55:18.030267] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-5: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:56:20.621662] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-2: server 172.16.5.2:24010 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.621854] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-18: server 172.16.5.2:24018 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.621896] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-20: server 172.16.5.2:24019 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.621939] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656660 (xid=0x301373x)
[2013-02-25 19:56:20.621967] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-16: server 172.16.5.2:24017 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.621984] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622172] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000399 (xid=0x301374x)
[2013-02-25 19:56:20.622184] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:55:27.622084] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-6: server 172.16.5.2:24012 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.622208] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689190 (xid=0x301375x)
[2013-02-25 19:56:20.622230] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-14: server 172.16.5.2:24016 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.622237] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622295] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-8: server 172.16.5.2:24013 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.622323] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741742 (xid=0x301376x)
[2013-02-25 19:56:20.622360] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:55:28.622329] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-12: server 172.16.5.2:24015 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:20.622392] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757087 (xid=0x301377x)
[2013-02-25 19:56:20.622451] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622482] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.462879 (xid=0x301378x)
[2013-02-25 19:56:20.622496] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622541] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615784 (xid=0x301379x)
[2013-02-25 19:56:20.622556] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-2: timer must have expired
[2013-02-25 19:56:20.622585] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876182 (xid=0x301380x)
[2013-02-25 19:56:20.622594] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-2: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622605] I [client.c:2090:client_rpc_notify] 0-adata-client-2: disconnected
[2013-02-25 19:56:20.622641] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:54:13.649636 (xid=0x327611x)
[2013-02-25 19:56:20.622651] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622697] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:55:51.927290 (xid=0x327849x)
[2013-02-25 19:56:20.622709] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.622754] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.009673 (xid=0x327791x)
[2013-02-25 19:56:20.622769] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.622793] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.622806] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.622984] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.010935 (xid=0x327792x)
[2013-02-25 19:56:20.622999] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623036] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623048] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623120] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.013204 (xid=0x327793x)
[2013-02-25 19:56:20.623136] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623158] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623169] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623159] I [socket.c:2315:socket_submit_request] 0-adata-client-16: not connected (priv->connected = 255)
[2013-02-25 19:56:20.623194] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293392x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.623207] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.623235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.014043 (xid=0x327794x)
[2013-02-25 19:56:20.623249] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623263] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623270] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623340] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.014932 (xid=0x327795x)
[2013-02-25 19:56:20.623361] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623378] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623399] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:56:20.623404] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623413] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:56:20.623467] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:56:20.623477] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:56:20.623483] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.017796 (xid=0x327796x)
[2013-02-25 19:56:20.623498] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623486] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960614: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:56:20.623514] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623543] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623663] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.020786 (xid=0x327797x)
[2013-02-25 19:56:20.623674] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293393x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.623686] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623703] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.623719] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623759] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623832] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.042915 (xid=0x327798x)
[2013-02-25 19:56:20.623840] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293394x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.623846] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.623858] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (c006bd44-4637-4093-92f1-ba49c0fc1c4f)
[2013-02-25 19:56:20.623874] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.623882] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.623977] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:56:20.623990] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:56:20.624001] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.044280 (xid=0x327799x)
[2013-02-25 19:56:20.624013] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:56:20.624017] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624022] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:56:20.624040] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624042] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960617: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:56:20.624056] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.624143] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.045179 (xid=0x327800x)
[2013-02-25 19:56:20.624151] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293395x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.624161] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624175] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.624204] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624219] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.624290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.045867 (xid=0x327801x)
[2013-02-25 19:56:20.624304] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624319] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624327] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.624373] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293396x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.624389] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.624442] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.046750 (xid=0x327802x)
[2013-02-25 19:56:20.624460] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624477] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624485] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.624565] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.047416 (xid=0x327803x)
[2013-02-25 19:56:20.624587] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293397x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.624603] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624616] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.624633] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624658] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.624729] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.048265 (xid=0x327804x)
[2013-02-25 19:56:20.624743] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624748] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293398x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.624758] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624766] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.624773] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.624874] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293399x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.624875] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.129691 (xid=0x327805x)
[2013-02-25 19:56:20.624892] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.624905] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.624944] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.624956] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625005] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293400x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625022] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625033] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.131159 (xid=0x327806x)
[2013-02-25 19:56:20.625053] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.625069] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.625076] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625154] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293401x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625168] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.133394 (xid=0x327807x)
[2013-02-25 19:56:20.625170] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625185] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.625271] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.625282] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625294] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293402x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625310] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625351] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.151124 (xid=0x327808x)
[2013-02-25 19:56:20.625366] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.625380] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.625405] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625413] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293403x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625429] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625476] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.154251 (xid=0x327809x)
[2013-02-25 19:56:20.625513] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.625528] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293404x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625532] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.625545] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625553] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625655] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293405x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625670] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.154790 (xid=0x327810x)
[2013-02-25 19:56:20.625671] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625688] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.625712] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.625720] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625783] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293406x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625791] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.156692 (xid=0x327811x)
[2013-02-25 19:56:20.625801] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625815] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.625847] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.625857] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.625947] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293407x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.625963] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.625978] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.158488 (xid=0x327812x)
[2013-02-25 19:56:20.625994] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626011] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626031] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626067] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293408x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626083] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626155] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.159964 (xid=0x327813x)
[2013-02-25 19:56:20.626171] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626182] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293409x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626186] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626200] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626208] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626356] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.161633 (xid=0x327814x)
[2013-02-25 19:56:20.626366] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293410x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626373] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626384] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626409] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626418] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626513] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293411x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626529] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626539] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.164978 (xid=0x327815x)
[2013-02-25 19:56:20.626554] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626577] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626586] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626631] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293412x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626647] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626680] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.165670 (xid=0x327816x)
[2013-02-25 19:56:20.626699] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626716] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626726] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626739] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293413x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626754] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626790] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.167317 (xid=0x327817x)
[2013-02-25 19:56:20.626804] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626818] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626825] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626862] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293414x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626885] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.626916] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.173090 (xid=0x327818x)
[2013-02-25 19:56:20.626934] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.626950] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.626959] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.626982] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293415x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.626999] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.627022] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.176785 (xid=0x327819x)
[2013-02-25 19:56:20.627035] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.627049] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.627056] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.627094] I [socket.c:2315:socket_submit_request] 0-adata-client-8: not connected (priv->connected = 255)
[2013-02-25 19:56:20.627107] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-8: failed to submit rpc-request (XID: 0x305949x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-8)
[2013-02-25 19:56:20.627117] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (dd80036b-0d7e-4c3c-bac8-9c7567847bd2)
[2013-02-25 19:56:20.627135] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.177478 (xid=0x327820x)
[2013-02-25 19:56:20.627150] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.627165] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.627174] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.627226] I [socket.c:2315:socket_submit_request] 0-adata-client-20: not connected (priv->connected = 255)
[2013-02-25 19:56:20.627245] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-20: failed to submit rpc-request (XID: 0x340763x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-20)
[2013-02-25 19:56:20.627256] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (286796c4-c67a-4c6f-ad23-a47389b87d79)
[2013-02-25 19:56:20.627271] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.179155 (xid=0x327821x)
[2013-02-25 19:56:20.627286] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.627356] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-20: failed to submit rpc-request (XID: 0x340764x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-20)
[2013-02-25 19:56:20.627374] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (a68325b7-5da4-476d-8db5-0d0689aa0257)
[2013-02-25 19:56:20.627383] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.181525 (xid=0x327822x)
[2013-02-25 19:56:20.627398] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.627471] I [socket.c:2315:socket_submit_request] 0-adata-client-14: not connected (priv->connected = 255)
[2013-02-25 19:56:20.627483] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-14: failed to submit rpc-request (XID: 0x304882x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-14)
[2013-02-25 19:56:20.627494] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (4c3c73c8-7836-4570-b2f4-36a94bbd4aae)
[2013-02-25 19:56:20.627522] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.185717 (xid=0x327823x)
[2013-02-25 19:56:20.627540] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 19:56:20.627555] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 19:56:20.627563] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 19:56:20.627601] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-14: failed to submit rpc-request (XID: 0x304883x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-14)
[2013-02-25 19:56:20.627617] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (4c3c73c8-7836-4570-b2f4-36a94bbd4aae)
[2013-02-25 19:56:20.627646] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.186771 (xid=0x327824x)
[2013-02-25 19:56:20.627684] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.627711] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-8: failed to submit rpc-request (XID: 0x305950x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-8)
[2013-02-25 19:56:20.627727] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (dd80036b-0d7e-4c3c-bac8-9c7567847bd2)
[2013-02-25 19:56:20.628135] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.187364 (xid=0x327825x)
[2013-02-25 19:56:20.628153] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.628168] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-14: failed to submit rpc-request (XID: 0x304884x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-14)
[2013-02-25 19:56:20.628184] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (009eead1-5e90-48ca-a2eb-c1f5f5e3f1ad)
[2013-02-25 19:56:20.628235] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.190654 (xid=0x327826x)
[2013-02-25 19:56:20.628252] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.628304] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293416x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.628319] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.628335] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.197213 (xid=0x327827x)
[2013-02-25 19:56:20.628349] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.628416] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293417x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.628434] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.264096 (xid=0x327828x)
[2013-02-25 19:56:20.628438] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.628470] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.628544] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:56:20.628559] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:56:20.628574] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.265985 (xid=0x327829x)
[2013-02-25 19:56:20.628583] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:56:20.628604] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.628610] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:56:20.628628] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960651: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:56:20.628708] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.266594 (xid=0x327830x)
[2013-02-25 19:56:20.628712] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-16: failed to submit rpc-request (XID: 0x293418x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-16)
[2013-02-25 19:56:20.628724] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.628737] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (913cc144-763e-4876-80e0-634be7f49fa6)
[2013-02-25 19:56:20.629165] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.267729 (xid=0x327831x)
[2013-02-25 19:56:20.629181] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.629516] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.268408 (xid=0x327832x)
[2013-02-25 19:56:20.629535] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.629643] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.269173 (xid=0x327833x)
[2013-02-25 19:56:20.629660] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.629781] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.270145 (xid=0x327834x)
[2013-02-25 19:56:20.629807] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.629893] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.270976 (xid=0x327835x)
[2013-02-25 19:56:20.629909] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.630270] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.271858 (xid=0x327836x)
[2013-02-25 19:56:20.630289] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.630389] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:17.273344 (xid=0x327837x)
[2013-02-25 19:56:20.630405] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.630518] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594494 (xid=0x327838x)
[2013-02-25 19:56:20.630546] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.630600] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.595166 (xid=0x327839x)
[2013-02-25 19:56:20.630632] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.630707] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656809 (xid=0x327840x)
[2013-02-25 19:56:20.630724] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.630750] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000559 (xid=0x327841x)
[2013-02-25 19:56:20.630759] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.630781] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689401 (xid=0x327842x)
[2013-02-25 19:56:20.630790] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.630812] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741889 (xid=0x327843x)
[2013-02-25 19:56:20.630821] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.630843] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757296 (xid=0x327844x)
[2013-02-25 19:56:20.630852] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.630873] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.463067 (xid=0x327845x)
[2013-02-25 19:56:20.630882] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.630914] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615800 (xid=0x327846x)
[2013-02-25 19:56:20.630932] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-18: timer must have expired
[2013-02-25 19:56:20.630954] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:39.519683 (xid=0x327847x)
[2013-02-25 19:56:20.630963] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.631011] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:46.627333 (xid=0x327848x)
[2013-02-25 19:56:20.631024] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.631064] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876419 (xid=0x327850x)
[2013-02-25 19:56:20.631075] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.631102] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.169281 (xid=0x327851x)
[2013-02-25 19:56:20.631112] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.640924] I [socket.c:2315:socket_submit_request] 0-adata-client-18: not connected (priv->connected = 0)
[2013-02-25 19:56:20.640939] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-18: failed to submit rpc-request (XID: 0x327854x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-18)
[2013-02-25 19:56:20.640949] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.640997] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.169606 (xid=0x327852x)
[2013-02-25 19:56:20.641010] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641023] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-18: failed to submit rpc-request (XID: 0x327855x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-18)
[2013-02-25 19:56:20.641032] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641068] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.170120 (xid=0x327853x)
[2013-02-25 19:56:20.641088] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641101] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-18: failed to submit rpc-request (XID: 0x327856x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-18)
[2013-02-25 19:56:20.641109] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-18: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641134] I [client.c:2090:client_rpc_notify] 0-adata-client-18: disconnected
[2013-02-25 19:56:20.641173] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:23.522788 (xid=0x340752x)
[2013-02-25 19:56:20.641185] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594553 (xid=0x340753x)
[2013-02-25 19:56:20.641249] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.641299] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.595215 (xid=0x340754x)
[2013-02-25 19:56:20.641311] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.641356] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656827 (xid=0x340755x)
[2013-02-25 19:56:20.641367] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641390] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000581 (xid=0x340756x)
[2013-02-25 19:56:20.641399] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641421] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689422 (xid=0x340757x)
[2013-02-25 19:56:20.641437] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641460] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741906 (xid=0x340758x)
[2013-02-25 19:56:20.641469] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641490] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757318 (xid=0x340759x)
[2013-02-25 19:56:20.641499] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641520] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.463085 (xid=0x340760x)
[2013-02-25 19:56:20.641530] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641551] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615815 (xid=0x340761x)
[2013-02-25 19:56:20.641560] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-20: timer must have expired
[2013-02-25 19:56:20.641579] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876440 (xid=0x340762x)
[2013-02-25 19:56:20.641588] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-20: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641597] I [client.c:2090:client_rpc_notify] 0-adata-client-20: disconnected
[2013-02-25 19:56:20.641629] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:56:14.863105 (xid=0x293391x)
[2013-02-25 19:56:20.641640] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641683] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:34.373987 (xid=0x293382x)
[2013-02-25 19:56:20.641701] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641742] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615831 (xid=0x293383x)
[2013-02-25 19:56:20.641753] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-16: timer must have expired
[2013-02-25 19:56:20.641772] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:50.152492 (xid=0x293384x)
[2013-02-25 19:56:20.641782] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641819] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:50.523075 (xid=0x293385x)
[2013-02-25 19:56:20.641830] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641867] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:50.534045 (xid=0x293386x)
[2013-02-25 19:56:20.641878] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641915] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:53.253013 (xid=0x293387x)
[2013-02-25 19:56:20.641926] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.641963] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:58.333570 (xid=0x293388x)
[2013-02-25 19:56:20.641974] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642012] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876389 (xid=0x293389x)
[2013-02-25 19:56:20.642023] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642054] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:56:12.026617 (xid=0x293390x)
[2013-02-25 19:56:20.642064] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-16: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642088] I [client.c:2090:client_rpc_notify] 0-adata-client-16: disconnected
[2013-02-25 19:56:20.642123] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 19:54:23.231165 (xid=0x305298x)
[2013-02-25 19:56:20.642134] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642141] E [afr-lk-common.c:713:afr_unlock_entrylk_cbk] 0-adata-replicate-3: /files/random-file.dat: unlock failed on 0, reason: Transport endpoint is not connected
[2013-02-25 19:56:20.642148] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 19:56:20.642157] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:56:20.642182] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594099 (xid=0x305432x)
[2013-02-25 19:56:20.642194] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.642241] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594841 (xid=0x305433x)
[2013-02-25 19:56:20.642253] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.642297] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656695 (xid=0x305434x)
[2013-02-25 19:56:20.642308] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642330] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000442 (xid=0x305435x)
[2013-02-25 19:56:20.642339] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642372] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689269 (xid=0x305436x)
[2013-02-25 19:56:20.642381] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642403] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741780 (xid=0x305437x)
[2013-02-25 19:56:20.642412] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642433] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757135 (xid=0x305438x)
[2013-02-25 19:56:20.642442] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642464] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.462929 (xid=0x305439x)
[2013-02-25 19:56:20.642473] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642494] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615847 (xid=0x305440x)
[2013-02-25 19:56:20.642503] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-6: timer must have expired
[2013-02-25 19:56:20.642522] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876256 (xid=0x305441x)
[2013-02-25 19:56:20.642531] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642552] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:56:17.095268 (xid=0x305442x)
[2013-02-25 19:56:20.642561] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.642923] I [socket.c:2315:socket_submit_request] 0-adata-client-6: not connected (priv->connected = 0)
[2013-02-25 19:56:20.642950] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-6: failed to submit rpc-request (XID: 0x305447x Program: GlusterFS 3.1, ProgVers: 330, Proc: 12) to rpc-transport (adata-client-6)
[2013-02-25 19:56:20.642964] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643000] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-6: failed to submit rpc-request (XID: 0x305448x Program: GlusterFS 3.1, ProgVers: 330, Proc: 12) to rpc-transport (adata-client-6)
[2013-02-25 19:56:20.643014] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643058] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.388582 (xid=0x305443x)
[2013-02-25 19:56:20.643075] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643092] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-6: failed to submit rpc-request (XID: 0x305449x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-6)
[2013-02-25 19:56:20.643101] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643157] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.388636 (xid=0x305444x)
[2013-02-25 19:56:20.643175] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643196] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-6: failed to submit rpc-request (XID: 0x305450x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-6)
[2013-02-25 19:56:20.643207] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.389040 (xid=0x305445x)
[2013-02-25 19:56:20.643277] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643299] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-6: failed to submit rpc-request (XID: 0x305451x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-6)
[2013-02-25 19:56:20.643310] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643361] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 19:56:20.390810 (xid=0x305446x)
[2013-02-25 19:56:20.643380] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643412] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-6: failed to submit rpc-request (XID: 0x305452x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-6)
[2013-02-25 19:56:20.643427] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-6: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643453] I [client.c:2090:client_rpc_notify] 0-adata-client-6: disconnected
[2013-02-25 19:56:20.643499] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:56:17.984941 (xid=0x304881x)
[2013-02-25 19:56:20.643514] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643572] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594380 (xid=0x304871x)
[2013-02-25 19:56:20.643588] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.643647] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.595058 (xid=0x304872x)
[2013-02-25 19:56:20.643663] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.643733] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656773 (xid=0x304873x)
[2013-02-25 19:56:20.643754] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643783] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000521 (xid=0x304874x)
[2013-02-25 19:56:20.643792] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643815] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689357 (xid=0x304875x)
[2013-02-25 19:56:20.643824] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643845] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741854 (xid=0x304876x)
[2013-02-25 19:56:20.643861] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643884] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757239 (xid=0x304877x)
[2013-02-25 19:56:20.643893] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643915] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.463020 (xid=0x304878x)
[2013-02-25 19:56:20.643924] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643945] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615863 (xid=0x304879x)
[2013-02-25 19:56:20.643954] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-14: timer must have expired
[2013-02-25 19:56:20.643973] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876359 (xid=0x304880x)
[2013-02-25 19:56:20.643982] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-14: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.643992] I [client.c:2090:client_rpc_notify] 0-adata-client-14: disconnected
[2013-02-25 19:56:20.644024] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689292 (xid=0x305943x)
[2013-02-25 19:56:20.644034] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644057] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741798 (xid=0x305944x)
[2013-02-25 19:56:20.644066] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644087] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757157 (xid=0x305945x)
[2013-02-25 19:56:20.644103] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644126] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.462954 (xid=0x305946x)
[2013-02-25 19:56:20.644135] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644156] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615875 (xid=0x305947x)
[2013-02-25 19:56:20.644165] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-8: timer must have expired
[2013-02-25 19:56:20.644184] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876279 (xid=0x305948x)
[2013-02-25 19:56:20.644193] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-8: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644202] I [client.c:2090:client_rpc_notify] 0-adata-client-8: disconnected
[2013-02-25 19:56:20.644230] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594290 (xid=0x323791x)
[2013-02-25 19:56:20.644242] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.644290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594991 (xid=0x323792x)
[2013-02-25 19:56:20.644302] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:20.644346] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656753 (xid=0x323793x)
[2013-02-25 19:56:20.644356] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644379] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000501 (xid=0x323794x)
[2013-02-25 19:56:20.644394] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644417] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689334 (xid=0x323795x)
[2013-02-25 19:56:20.644426] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644448] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741835 (xid=0x323796x)
[2013-02-25 19:56:20.644457] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644478] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757203 (xid=0x323797x)
[2013-02-25 19:56:20.644487] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644508] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.463003 (xid=0x323798x)
[2013-02-25 19:56:20.644517] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644538] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:38.615885 (xid=0x323799x)
[2013-02-25 19:56:20.644546] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-12: timer must have expired
[2013-02-25 19:56:20.644566] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876330 (xid=0x323800x)
[2013-02-25 19:56:20.644574] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-12: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:20.644583] I [client.c:2090:client_rpc_notify] 0-adata-client-12: disconnected
[2013-02-25 19:56:20.644614] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-18:  (77d32a7f-9d76-4cf2-9bdc-9fdce4eba39a) remote_fd is -1. EBADFD
[2013-02-25 19:56:20.646021] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-14:  (6146872e-d684-4a49-a8bb-c986bfb6caec) remote_fd is -1. EBADFD
[2013-02-25 19:56:20.646244] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-18:  (b8cbff9c-291b-47a7-b50f-60ba499d59e9) remote_fd is -1. EBADFD
[2013-02-25 19:56:20.646744] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-18:  (77d32a7f-9d76-4cf2-9bdc-9fdce4eba39a) remote_fd is -1. EBADFD
[2013-02-25 19:56:20.646850] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-14:  (6146872e-d684-4a49-a8bb-c986bfb6caec) remote_fd is -1. EBADFD
[2013-02-25 19:56:20.649195] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-16:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:56:20.651072] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-16:  (913cc144-763e-4876-80e0-634be7f49fa6) remote_fd is -1. EBADFD
[2013-02-25 19:56:21.185019] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-8:  (71d45ba6-1bd2-48cd-a836-f6caadc1e002) remote_fd is -1. EBADFD
[2013-02-25 19:56:21.185411] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-8:  (71d45ba6-1bd2-48cd-a836-f6caadc1e002) remote_fd is -1. EBADFD
[2013-02-25 19:56:23.379240] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-18:  (681949b0-cdcb-4c97-8e2e-2601e1490e93) remote_fd is -1. EBADFD
[2013-02-25 19:56:23.379504] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-18:  (681949b0-cdcb-4c97-8e2e-2601e1490e93) remote_fd is -1. EBADFD
[2013-02-25 19:56:23.443423] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-8:  (e0ef5d98-be23-452f-b246-30d08192c2f1) remote_fd is -1. EBADFD
[2013-02-25 19:56:23.443675] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-8:  (e0ef5d98-be23-452f-b246-30d08192c2f1) remote_fd is -1. EBADFD
[2013-02-25 19:56:23.444538] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-8:  (2eb8e10a-ab1a-49ad-895a-7ef4c4e91481) remote_fd is -1. EBADFD
[2013-02-25 19:56:23.444786] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-8:  (2eb8e10a-ab1a-49ad-895a-7ef4c4e91481) remote_fd is -1. EBADFD
[2013-02-25 19:56:40.628409] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-10: server 172.16.5.2:24014 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:56:40.628675] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:24.651006 (xid=0x308967x)
[2013-02-25 19:56:40.628707] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.628784] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594225 (xid=0x308968x)
[2013-02-25 19:56:40.628799] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:40.628880] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594942 (xid=0x308969x)
[2013-02-25 19:56:40.628892] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:56:40.628957] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:26.656735 (xid=0x308970x)
[2013-02-25 19:56:40.628969] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.628993] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.000481 (xid=0x308971x)
[2013-02-25 19:56:40.629003] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629025] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.689313 (xid=0x308972x)
[2013-02-25 19:56:40.629034] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629056] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:27.741816 (xid=0x308973x)
[2013-02-25 19:56:40.629065] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629087] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:28.757180 (xid=0x308974x)
[2013-02-25 19:56:40.629095] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629117] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:55:29.462977 (xid=0x308975x)
[2013-02-25 19:56:40.629126] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629148] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:55:48.595914 (xid=0x308976x)
[2013-02-25 19:56:40.629157] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:55:58.618711 (xid=0x308977x)
[2013-02-25 19:56:40.629218] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-10: timer must have expired
[2013-02-25 19:56:40.629253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:56:01.876301 (xid=0x308978x)
[2013-02-25 19:56:40.629265] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-10: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:56:40.629279] I [client.c:2090:client_rpc_notify] 0-adata-client-10: disconnected
[2013-02-25 19:56:57.882115] I [afr-self-heal-common.c:1811:afr_sh_find_fresh_parents] 0-adata-replicate-5: /files: Performing conservative merge
[2013-02-25 19:56:57.883595] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-5: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 19:56:57.884601] E [afr-self-heal-data.c:1047:afr_sh_data_fxattrop_cbk] 0-adata-replicate-5: /files/random-file.dat, inspecting change log succeeded on < 2 children
[2013-02-25 19:56:57.884897] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-5: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 19:57:02.633338] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-0: server 172.16.5.2:24009 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 19:57:02.633540] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 19:56:25.165368 (xid=0x310761x)
[2013-02-25 19:57:02.633560] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:57:02.633622] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.593923 (xid=0x310688x)
[2013-02-25 19:57:02.633637] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.633847] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:55:26.594678 (xid=0x310689x)
[2013-02-25 19:57:02.675812] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:57:00.067423 (xid=0x310783x)
[2013-02-25 19:57:02.675845] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.675885] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-0: failed to submit rpc-request (XID: 0x310859x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-0)
[2013-02-25 19:57:02.675917] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.676291] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:57:01.751659 (xid=0x310784x)
[2013-02-25 19:57:02.676321] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected
[2013-02-25 19:57:02.676476] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:57:01.777581 (xid=0x310785x)
[2013-02-25 19:57:02.676492] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.676530] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-0: failed to submit rpc-request (XID: 0x310860x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-0)
[2013-02-25 19:57:02.676542] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.676902] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:57:01.777697 (xid=0x310786x)
[2013-02-25 19:57:02.676932] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.676990] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-0: failed to submit rpc-request (XID: 0x310861x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-0)
[2013-02-25 19:57:02.677004] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 19:57:02.677429] I [client.c:2090:client_rpc_notify] 0-adata-client-0: disconnected
[2013-02-25 19:57:02.677489] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-0:  (65efd052-c74c-4cef-9dbe-4dbbe2ebce26) remote_fd is -1. EBADFD
[2013-02-25 19:57:02.707532] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-0:  (65efd052-c74c-4cef-9dbe-4dbbe2ebce26) remote_fd is -1. EBADFD
[2013-02-25 19:57:02.780667] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.780700] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.780745] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.780774] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.780792] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960897: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.780883] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.780899] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.780937] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.780952] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.780965] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960898: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.781385] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.781402] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.781442] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.781456] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.781470] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960900: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.794643] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 19:57:02.794673] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.794714] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 19:57:02.794732] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.794749] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3960993: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.795579] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.795597] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.795634] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.795649] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.795663] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961000: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.796126] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.796143] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.796180] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:02.796195] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.796209] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961006: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.796546] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2749541206
[2013-02-25 19:57:02.796563] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.796577] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961009: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.796753] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2908800651
[2013-02-25 19:57:02.796784] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.796830] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2908800651
[2013-02-25 19:57:02.796844] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.796857] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961011: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.796966] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2749541206
[2013-02-25 19:57:02.796983] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.796996] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961012: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.797430] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 19:57:02.797447] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.797482] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 19:57:02.797500] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.797514] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961016: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.798119] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 19:57:02.798137] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.798163] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 19:57:02.798171] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.798180] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961022: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.798646] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2908800651
[2013-02-25 19:57:02.798664] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.798678] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961026: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:02.798903] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2908800651
[2013-02-25 19:57:02.798916] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:02.798925] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961028: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:23.640957] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.640979] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.641064] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.641716] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.641752] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961588: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:23.641836] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.641848] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.641878] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.641897] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.641906] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961589: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:23.960991] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.961011] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.961046] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.961054] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.961064] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961593: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:23.961124] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.961136] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.961159] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:23.961166] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:23.961174] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961594: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:37.102719] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:37.102744] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:37.102774] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:37.102783] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:37.102792] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961839: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:37.102903] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:37.102916] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:37.102941] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:37.102949] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:37.102956] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961840: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:39.242840] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:39.242861] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:39.242887] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:39.242894] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:39.242904] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961880: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:39.242964] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:39.242976] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:39.242998] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:39.243007] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:39.243028] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3961881: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:51.880731] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:51.880757] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:51.880788] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:51.880796] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:51.880805] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962116: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:51.880874] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:51.880885] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:51.880909] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:51.880916] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:51.880924] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962117: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:54.581696] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:54.581718] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:54.581750] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:54.581759] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:54.581768] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962202: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:57:54.581828] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:54.581839] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:54.581862] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:57:54.581870] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:57:54.581877] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962203: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:06.524175] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:06.524199] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:06.524251] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:06.524262] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:06.524273] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962388: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:06.524427] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:06.524439] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:06.524500] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:06.524530] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:06.524539] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962389: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:08.921606] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:08.921630] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:08.921666] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:08.921675] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:08.921684] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962432: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:08.921745] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:08.921757] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:08.921780] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2967408968
[2013-02-25 19:58:08.921787] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:08.921795] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962433: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:46.210542] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2760918429
[2013-02-25 19:58:46.210571] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:46.210596] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2760918429
[2013-02-25 19:58:46.210603] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:46.210613] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962887: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:46.211217] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2760918429
[2013-02-25 19:58:46.211229] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:46.211237] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962889: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:46.211427] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2760918429
[2013-02-25 19:58:46.211441] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:46.211450] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962890: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:48.974601] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2760918429
[2013-02-25 19:58:48.974626] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:48.974637] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962903: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:58:48.974803] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2760918429
[2013-02-25 19:58:48.974815] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 19:58:48.974823] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3962904: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 19:59:48.921303] W [socket.c:195:__socket_rwv] 0-adata-client-10: readv failed (Connection timed out)
[2013-02-25 19:59:48.921349] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-10: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24014)
[2013-02-25 19:59:48.921401] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-10: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:50.631511 (xid=0x308979x)
[2013-02-25 19:59:48.921428] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-10: received RPC status error
[2013-02-25 19:59:48.921442] I [client.c:2090:client_rpc_notify] 0-adata-client-10: disconnected
[2013-02-25 19:59:48.921451] W [socket.c:195:__socket_rwv] 0-adata-client-2: readv failed (Connection timed out)
[2013-02-25 19:59:48.921458] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-2: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24010)
[2013-02-25 19:59:48.921485] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-2: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:30.624754 (xid=0x301381x)
[2013-02-25 19:59:48.921495] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-2: received RPC status error
[2013-02-25 19:59:48.921502] I [client.c:2090:client_rpc_notify] 0-adata-client-2: disconnected
[2013-02-25 19:59:49.945403] W [socket.c:195:__socket_rwv] 0-adata-client-14: readv failed (Connection timed out)
[2013-02-25 19:59:49.945446] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-14: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24016)
[2013-02-25 19:59:49.945509] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-14: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:31.626223 (xid=0x304885x)
[2013-02-25 19:59:49.945521] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-14: received RPC status error
[2013-02-25 19:59:49.945531] I [client.c:2090:client_rpc_notify] 0-adata-client-14: disconnected
[2013-02-25 19:59:49.945544] W [socket.c:195:__socket_rwv] 0-adata-client-20: readv failed (Connection timed out)
[2013-02-25 19:59:49.945550] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-20: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24019)
[2013-02-25 19:59:49.945577] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-20: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:31.625469 (xid=0x340765x)
[2013-02-25 19:59:49.945586] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-20: received RPC status error
[2013-02-25 19:59:49.945593] I [client.c:2090:client_rpc_notify] 0-adata-client-20: disconnected
[2013-02-25 19:59:49.945601] W [socket.c:195:__socket_rwv] 0-adata-client-16: readv failed (Connection timed out)
[2013-02-25 19:59:49.945608] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-16: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24017)
[2013-02-25 19:59:49.945633] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-16: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:31.625941 (xid=0x293419x)
[2013-02-25 19:59:49.945642] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-16: received RPC status error
[2013-02-25 19:59:49.945650] I [client.c:2090:client_rpc_notify] 0-adata-client-16: disconnected
[2013-02-25 19:59:49.945670] W [socket.c:195:__socket_rwv] 0-adata-client-12: readv failed (Connection timed out)
[2013-02-25 19:59:49.945677] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-12: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24015)
[2013-02-25 19:59:49.945703] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-12: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:31.627034 (xid=0x323801x)
[2013-02-25 19:59:49.945713] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-12: received RPC status error
[2013-02-25 19:59:49.945719] I [client.c:2090:client_rpc_notify] 0-adata-client-12: disconnected
[2013-02-25 19:59:49.949364] W [socket.c:195:__socket_rwv] 0-adata-client-8: readv failed (Connection timed out)
[2013-02-25 19:59:49.949395] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-8: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24013)
[2013-02-25 19:59:49.949459] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-8: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:31.626633 (xid=0x305951x)
[2013-02-25 19:59:49.949479] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-8: received RPC status error
[2013-02-25 19:59:49.949493] I [client.c:2090:client_rpc_notify] 0-adata-client-8: disconnected
[2013-02-25 19:59:58.673393] E [socket.c:1715:socket_connect_finish] 0-adata-client-10: connection to 172.16.5.2:24014 failed (No route to host)
[2013-02-25 19:59:58.969478] W [socket.c:195:__socket_rwv] 0-adata-client-6: readv failed (Connection timed out)
[2013-02-25 19:59:58.969522] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-6: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24012)
[2013-02-25 19:59:58.969580] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-6: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:20.644905 (xid=0x305453x)
[2013-02-25 19:59:58.969593] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-6: received RPC status error
[2013-02-25 19:59:58.969604] I [client.c:2090:client_rpc_notify] 0-adata-client-6: disconnected
[2013-02-25 19:59:58.969617] W [socket.c:195:__socket_rwv] 0-adata-client-18: readv failed (Connection timed out)
[2013-02-25 19:59:58.969624] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-18: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24018)
[2013-02-25 19:59:58.969650] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-18: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:56:20.644881 (xid=0x327857x)
[2013-02-25 19:59:58.969659] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-18: received RPC status error
[2013-02-25 19:59:58.969668] I [client.c:2090:client_rpc_notify] 0-adata-client-18: disconnected
[2013-02-25 20:00:00.957428] W [socket.c:195:__socket_rwv] 0-adata-client-0: readv failed (Connection timed out)
[2013-02-25 20:00:00.957472] W [socket.c:1512:__socket_proto_state_machine] 0-adata-client-0: reading from socket failed. Error (Connection timed out), peer (172.16.5.2:24009)
[2013-02-25 20:00:00.957541] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2013-02-25 19:57:02.677742 (xid=0x310862x)
[2013-02-25 20:00:00.957554] W [client-handshake.c:1819:client_dump_version_cbk] 0-adata-client-0: received RPC status error
[2013-02-25 20:00:00.957564] I [client.c:2090:client_rpc_notify] 0-adata-client-0: disconnected
[2013-02-25 20:00:01.677357] E [socket.c:1715:socket_connect_finish] 0-adata-client-0: connection to 172.16.5.2:24009 failed (No route to host)
[2013-02-25 20:00:01.677403] E [socket.c:1715:socket_connect_finish] 0-adata-client-18: connection to 172.16.5.2:24018 failed (No route to host)
[2013-02-25 20:00:01.677431] E [socket.c:1715:socket_connect_finish] 0-adata-client-6: connection to 172.16.5.2:24012 failed (No route to host)
[2013-02-25 20:00:08.168974] W [page.c:984:__ioc_page_error] 0-adata-io-cache: page error for page = 0x7f63f00caa80 & waitq = 0x7f63f03a5ba0
[2013-02-25 20:00:25.689468] E [socket.c:1715:socket_connect_finish] 0-adata-client-16: connection to 172.16.5.2:24017 failed (No route to host)
[2013-02-25 20:00:32.695881] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-4: server 172.16.5.2:24011 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:00:32.696125] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 19:57:05.333198 (xid=0x344728x)
[2013-02-25 20:00:32.696149] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:00:32.696214] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:29.391286 (xid=0x344949x)
[2013-02-25 20:00:32.696229] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.696337] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 19:59:29.735203 (xid=0x344950x)
[2013-02-25 20:00:32.696350] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:00:32.696442] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:36.292949 (xid=0x344951x)
[2013-02-25 20:00:32.696465] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.696579] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:44.398721 (xid=0x344952x)
[2013-02-25 20:00:32.696609] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.696690] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:45.727592 (xid=0x344953x)
[2013-02-25 20:00:32.696704] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.696782] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:45.785487 (xid=0x344954x)
[2013-02-25 20:00:32.696797] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.696884] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 19:59:50.660078 (xid=0x344955x)
[2013-02-25 20:00:32.696897] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-4: timer must have expired
[2013-02-25 20:00:32.696918] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:51.855116 (xid=0x344956x)
[2013-02-25 20:00:32.696930] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697009] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:53.038078 (xid=0x344957x)
[2013-02-25 20:00:32.697023] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697100] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 19:59:59.727535 (xid=0x344958x)
[2013-02-25 20:00:32.697116] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697201] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:00:01.886160 (xid=0x344959x)
[2013-02-25 20:00:32.697214] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:00:32.697255] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:05.837230 (xid=0x344960x)
[2013-02-25 20:00:32.697267] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697345] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:05.903057 (xid=0x344961x)
[2013-02-25 20:00:32.697359] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697438] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:07.316493 (xid=0x344962x)
[2013-02-25 20:00:32.697452] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697528] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:10.697135 (xid=0x344963x)
[2013-02-25 20:00:32.697542] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697618] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:14.555437 (xid=0x344964x)
[2013-02-25 20:00:32.697632] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697709] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:14.687264 (xid=0x344965x)
[2013-02-25 20:00:32.697725] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697828] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:23.709517 (xid=0x344966x)
[2013-02-25 20:00:32.697844] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.697921] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 20:00:25.083995 (xid=0x344967x)
[2013-02-25 20:00:32.697934] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:00:32.697984] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:26.804511 (xid=0x344968x)
[2013-02-25 20:00:32.697998] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.698077] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:29.981851 (xid=0x344969x)
[2013-02-25 20:00:32.698091] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.698167] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-4: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:00:31.544665 (xid=0x344970x)
[2013-02-25 20:00:32.698182] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-4: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:00:32.698239] I [client.c:2090:client_rpc_notify] 0-adata-client-4: disconnected
[2013-02-25 20:00:32.713515] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.713568] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.713881] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.713898] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.714320] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.714334] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.714581] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.714607] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.714855] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.714870] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.715003] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.715029] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.715140] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.715156] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.715309] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.715330] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.715608] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.715626] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.715869] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.715886] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.716090] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.716108] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.716336] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.716354] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.716583] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:32.716605] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:32.716652] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:32.716668] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:32.716683] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963212: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:32.716695] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.716745] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.716779] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:32.716794] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:32.716837] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:32.716855] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:32.716869] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963213: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:32.716965] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:32.717002] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:32.717057] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:32.717073] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:32.717087] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963214: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:32.717097] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.717144] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.717534] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.717553] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.717681] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.717700] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.717940] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.717983] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:32.718238] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:00:32.718283] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:00:34.509489] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:00:34.509608] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:35.961708] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:00:35.961845] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:36.218469] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:36.218505] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:36.218545] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:36.218553] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:36.218562] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963366: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:36.218625] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:36.218637] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:36.218659] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:36.218666] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:36.218673] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963367: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:37.030526] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:00:37.030602] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:37.030775] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.030788] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.030826] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.030834] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.030841] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963391: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:37.030997] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.031012] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.031035] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.031043] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.031050] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963392: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:37.817193] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.817210] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.817253] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.817263] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.817272] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963407: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:37.817338] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.817349] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.817372] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:00:37.817379] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:00:37.817386] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3963408: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:00:38.178261] I [dht-layout.c:698:dht_layout_dir_mismatch] 0-adata-dht: subvol: adata-replicate-3; inode layout - 0 - 0; disk layout - 390451572 - 780903143
[2013-02-25 20:00:38.178288] I [dht-common.c:596:dht_revalidate_cbk] 0-adata-dht: mismatching layouts for /files
[2013-02-25 20:00:38.178423] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:38.179115] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:00:38.179129] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:00:39.866967] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:41.169815] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:42.197369] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:43.697460] E [socket.c:1715:socket_connect_finish] 0-adata-client-4: connection to 172.16.5.2:24011 failed (No route to host)
[2013-02-25 20:00:44.046187] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:45.866424] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:47.566817] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:49.263474] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:51.308262] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:00:52.985074] I [dht-layout.c:698:dht_layout_dir_mismatch] 0-adata-dht: subvol: adata-replicate-9; inode layout - 0 - 0; disk layout - 2733161004 - 3123612575
[2013-02-25 20:00:52.985095] I [dht-common.c:596:dht_revalidate_cbk] 0-adata-dht: mismatching layouts for /files
[2013-02-25 20:00:54.789383] E [socket.c:1715:socket_connect_finish] 0-adata-client-2: connection to 172.16.5.2:24010 failed (Connection timed out)
[2013-02-25 20:00:55.813390] E [socket.c:1715:socket_connect_finish] 0-adata-client-12: connection to 172.16.5.2:24015 failed (Connection timed out)
[2013-02-25 20:00:55.813450] E [socket.c:1715:socket_connect_finish] 0-adata-client-20: connection to 172.16.5.2:24019 failed (Connection timed out)
[2013-02-25 20:00:55.813479] E [socket.c:1715:socket_connect_finish] 0-adata-client-8: connection to 172.16.5.2:24013 failed (Connection timed out)
[2013-02-25 20:00:55.813502] E [socket.c:1715:socket_connect_finish] 0-adata-client-14: connection to 172.16.5.2:24016 failed (Connection timed out)
[2013-02-25 20:00:58.588846] W [page.c:984:__ioc_page_error] 0-adata-io-cache: page error for page = 0x7f63fc3c3e40 & waitq = 0x7f63fc1f78b0
[2013-02-25 20:01:19.679729] W [page.c:984:__ioc_page_error] 0-adata-io-cache: page error for page = 0x7f63fc24ad20 & waitq = 0x7f63fc25be30
[2013-02-25 20:01:31.478762] W [page.c:984:__ioc_page_error] 0-adata-io-cache: page error for page = 0x7f63fc165b60 & waitq = 0x7f63fc06a820
[2013-02-25 20:01:42.760843] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-4: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:42.761120] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-4: Connected to 172.16.5.2:24011, attached to remote volume '/var/gfs/sdd/adata-s002-3'.
[2013-02-25 20:01:42.761163] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:42.761173] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-4: 2 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:42.761264] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-2: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:42.763628] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-2: Connected to 172.16.5.2:24010, attached to remote volume '/var/gfs/sdc/adata-s002-2'.
[2013-02-25 20:01:42.763652] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:42.763830] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-2: Server lk version = 1
[2013-02-25 20:01:42.815421] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-4: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:42.815672] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-4: Server lk version = 1
[2013-02-25 20:01:43.762173] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-0: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:43.762461] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-0: Connected to 172.16.5.2:24009, attached to remote volume '/var/gfs/sdb/adata-s002-1'.
[2013-02-25 20:01:43.762481] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:43.762503] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-0: 7 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:43.762638] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-16: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:43.762910] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-16: Connected to 172.16.5.2:24017, attached to remote volume '/var/gfs/sdj/adata-s002-9'.
[2013-02-25 20:01:43.762925] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-16: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:43.762933] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-16: 12 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:43.763115] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-8: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:43.763440] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-8: Connected to 172.16.5.2:24013, attached to remote volume '/var/gfs/sdf/adata-s002-5'.
[2013-02-25 20:01:43.763454] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-8: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:43.763462] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-8: 5 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:43.763542] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-20: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:43.763799] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-20: Connected to 172.16.5.2:24019, attached to remote volume '/var/gfs/sdl/adata-s002-11'.
[2013-02-25 20:01:43.763814] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-20: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:43.763837] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-20: 1 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:43.763897] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-12: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:43.764270] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-12: Connected to 172.16.5.2:24015, attached to remote volume '/var/gfs/sdh/adata-s002-7'.
[2013-02-25 20:01:43.764294] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-12: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:43.764350] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-14: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:43.764467] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-12: Server lk version = 1
[2013-02-25 20:01:43.766668] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-14: Connected to 172.16.5.2:24016, attached to remote volume '/var/gfs/sdi/adata-s002-8'.
[2013-02-25 20:01:43.766690] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-14: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:43.766702] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-14: 2 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:43.785667] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:43.785894] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-0: Server lk version = 1
[2013-02-25 20:01:43.792301] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-8: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:43.792510] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-8: Server lk version = 1
[2013-02-25 20:01:43.803542] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-16: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:43.803737] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-16: Server lk version = 1
[2013-02-25 20:01:43.813305] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-20: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:43.813527] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-20: Server lk version = 1
[2013-02-25 20:01:43.863908] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-14: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:43.864182] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-14: Server lk version = 1
[2013-02-25 20:01:44.765350] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-10: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:44.765681] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-10: Connected to 172.16.5.2:24014, attached to remote volume '/var/gfs/sdg/adata-s002-6'.
[2013-02-25 20:01:44.765709] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-10: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:44.765727] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-10: 25 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:44.817437] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-10: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:44.817617] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-10: Server lk version = 1
[2013-02-25 20:01:50.789688] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-18: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:50.790059] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-18: Connected to 172.16.5.2:24018, attached to remote volume '/var/gfs/sdk/adata-s002-10'.
[2013-02-25 20:01:50.790086] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-18: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:50.790101] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-18: 9 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:50.790779] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-18: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:50.790936] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-18: Server lk version = 1
[2013-02-25 20:01:55.777332] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-6: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:01:55.777727] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-6: Connected to 172.16.5.2:24012, attached to remote volume '/var/gfs/sde/adata-s002-4'.
[2013-02-25 20:01:55.777752] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-6: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:01:55.777766] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-6: 9 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:01:55.800221] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-6: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:01:55.800521] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-6: Server lk version = 1
[2013-02-25 20:01:56.137755] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc0098bc
[2013-02-25 20:01:56.138328] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-16
[2013-02-25 20:01:59.799565] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 20:01:59.799621] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:01:59.799661] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 20:01:59.799685] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:01:59.940926] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 20:01:59.940980] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:01:59.941020] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 20:01:59.941035] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:02.898925] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00add4
[2013-02-25 20:02:02.899082] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc011558
[2013-02-25 20:02:02.899219] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-4: Opening fd 0x7f63fc0102fc
[2013-02-25 20:02:02.899264] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-0
[2013-02-25 20:02:02.899369] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc0080e8
[2013-02-25 20:02:02.899526] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc009e98
[2013-02-25 20:02:02.899694] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-0
[2013-02-25 20:02:02.899736] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-0
[2013-02-25 20:02:02.899699] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc007850
[2013-02-25 20:02:02.899957] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc00bea0
[2013-02-25 20:02:02.899979] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-0
[2013-02-25 20:02:02.918382] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-10: Opening fd 0x7f63fc01255c
[2013-02-25 20:02:02.936092] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-0
[2013-02-25 20:02:02.937522] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-8
[2013-02-25 20:02:02.937580] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-4: fd for /files/random-file.dat opened successfully on subvolume adata-client-8
[2013-02-25 20:02:02.968519] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-10: fd for /files/random-file.dat opened successfully on subvolume adata-client-20
[2013-02-25 20:02:03.179923] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-8: Opening fd 0x7f63fc00a9ec
[2013-02-25 20:02:03.180229] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-8: fd for /files/random-file.dat opened successfully on subvolume adata-client-16
[2013-02-25 20:02:09.485086] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-0: Opening fd 0x7f63fc01110c
[2013-02-25 20:02:09.518010] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-0: fd for /files/random-file.dat opened successfully on subvolume adata-client-0
[2013-02-25 20:02:11.858662] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-10: Opening fd 0x7f63fc00fe4c
[2013-02-25 20:02:11.910529] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-10: fd for /files/random-file.dat opened successfully on subvolume adata-client-20
[2013-02-25 20:02:13.501378] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-2: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 20:02:49.434770] I [afr-inode-write.c:428:afr_open_fd_fix] 0-adata-replicate-3: Opening fd 0x7f63fc007c38
[2013-02-25 20:02:49.435182] I [afr-open.c:318:afr_openfd_fix_open_cbk] 0-adata-replicate-3: fd for /files/random-file.dat opened successfully on subvolume adata-client-6
[2013-02-25 20:02:50.049532] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 20:02:50.049567] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:50.784240] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-9: server 172.16.5.3:24013 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:02:50.784364] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-21: server 172.16.5.3:24019 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:02:50.784436] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.899081 (xid=0x296587x)
[2013-02-25 20:02:50.784479] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.784497] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 20:02:50.784516] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:50.784557] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.899127 (xid=0x296588x)
[2013-02-25 20:02:50.784568] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.794808] I [socket.c:2315:socket_submit_request] 0-adata-client-9: not connected (priv->connected = 0)
[2013-02-25 20:02:50.794823] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-9: failed to submit rpc-request (XID: 0x296593x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-9)
[2013-02-25 20:02:50.794849] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.794857] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-4: unable to lock on even one child
[2013-02-25 20:02:50.794874] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-4: Blocking inodelks failed.
[2013-02-25 20:02:50.794896] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-9: failed to submit rpc-request (XID: 0x296594x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-9)
[2013-02-25 20:02:50.794966] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.899218 (xid=0x296589x)
[2013-02-25 20:02:50.795002] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795011] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-4: Non blocking entrylks failed.
[2013-02-25 20:02:50.795020] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-4: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:50.795047] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.899260 (xid=0x296590x)
[2013-02-25 20:02:50.795060] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795073] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-9: failed to submit rpc-request (XID: 0x296595x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-9)
[2013-02-25 20:02:50.795082] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795088] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-4: unable to lock on even one child
[2013-02-25 20:02:50.795110] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-4: Blocking inodelks failed.
[2013-02-25 20:02:50.795125] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-9: failed to submit rpc-request (XID: 0x296596x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-9)
[2013-02-25 20:02:50.795174] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2013-02-25 20:02:03.190334 (xid=0x296591x)
[2013-02-25 20:02:50.795186] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795194] I [afr-self-heal-entry.c:780:afr_sh_entry_expunge_readdir_cbk] 0-adata-replicate-4: readdir of /files on subvolume adata-client-9 failed (Transport endpoint is not connected)
[2013-02-25 20:02:50.795237] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:02:08.778459 (xid=0x296592x)
[2013-02-25 20:02:50.795248] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-9: timer must have expired
[2013-02-25 20:02:50.795284] I [client.c:2090:client_rpc_notify] 0-adata-client-9: disconnected
[2013-02-25 20:02:50.795322] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.918381 (xid=0x381275x)
[2013-02-25 20:02:50.795333] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795341] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 20:02:50.795355] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:50.795382] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.918427 (xid=0x381276x)
[2013-02-25 20:02:50.795392] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795752] I [socket.c:2315:socket_submit_request] 0-adata-client-21: not connected (priv->connected = 0)
[2013-02-25 20:02:50.795767] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x381280x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-21)
[2013-02-25 20:02:50.795777] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795785] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-10: unable to lock on even one child
[2013-02-25 20:02:50.795792] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-10: Blocking inodelks failed.
[2013-02-25 20:02:50.795810] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x381281x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-21)
[2013-02-25 20:02:50.795873] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:11.858661 (xid=0x381278x)
[2013-02-25 20:02:50.795887] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795895] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-10: Non blocking entrylks failed.
[2013-02-25 20:02:50.795903] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-10: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:50.795929] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:11.858720 (xid=0x381279x)
[2013-02-25 20:02:50.795939] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795952] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x381282x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-21)
[2013-02-25 20:02:50.795961] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:50.795967] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-10: unable to lock on even one child
[2013-02-25 20:02:50.795973] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-10: Blocking inodelks failed.
[2013-02-25 20:02:50.795988] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-21: failed to submit rpc-request (XID: 0x381283x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-21)
[2013-02-25 20:02:50.796035] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:02:08.778481 (xid=0x381277x)
[2013-02-25 20:02:50.796053] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-21: timer must have expired
[2013-02-25 20:02:50.796065] I [client.c:2090:client_rpc_notify] 0-adata-client-21: disconnected
[2013-02-25 20:02:52.784717] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-3: server 172.16.5.3:24010 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:02:52.784895] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:01:48.865179 (xid=0x303686x)
[2013-02-25 20:02:52.784978] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:52.785030] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(READDIRP(40)) called at 2013-02-25 20:01:49.950891 (xid=0x303687x)
[2013-02-25 20:02:52.785041] W [client3_1-fops.c:2315:client3_1_readdirp_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:52.785049] I [afr-self-heal-entry.c:780:afr_sh_entry_expunge_readdir_cbk] 0-adata-replicate-1: readdir of /files on subvolume adata-client-3 failed (Transport endpoint is not connected)
[2013-02-25 20:02:52.785119] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:01:50.112731 (xid=0x303688x)
[2013-02-25 20:02:52.785131] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:52.785175] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:00.039956 (xid=0x303689x)
[2013-02-25 20:02:52.785204] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:52.785347] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:02:01.126639 (xid=0x303690x)
[2013-02-25 20:02:52.785362] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:52.785397] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:02:10.778770 (xid=0x303691x)
[2013-02-25 20:02:52.785421] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-3: timer must have expired
[2013-02-25 20:02:52.785434] I [client.c:2090:client_rpc_notify] 0-adata-client-3: disconnected
[2013-02-25 20:02:52.788860] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:52.788917] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:52.995301] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:52.995341] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:53.784887] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-1: server 172.16.5.3:24009 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:02:53.785150] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.898924 (xid=0x378587x)
[2013-02-25 20:02:53.785178] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.785190] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 20:02:53.785200] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:53.785233] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.898983 (xid=0x378588x)
[2013-02-25 20:02:53.785245] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.785848] I [socket.c:2315:socket_submit_request] 0-adata-client-1: not connected (priv->connected = 0)
[2013-02-25 20:02:53.785864] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378607x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.785874] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.785882] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-0: unable to lock on even one child
[2013-02-25 20:02:53.785890] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-0: Blocking inodelks failed.
[2013-02-25 20:02:53.785913] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378608x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.785972] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.899368 (xid=0x378589x)
[2013-02-25 20:02:53.785985] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.785993] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 20:02:53.786001] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:53.786028] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.899409 (xid=0x378590x)
[2013-02-25 20:02:53.786038] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786050] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378609x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786059] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786066] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-0: unable to lock on even one child
[2013-02-25 20:02:53.786079] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-0: Blocking inodelks failed.
[2013-02-25 20:02:53.786095] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378610x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786144] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.899525 (xid=0x378591x)
[2013-02-25 20:02:53.786156] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786163] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 20:02:53.786171] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:53.786196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.899565 (xid=0x378592x)
[2013-02-25 20:02:53.786206] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786218] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378611x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786227] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786233] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-0: unable to lock on even one child
[2013-02-25 20:02:53.786239] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-0: Blocking inodelks failed.
[2013-02-25 20:02:53.786254] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378612x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786299] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.899698 (xid=0x378593x)
[2013-02-25 20:02:53.786312] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786319] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 20:02:53.786327] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:53.786352] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.899836 (xid=0x378594x)
[2013-02-25 20:02:53.786361] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786380] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378613x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786390] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786396] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-0: unable to lock on even one child
[2013-02-25 20:02:53.786403] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-0: Blocking inodelks failed.
[2013-02-25 20:02:53.786417] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378614x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786464] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:02.899956 (xid=0x378595x)
[2013-02-25 20:02:53.786476] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786483] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 20:02:53.786491] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:53.786516] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:02.899998 (xid=0x378596x)
[2013-02-25 20:02:53.786525] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786537] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378615x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786546] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786552] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-0: unable to lock on even one child
[2013-02-25 20:02:53.786559] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-0: Blocking inodelks failed.
[2013-02-25 20:02:53.786573] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378616x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786619] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:02:09.485084 (xid=0x378597x)
[2013-02-25 20:02:53.786631] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786639] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-0: Non blocking entrylks failed.
[2013-02-25 20:02:53.786647] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-0: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:53.786678] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:02:09.485140 (xid=0x378598x)
[2013-02-25 20:02:53.786688] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786702] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378617x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786710] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:02:53.786717] I [afr-lk-common.c:1006:afr_lock_blocking] 0-adata-replicate-0: unable to lock on even one child
[2013-02-25 20:02:53.786723] I [afr-transaction.c:995:afr_post_blocking_inodelk_cbk] 0-adata-replicate-0: Blocking inodelks failed.
[2013-02-25 20:02:53.786737] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378618x Program: GlusterFS 3.1, ProgVers: 330, Proc: 41) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786784] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:02:11.778945 (xid=0x378599x)
[2013-02-25 20:02:53.786796] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-1: timer must have expired
[2013-02-25 20:02:53.786816] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:27.931056 (xid=0x378600x)
[2013-02-25 20:02:53.786828] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.786894] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378619x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.786909] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.787306] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:29.481819 (xid=0x378601x)
[2013-02-25 20:02:53.787320] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.787375] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378620x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.787390] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.787755] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:33.674571 (xid=0x378602x)
[2013-02-25 20:02:53.787776] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.787829] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378621x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.787844] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.788196] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:42.363258 (xid=0x378603x)
[2013-02-25 20:02:53.788211] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.788262] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378622x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.788277] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.788633] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:47.131516 (xid=0x378604x)
[2013-02-25 20:02:53.788647] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.788699] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-1: failed to submit rpc-request (XID: 0x378623x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-1)
[2013-02-25 20:02:53.788713] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:02:53.789070] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:52.785841 (xid=0x378605x)
[2013-02-25 20:02:53.789084] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:02:53.789126] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-1: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:52.945869 (xid=0x378606x)
[2013-02-25 20:02:53.789137] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-1: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:02:53.789173] I [client.c:2090:client_rpc_notify] 0-adata-client-1: disconnected
[2013-02-25 20:02:54.414695] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:54.414762] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.419633] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.419685] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.419725] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.419777] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.793561] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-8: Non blocking entrylks failed.
[2013-02-25 20:02:59.793612] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-8: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:59.794083] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-9: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:02:59.794231] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.794496] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.810764] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 20:02:59.810774] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:02:59.810792] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.810812] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.817726] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.817753] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:02:59.817853] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-9: Connected to 172.16.5.3:24013, attached to remote volume '/var/gfs/sdf/adata-s003-5'.
[2013-02-25 20:02:59.817872] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:02:59.817882] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-9: 18 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:02:59.889304] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-1: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:02:59.889944] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-1: Connected to 172.16.5.3:24009, attached to remote volume '/var/gfs/sdb/adata-s003-1'.
[2013-02-25 20:02:59.889972] I [client-handshake.c:1454:client_setvolume_cbk] 0-adata-client-1: Server and Client lk-version numbers are same, no need to reopen the fds
[2013-02-25 20:02:59.998654] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-adata-client-9: reopendir on /files succeeded (fd = 17)
[2013-02-25 20:02:59.998670] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:03:00.088675] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-7: Non blocking entrylks failed.
[2013-02-25 20:03:00.088705] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-7: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:03:00.091051] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:03:00.091065] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:03:00.091251] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:03:00.091265] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 2 subvolumes have unrecoverable errors
[2013-02-25 20:03:00.091365] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091383] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091417] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091440] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091452] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964198: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.091521] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091533] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091557] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091565] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091572] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964199: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.091665] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091677] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091699] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091706] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091714] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964200: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.091840] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091863] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091888] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091896] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091903] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964201: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.091963] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.091975] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.091997] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.092005] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.092012] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964202: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.092262] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.092276] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.092301] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.092309] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.092317] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964204: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.092465] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.092477] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.092501] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.092508] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.092516] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964207: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.429015] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.429042] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.429070] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:00.429078] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:00.429088] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964215: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:00.506882] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:03:00.506926] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:03:02.656651] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:03:02.656706] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:03:13.611055] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-21: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:03:13.611725] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-21: Connected to 172.16.5.3:24019, attached to remote volume '/var/gfs/sdl/adata-s003-11'.
[2013-02-25 20:03:13.611754] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-21: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:03:13.611770] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-21: 16 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:03:13.616101] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-21: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:03:13.616686] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-21: Server lk version = 1
[2013-02-25 20:03:16.694040] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-9: Server lk version = 1
[2013-02-25 20:03:16.897655] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.897680] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.897926] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.897943] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.897972] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.897980] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.897990] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964240: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.898120] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.898133] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.898297] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.898312] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.898338] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.898346] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.898354] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964241: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.898447] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.898468] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.898643] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.898659] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.898663] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.898701] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.898703] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.898727] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.898742] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964242: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.898795] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.898809] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.906225] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.906240] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.906318] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.906331] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.906475] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.906488] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.906505] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.906489] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.906545] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.906555] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.906563] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964252: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.906662] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.906676] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.906695] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.906711] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.906700] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.906740] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.906756] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964253: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.906850] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.906863] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.906888] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.906895] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.906902] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964254: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.907014] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907033] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907057] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907065] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907072] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964255: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.907136] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907148] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907171] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907179] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907186] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964256: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.907199] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.907216] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.907304] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.907319] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.907367] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907379] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907402] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907409] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907417] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964258: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:16.907429] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:03:16.907443] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 1 subvolumes have unrecoverable errors
[2013-02-25 20:03:16.907731] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907744] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907768] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:16.907776] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:16.907783] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964260: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:59.925579] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:03:59.926201] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:03:59.926824] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:03:59.927312] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:03:59.951877] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.951891] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.951931] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.951940] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.951947] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964300: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:59.952124] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952137] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952162] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952171] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952178] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964302: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:59.952339] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952354] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952390] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952403] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952414] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964303: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:59.952584] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952605] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952634] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952641] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952649] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964304: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:59.952797] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952809] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952833] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952840] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952848] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964305: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:03:59.952906] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952917] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952939] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:03:59.952946] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:03:59.952954] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964306: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:04:28.575695] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-3: Non blocking entrylks failed.
[2013-02-25 20:04:28.575748] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-3: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:04:34.086576] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-5: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 20:05:00.108549] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:05:00.201814] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:05:00.239048] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 20:05:00.239081] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:05:00.264095] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:05:00.265754] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:05:00.359694] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.359720] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.359752] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.359760] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.359769] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964367: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:05:00.359840] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.359852] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.359874] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.359881] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.359888] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964368: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:05:00.381564] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.381578] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.381601] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.381609] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.381616] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964371: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:05:00.381682] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.381694] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.381716] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:05:00.381724] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:05:00.381731] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964372: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:06.818500] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-5: server 172.16.5.3:24011 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:06.818780] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938577 (xid=0x306029x)
[2013-02-25 20:06:06.818820] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:06.818860] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.404855 (xid=0x306030x)
[2013-02-25 20:06:06.818872] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:06.818895] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861556 (xid=0x306031x)
[2013-02-25 20:06:06.818904] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:06.818925] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495317 (xid=0x306032x)
[2013-02-25 20:06:06.818934] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:06.818956] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:24.811088 (xid=0x306033x)
[2013-02-25 20:06:06.818965] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-5: timer must have expired
[2013-02-25 20:06:06.818985] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.052656 (xid=0x306034x)
[2013-02-25 20:06:06.818997] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:06.819061] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.869846 (xid=0x306035x)
[2013-02-25 20:06:06.819073] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:06.819119] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686209 (xid=0x306036x)
[2013-02-25 20:06:06.819137] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:06.819183] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.364786 (xid=0x306037x)
[2013-02-25 20:06:06.819195] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:06.819238] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448455 (xid=0x306038x)
[2013-02-25 20:06:06.819249] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:06.819293] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984525 (xid=0x306039x)
[2013-02-25 20:06:06.819303] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:06.819325] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.484729 (xid=0x306040x)
[2013-02-25 20:06:06.819335] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:06.819381] I [client.c:2090:client_rpc_notify] 0-adata-client-5: disconnected
[2013-02-25 20:06:16.820250] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-11: server 172.16.5.3:24014 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:16.820516] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938647 (xid=0x301369x)
[2013-02-25 20:06:16.820539] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:16.820572] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.404921 (xid=0x301370x)
[2013-02-25 20:06:16.820596] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:16.820622] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861624 (xid=0x301371x)
[2013-02-25 20:06:16.820631] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:16.820653] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495383 (xid=0x301372x)
[2013-02-25 20:06:16.820663] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:16.820685] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:34.812857 (xid=0x301373x)
[2013-02-25 20:06:16.820695] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-11: timer must have expired
[2013-02-25 20:06:16.820715] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.052812 (xid=0x301374x)
[2013-02-25 20:06:16.820728] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:16.820810] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.869961 (xid=0x301375x)
[2013-02-25 20:06:16.820823] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:16.820871] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686329 (xid=0x301376x)
[2013-02-25 20:06:16.820883] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:16.820928] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.364904 (xid=0x301377x)
[2013-02-25 20:06:16.820946] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:16.821001] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448583 (xid=0x301378x)
[2013-02-25 20:06:16.821014] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:16.821059] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984630 (xid=0x301379x)
[2013-02-25 20:06:16.821069] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:16.821091] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.484858 (xid=0x301380x)
[2013-02-25 20:06:16.821102] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:16.821132] I [client.c:2090:client_rpc_notify] 0-adata-client-11: disconnected
[2013-02-25 20:06:23.831868] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-9: server 172.16.5.3:24013 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:23.831996] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-15: server 172.16.5.3:24016 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:23.832078] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-17: server 172.16.5.3:24017 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:23.832135] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:59.825862 (xid=0x297108x)
[2013-02-25 20:06:23.832165] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:23.832218] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:59.825884 (xid=0x297109x)
[2013-02-25 20:06:23.832231] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:23.840229] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:23.840254] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:02:59.832187 (xid=0x297327x)
[2013-02-25 20:06:23.840264] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:23.840325] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938625 (xid=0x297433x)
[2013-02-25 20:06:23.840343] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840372] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.404898 (xid=0x297434x)
[2013-02-25 20:06:23.840384] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840406] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861602 (xid=0x297435x)
[2013-02-25 20:06:23.840414] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840435] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495360 (xid=0x297436x)
[2013-02-25 20:06:23.840443] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840464] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:41.814019 (xid=0x297437x)
[2013-02-25 20:06:23.840473] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-9: timer must have expired
[2013-02-25 20:06:23.840492] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.052762 (xid=0x297438x)
[2013-02-25 20:06:23.840502] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.840559] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.869922 (xid=0x297439x)
[2013-02-25 20:06:23.840570] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.840613] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686290 (xid=0x297440x)
[2013-02-25 20:06:23.840631] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.840676] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.364865 (xid=0x297441x)
[2013-02-25 20:06:23.840687] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.840730] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448538 (xid=0x297442x)
[2013-02-25 20:06:23.840742] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.840784] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984596 (xid=0x297443x)
[2013-02-25 20:06:23.840795] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840816] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.484817 (xid=0x297444x)
[2013-02-25 20:06:23.840826] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.840857] I [client.c:2090:client_rpc_notify] 0-adata-client-9: disconnected
[2013-02-25 20:06:23.840896] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.404965 (xid=0x312033x)
[2013-02-25 20:06:23.840906] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840929] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861668 (xid=0x312034x)
[2013-02-25 20:06:23.840937] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840958] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495426 (xid=0x312035x)
[2013-02-25 20:06:23.840973] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.840995] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:41.814043 (xid=0x312036x)
[2013-02-25 20:06:23.841004] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-15: timer must have expired
[2013-02-25 20:06:23.841023] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.052907 (xid=0x312037x)
[2013-02-25 20:06:23.841033] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841076] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.870039 (xid=0x312038x)
[2013-02-25 20:06:23.841088] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841129] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686408 (xid=0x312039x)
[2013-02-25 20:06:23.841141] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841182] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.364983 (xid=0x312040x)
[2013-02-25 20:06:23.841193] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841246] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448662 (xid=0x312041x)
[2013-02-25 20:06:23.841260] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841304] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984699 (xid=0x312042x)
[2013-02-25 20:06:23.841320] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.841344] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.484940 (xid=0x312043x)
[2013-02-25 20:06:23.841354] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841382] I [client.c:2090:client_rpc_notify] 0-adata-client-15: disconnected
[2013-02-25 20:06:23.841416] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938714 (xid=0x304485x)
[2013-02-25 20:06:23.841426] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.841448] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.404987 (xid=0x304486x)
[2013-02-25 20:06:23.841456] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.841477] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861690 (xid=0x304487x)
[2013-02-25 20:06:23.841485] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.841506] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495447 (xid=0x304488x)
[2013-02-25 20:06:23.841514] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.841535] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:41.814060 (xid=0x304489x)
[2013-02-25 20:06:23.841544] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-17: timer must have expired
[2013-02-25 20:06:23.841562] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.052959 (xid=0x304490x)
[2013-02-25 20:06:23.841578] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841623] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.870078 (xid=0x304491x)
[2013-02-25 20:06:23.841635] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841677] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686450 (xid=0x304492x)
[2013-02-25 20:06:23.841688] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841730] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.365021 (xid=0x304493x)
[2013-02-25 20:06:23.841741] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841783] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448701 (xid=0x304494x)
[2013-02-25 20:06:23.841794] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841834] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984735 (xid=0x304495x)
[2013-02-25 20:06:23.841844] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:23.841866] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-17: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.484983 (xid=0x304496x)
[2013-02-25 20:06:23.841876] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-17: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:23.841931] I [client.c:2090:client_rpc_notify] 0-adata-client-17: disconnected
[2013-02-25 20:06:23.843951] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:23.843994] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:23.844039] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:24.013706] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:24.013747] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:24.832233] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-21: server 172.16.5.3:24019 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:24.832396] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-19: server 172.16.5.3:24018 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:24.832422] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-13: server 172.16.5.3:24015 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:06:24.832524] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938759 (xid=0x381384x)
[2013-02-25 20:06:24.832545] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.832574] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.405030 (xid=0x381385x)
[2013-02-25 20:06:24.832583] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.832604] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861734 (xid=0x381386x)
[2013-02-25 20:06:24.832613] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.832647] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495493 (xid=0x381387x)
[2013-02-25 20:06:24.832656] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.832677] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:42.814310 (xid=0x381388x)
[2013-02-25 20:06:24.832686] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-21: timer must have expired
[2013-02-25 20:06:24.832706] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.053057 (xid=0x381389x)
[2013-02-25 20:06:24.832717] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.832777] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.870164 (xid=0x381390x)
[2013-02-25 20:06:24.832789] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.832834] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686530 (xid=0x381391x)
[2013-02-25 20:06:24.832846] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.832890] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.365100 (xid=0x381392x)
[2013-02-25 20:06:24.832902] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.832946] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448782 (xid=0x381393x)
[2013-02-25 20:06:24.832964] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833011] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984804 (xid=0x381394x)
[2013-02-25 20:06:24.833021] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833042] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.485069 (xid=0x381395x)
[2013-02-25 20:06:24.833053] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833086] I [client.c:2090:client_rpc_notify] 0-adata-client-21: disconnected
[2013-02-25 20:06:24.833124] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938736 (xid=0x306086x)
[2013-02-25 20:06:24.833135] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833157] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.405008 (xid=0x306087x)
[2013-02-25 20:06:24.833166] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833187] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 20:05:11.523205 (xid=0x306088x)
[2013-02-25 20:06:24.833195] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861711 (xid=0x306089x)
[2013-02-25 20:06:24.833263] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833288] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495469 (xid=0x306090x)
[2013-02-25 20:06:24.833306] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833329] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:42.814333 (xid=0x306091x)
[2013-02-25 20:06:24.833338] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-19: timer must have expired
[2013-02-25 20:06:24.833357] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.053008 (xid=0x306092x)
[2013-02-25 20:06:24.833367] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833413] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.870122 (xid=0x306093x)
[2013-02-25 20:06:24.833425] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833467] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686490 (xid=0x306094x)
[2013-02-25 20:06:24.833478] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833519] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.365060 (xid=0x306095x)
[2013-02-25 20:06:24.833531] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833573] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448742 (xid=0x306096x)
[2013-02-25 20:06:24.833584] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.833625] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984768 (xid=0x306097x)
[2013-02-25 20:06:24.833641] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.833664] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.485024 (xid=0x306098x)
[2013-02-25 20:06:24.833674] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.834001] I [client.c:2090:client_rpc_notify] 0-adata-client-19: disconnected
[2013-02-25 20:06:24.834040] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:09.938669 (xid=0x299276x)
[2013-02-25 20:06:24.834050] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.834143] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:10.404943 (xid=0x299277x)
[2013-02-25 20:06:24.834162] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.834202] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:13.861646 (xid=0x299278x)
[2013-02-25 20:06:24.834212] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.834244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:05:14.495405 (xid=0x299279x)
[2013-02-25 20:06:24.834254] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.834286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:05:42.814349 (xid=0x299280x)
[2013-02-25 20:06:24.834295] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-13: timer must have expired
[2013-02-25 20:06:24.834315] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.052859 (xid=0x299281x)
[2013-02-25 20:06:24.834333] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.834426] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:43.869999 (xid=0x299282x)
[2013-02-25 20:06:24.834441] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.834548] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:51.686369 (xid=0x299283x)
[2013-02-25 20:06:24.834564] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.834652] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:05:53.364943 (xid=0x299284x)
[2013-02-25 20:06:24.834669] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.834765] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:01.448623 (xid=0x299285x)
[2013-02-25 20:06:24.834781] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.834906] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:06:01.984664 (xid=0x299286x)
[2013-02-25 20:06:24.834946] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:06:24.835000] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:06:04.484898 (xid=0x299287x)
[2013-02-25 20:06:24.835036] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:06:24.835170] I [client.c:2090:client_rpc_notify] 0-adata-client-13: disconnected
[2013-02-25 20:06:24.839313] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:06:24.839336] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 5 subvolumes have unrecoverable errors
[2013-02-25 20:06:24.839487] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:06:24.839501] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 5 subvolumes have unrecoverable errors
[2013-02-25 20:06:24.839809] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:06:24.839824] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 5 subvolumes have unrecoverable errors
[2013-02-25 20:06:24.841071] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:06:24.841089] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 5 subvolumes have unrecoverable errors
[2013-02-25 20:06:24.841310] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:06:24.841328] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 5 subvolumes have unrecoverable errors
[2013-02-25 20:06:24.841417] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:06:24.841433] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 5 subvolumes have unrecoverable errors
[2013-02-25 20:06:24.841469] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2632522787
[2013-02-25 20:06:24.841485] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.841510] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2632522787
[2013-02-25 20:06:24.841518] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.841528] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964407: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.841657] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 20:06:24.841670] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.841694] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 20:06:24.841701] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.841709] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964409: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.897916] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:06:24.897935] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.897960] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:06:24.897968] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.897976] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964416: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.898111] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:06:24.898123] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.898146] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:06:24.898154] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.898161] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964418: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.898505] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2632522787
[2013-02-25 20:06:24.898520] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.898529] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964419: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.898739] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2632522787
[2013-02-25 20:06:24.898752] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.898760] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964422: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.899002] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 20:06:24.899016] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.899040] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 20:06:24.899049] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.899057] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964423: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:24.899230] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 20:06:24.899245] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.899269] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3014619399
[2013-02-25 20:06:24.899277] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:06:24.899285] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964424: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:06:25.515709] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:25.515773] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:25.515812] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:25.515858] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:26.451462] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:06:26.451641] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:06:26.451726] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:06:26.451757] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:06:26.452100] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:06:27.291276] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:27.308098] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:27.308155] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:27.479284] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:06:27.479444] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:06:27.479499] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:06:27.479568] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:06:27.479643] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:06:28.764399] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:06:28.764603] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:06:28.764669] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:06:28.764699] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:06:28.764757] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:06:28.814830] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:28.831428] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:28.831469] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:06:30.243300] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:06:30.243440] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:06:30.243522] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:06:30.243558] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:06:30.243592] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:06:30.372289] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.309754] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-9: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.309922] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.310247] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.310456] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.310574] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.310725] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.310833] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.310982] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311098] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311129] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311170] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311272] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311351] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311437] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311530] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.311652] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.312995] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-11: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.313578] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-11: Connected to 172.16.5.3:24014, attached to remote volume '/var/gfs/sdg/adata-s003-6'.
[2013-02-25 20:07:07.313600] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-11: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.313613] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-11: 1 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:07:07.316502] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-5: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.317095] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-5: Connected to 172.16.5.3:24011, attached to remote volume '/var/gfs/sdd/adata-s003-3'.
[2013-02-25 20:07:07.317117] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-5: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.317133] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-5: 21 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:07:07.317365] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-21: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.317849] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-21: Connected to 172.16.5.3:24019, attached to remote volume '/var/gfs/sdl/adata-s003-11'.
[2013-02-25 20:07:07.317864] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-21: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.317873] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-21: 16 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:07:07.319456] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-5: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:07:07.320931] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-21: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:07:07.321153] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-5: Server lk version = 1
[2013-02-25 20:07:07.322152] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-15: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.322512] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-15: Connected to 172.16.5.3:24016, attached to remote volume '/var/gfs/sdi/adata-s003-8'.
[2013-02-25 20:07:07.322526] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-15: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.322532] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-15: 2 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:07:07.322631] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-21: Server lk version = 1
[2013-02-25 20:07:07.325008] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-15: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:07:07.330134] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-11: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:07:07.330318] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-11: Server lk version = 1
[2013-02-25 20:07:07.339860] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-13: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.340423] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-13: Connected to 172.16.5.3:24015, attached to remote volume '/var/gfs/sdh/adata-s003-7'.
[2013-02-25 20:07:07.340443] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-13: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.340721] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-13: Server lk version = 1
[2013-02-25 20:07:07.343453] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-19: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.344017] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-19: Connected to 172.16.5.3:24018, attached to remote volume '/var/gfs/sdk/adata-s003-10'.
[2013-02-25 20:07:07.344040] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-19: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.344054] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-19: 10 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:07:07.351750] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:07:07.351779] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:07:07.352746] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.352765] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.352808] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.352827] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.352842] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964534: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.352975] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.352992] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.353031] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.353046] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.353072] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964535: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.354905] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3009663882
[2013-02-25 20:07:07.354919] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.354943] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3009663882
[2013-02-25 20:07:07.354951] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.354959] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964538: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.355085] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3009663882
[2013-02-25 20:07:07.355097] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.355105] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964539: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.355171] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3009663882
[2013-02-25 20:07:07.355185] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.355194] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964541: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.356092] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.356224] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.356471] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.356545] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.356684] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.356864] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357009] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357145] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357341] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357503] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357564] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357645] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.357692] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.358442] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.359473] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.359732] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.376069] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-15: Server lk version = 1
[2013-02-25 20:07:07.376160] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:07:07.376184] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:07:07.376348] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=2 overlaps=0
[2013-02-25 20:07:07.376369] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:07:07.376559] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:07:07.376576] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 3 subvolumes have unrecoverable errors
[2013-02-25 20:07:07.376784] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.376787] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:07:07.376807] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.376822] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 3 subvolumes have unrecoverable errors
[2013-02-25 20:07:07.376863] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.376877] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.376890] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964549: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.377174] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.377187] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.377211] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.377219] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.377271] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964552: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.377441] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.377454] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.377477] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.377485] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.377501] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964553: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.377702] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.377715] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.377737] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.377745] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.377753] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964556: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.378311] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.378524] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.383886] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.383906] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.383927] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.383962] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384007] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=1 overlaps=0
[2013-02-25 20:07:07.384018] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 3 subvolumes have unrecoverable errors
[2013-02-25 20:07:07.384166] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384220] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384261] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-19: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:07:07.384319] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384352] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384390] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384412] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384445] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384492] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384549] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384578] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384600] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384621] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384653] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384688] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384715] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384749] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384769] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-19: Server lk version = 1
[2013-02-25 20:07:07.384783] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.384803] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.475346] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.475410] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.475524] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-9: Connected to 172.16.5.3:24013, attached to remote volume '/var/gfs/sdf/adata-s003-5'.
[2013-02-25 20:07:07.475540] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.475550] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-9: 18 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:07:07.488235] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.488250] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.488277] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.488284] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.488292] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964558: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.488357] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.488368] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.488391] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:07:07.488398] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:07:07.488405] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964559: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:07:07.510950] I [client-handshake.c:1041:client3_1_reopendir_cbk] 0-adata-client-9: reopendir on /files succeeded (fd = 0)
[2013-02-25 20:07:07.541812] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-3: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:07:07.541922] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.541973] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.542339] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.562735] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.562771] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:07:07.562841] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-3: Connected to 172.16.5.3:24010, attached to remote volume '/var/gfs/sdc/adata-s003-2'.
[2013-02-25 20:07:07.562854] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:07:07.562863] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-3: 21 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:31.857766] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-11: server 172.16.5.3:24014 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.857877] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-5: server 172.16.5.3:24011 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.857925] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-21: server 172.16.5.3:24019 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.857976] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-15: server 172.16.5.3:24016 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.857986] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635635 (xid=0x301393x)
[2013-02-25 20:08:31.858019] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-13: server 172.16.5.3:24015 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.858024] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858056] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-19: server 172.16.5.3:24018 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.858093] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044230 (xid=0x301394x)
[2013-02-25 20:08:31.858114] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:07:17.858093] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-9: server 172.16.5.3:24013 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.858156] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810120 (xid=0x301395x)
[2013-02-25 20:08:31.858214] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-3: server 172.16.5.3:24010 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:31.858225] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858277] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.397893 (xid=0x301396x)
[2013-02-25 20:08:31.858297] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858327] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235125 (xid=0x301397x)
[2013-02-25 20:08:31.858339] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858365] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230527 (xid=0x301398x)
[2013-02-25 20:08:31.858375] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858396] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971302 (xid=0x301399x)
[2013-02-25 20:08:31.858408] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.858473] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446535 (xid=0x301400x)
[2013-02-25 20:08:31.858485] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.858529] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851214 (xid=0x301401x)
[2013-02-25 20:08:31.858539] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-11: timer must have expired
[2013-02-25 20:08:31.858559] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704194 (xid=0x301402x)
[2013-02-25 20:08:31.858569] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.858613] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406122 (xid=0x301403x)
[2013-02-25 20:08:31.858629] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858652] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.188085 (xid=0x301404x)
[2013-02-25 20:08:31.858662] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.858706] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.027836 (xid=0x301405x)
[2013-02-25 20:08:31.858717] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.858760] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111853 (xid=0x301406x)
[2013-02-25 20:08:31.858769] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858779] I [client.c:2090:client_rpc_notify] 0-adata-client-11: disconnected
[2013-02-25 20:08:31.858815] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:07:08.262800 (xid=0x306107x)
[2013-02-25 20:08:31.858826] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.858878] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:08.725570 (xid=0x306108x)
[2013-02-25 20:08:31.858891] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.858946] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:09.272054 (xid=0x306109x)
[2013-02-25 20:08:31.858959] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859032] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635575 (xid=0x306110x)
[2013-02-25 20:08:31.859047] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859071] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044178 (xid=0x306111x)
[2013-02-25 20:08:31.859087] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859111] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810061 (xid=0x306112x)
[2013-02-25 20:08:31.859120] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859141] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.397834 (xid=0x306113x)
[2013-02-25 20:08:31.859150] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859170] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235066 (xid=0x306114x)
[2013-02-25 20:08:31.859179] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859200] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230467 (xid=0x306115x)
[2013-02-25 20:08:31.859208] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971174 (xid=0x306116x)
[2013-02-25 20:08:31.859239] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446400 (xid=0x306117x)
[2013-02-25 20:08:31.859302] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859343] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851242 (xid=0x306118x)
[2013-02-25 20:08:31.859353] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-5: timer must have expired
[2013-02-25 20:08:31.859372] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704032 (xid=0x306119x)
[2013-02-25 20:08:31.859382] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859421] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406060 (xid=0x306120x)
[2013-02-25 20:08:31.859432] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859453] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.187950 (xid=0x306121x)
[2013-02-25 20:08:31.859463] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859503] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.027701 (xid=0x306122x)
[2013-02-25 20:08:31.859514] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859554] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-5: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111797 (xid=0x306123x)
[2013-02-25 20:08:31.859564] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-5: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859583] I [client.c:2090:client_rpc_notify] 0-adata-client-5: disconnected
[2013-02-25 20:08:31.859614] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:08.725962 (xid=0x381438x)
[2013-02-25 20:08:31.859626] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859671] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:09.272601 (xid=0x381439x)
[2013-02-25 20:08:31.859682] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859723] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635767 (xid=0x381440x)
[2013-02-25 20:08:31.859733] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859755] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044349 (xid=0x381441x)
[2013-02-25 20:08:31.859763] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859784] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810252 (xid=0x381442x)
[2013-02-25 20:08:31.859793] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859813] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.398037 (xid=0x381443x)
[2013-02-25 20:08:31.859822] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859842] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235252 (xid=0x381444x)
[2013-02-25 20:08:31.859857] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859879] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230644 (xid=0x381445x)
[2013-02-25 20:08:31.859888] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.859909] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971573 (xid=0x381446x)
[2013-02-25 20:08:31.859918] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.859960] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446850 (xid=0x381447x)
[2013-02-25 20:08:31.859972] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860012] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851258 (xid=0x381448x)
[2013-02-25 20:08:31.860022] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-21: timer must have expired
[2013-02-25 20:08:31.860040] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704619 (xid=0x381449x)
[2013-02-25 20:08:31.860051] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860091] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406223 (xid=0x381450x)
[2013-02-25 20:08:31.860101] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860123] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.188371 (xid=0x381451x)
[2013-02-25 20:08:31.860139] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860180] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.028130 (xid=0x381452x)
[2013-02-25 20:08:31.860192] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860232] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111953 (xid=0x381453x)
[2013-02-25 20:08:31.860242] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860252] I [client.c:2090:client_rpc_notify] 0-adata-client-21: disconnected
[2013-02-25 20:08:31.860282] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:07:33.971743 (xid=0x312064x)
[2013-02-25 20:08:31.860292] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860300] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-7: Non blocking entrylks failed.
[2013-02-25 20:08:31.860309] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-7: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:08:31.860334] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:09.272371 (xid=0x312056x)
[2013-02-25 20:08:31.860345] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860388] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635695 (xid=0x312057x)
[2013-02-25 20:08:31.860398] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860419] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044275 (xid=0x312058x)
[2013-02-25 20:08:31.860428] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860456] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810186 (xid=0x312059x)
[2013-02-25 20:08:31.860466] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860487] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.397955 (xid=0x312060x)
[2013-02-25 20:08:31.860495] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860516] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235183 (xid=0x312061x)
[2013-02-25 20:08:31.860524] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860545] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230584 (xid=0x312062x)
[2013-02-25 20:08:31.860553] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860574] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971420 (xid=0x312063x)
[2013-02-25 20:08:31.860584] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860626] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 20:07:33.971759 (xid=0x312065x)
[2013-02-25 20:08:31.860636] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860681] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446652 (xid=0x312066x)
[2013-02-25 20:08:31.860694] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860751] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851273 (xid=0x312067x)
[2013-02-25 20:08:31.860763] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-15: timer must have expired
[2013-02-25 20:08:31.860782] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704386 (xid=0x312068x)
[2013-02-25 20:08:31.860792] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860834] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406166 (xid=0x312069x)
[2013-02-25 20:08:31.860844] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860866] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.188178 (xid=0x312070x)
[2013-02-25 20:08:31.860876] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860916] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.027941 (xid=0x312071x)
[2013-02-25 20:08:31.860927] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.860967] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-15: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111897 (xid=0x312072x)
[2013-02-25 20:08:31.860977] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-15: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.860987] I [client.c:2090:client_rpc_notify] 0-adata-client-15: disconnected
[2013-02-25 20:08:31.861017] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:08.725766 (xid=0x299297x)
[2013-02-25 20:08:31.861035] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861186] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:09.272317 (xid=0x299298x)
[2013-02-25 20:08:31.861200] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861253] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635670 (xid=0x299299x)
[2013-02-25 20:08:31.861266] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861289] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044253 (xid=0x299300x)
[2013-02-25 20:08:31.861298] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861320] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810146 (xid=0x299301x)
[2013-02-25 20:08:31.861329] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861351] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.397929 (xid=0x299302x)
[2013-02-25 20:08:31.861364] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861391] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235154 (xid=0x299303x)
[2013-02-25 20:08:31.861404] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861425] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230555 (xid=0x299304x)
[2013-02-25 20:08:31.861440] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861462] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971365 (xid=0x299305x)
[2013-02-25 20:08:31.861473] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861516] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446578 (xid=0x299306x)
[2013-02-25 20:08:31.861528] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861587] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851288 (xid=0x299307x)
[2013-02-25 20:08:31.861600] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-13: timer must have expired
[2013-02-25 20:08:31.861622] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704267 (xid=0x299308x)
[2013-02-25 20:08:31.861638] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861683] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406144 (xid=0x299309x)
[2013-02-25 20:08:31.861694] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861716] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.188132 (xid=0x299310x)
[2013-02-25 20:08:31.861726] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861766] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.027880 (xid=0x299311x)
[2013-02-25 20:08:31.861784] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.861826] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-13: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111875 (xid=0x299312x)
[2013-02-25 20:08:31.861836] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-13: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861845] I [client.c:2090:client_rpc_notify] 0-adata-client-13: disconnected
[2013-02-25 20:08:31.861876] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:07:22.540640 (xid=0x306128x)
[2013-02-25 20:08:31.861886] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861893] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 20:08:31.861901] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:08:31.861925] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(ENTRYLK(31)) called at 2013-02-25 20:08:04.118699 (xid=0x306136x)
[2013-02-25 20:08:31.861935] W [client3_1-fops.c:1595:client3_1_entrylk_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.861941] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-adata-replicate-9: Non blocking entrylks failed.
[2013-02-25 20:08:31.861949] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-adata-replicate-9: background  data missing-entry gfid self-heal failed on /files/random-file.dat
[2013-02-25 20:08:31.861971] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:09.272519 (xid=0x306122x)
[2013-02-25 20:08:31.861982] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.862121] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635740 (xid=0x306123x)
[2013-02-25 20:08:31.862134] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862158] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044315 (xid=0x306124x)
[2013-02-25 20:08:31.862175] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862199] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810227 (xid=0x306125x)
[2013-02-25 20:08:31.862208] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.398009 (xid=0x306126x)
[2013-02-25 20:08:31.862238] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862262] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235228 (xid=0x306127x)
[2013-02-25 20:08:31.862275] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862301] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 20:07:22.540650 (xid=0x306129x)
[2013-02-25 20:08:31.862310] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862354] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230622 (xid=0x306130x)
[2013-02-25 20:08:31.862365] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862388] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971517 (xid=0x306131x)
[2013-02-25 20:08:31.862404] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.862465] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446773 (xid=0x306132x)
[2013-02-25 20:08:31.862487] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.862531] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851304 (xid=0x306133x)
[2013-02-25 20:08:31.862541] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-19: timer must have expired
[2013-02-25 20:08:31.862560] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704525 (xid=0x306134x)
[2013-02-25 20:08:31.862571] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.862632] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406201 (xid=0x306135x)
[2013-02-25 20:08:31.862651] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862698] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.188291 (xid=0x306137x)
[2013-02-25 20:08:31.862728] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.862807] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.028067 (xid=0x306138x)
[2013-02-25 20:08:31.862835] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:31.862903] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-19: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111931 (xid=0x306139x)
[2013-02-25 20:08:31.862919] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-19: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:31.862954] I [client.c:2090:client_rpc_notify] 0-adata-client-19: disconnected
[2013-02-25 20:08:31.863148] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475682 (xid=0x297813x)
[2013-02-25 20:08:31.863191] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863220] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475687 (xid=0x297814x)
[2013-02-25 20:08:31.863248] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863292] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475691 (xid=0x297815x)
[2013-02-25 20:08:31.863305] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863345] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475696 (xid=0x297816x)
[2013-02-25 20:08:31.863358] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863397] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475701 (xid=0x297817x)
[2013-02-25 20:08:31.863412] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863450] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475706 (xid=0x297818x)
[2013-02-25 20:08:31.863464] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863506] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475710 (xid=0x297819x)
[2013-02-25 20:08:31.863518] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863559] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475715 (xid=0x297820x)
[2013-02-25 20:08:31.863594] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863638] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.475720 (xid=0x297821x)
[2013-02-25 20:08:31.863651] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-9: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.863659] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:31.873482] I [socket.c:2315:socket_submit_request] 0-adata-client-9: not connected (priv->connected = 0)
[2013-02-25 20:08:31.873496] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-9: failed to submit rpc-request (XID: 0x298187x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (adata-client-9)
[2013-02-25 20:08:31.873504] W [client-handshake.c:443:client_set_lk_version_cbk] 0-adata-client-9: received RPC status error
[2013-02-25 20:08:31.873536] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2013-02-25 20:07:07.511030 (xid=0x297822x)
[2013-02-25 20:08:31.873580] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2013-02-25 20:07:07.511079 (xid=0x297823x)
[2013-02-25 20:08:31.873607] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2013-02-25 20:07:07.511094 (xid=0x297824x)
[2013-02-25 20:08:31.873634] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(RELEASE(41)) called at 2013-02-25 20:07:07.511106 (xid=0x297825x)
[2013-02-25 20:08:31.887556] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887581] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.608845 (xid=0x298177x)
[2013-02-25 20:08:31.887591] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887616] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.608881 (xid=0x298178x)
[2013-02-25 20:08:31.887626] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887650] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.608902 (xid=0x298179x)
[2013-02-25 20:08:31.887660] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887685] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.608924 (xid=0x298180x)
[2013-02-25 20:08:31.887695] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887719] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.608959 (xid=0x298181x)
[2013-02-25 20:08:31.887730] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887763] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.608984 (xid=0x298182x)
[2013-02-25 20:08:31.887775] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887800] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.609006 (xid=0x298183x)
[2013-02-25 20:08:31.887810] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887835] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.609040 (xid=0x298184x)
[2013-02-25 20:08:31.887845] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887870] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.609063 (xid=0x298185x)
[2013-02-25 20:08:31.887880] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-9: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.887931] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-9: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851310 (xid=0x298186x)
[2013-02-25 20:08:31.887943] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-9: timer must have expired
[2013-02-25 20:08:31.887951] I [client.c:2090:client_rpc_notify] 0-adata-client-9: disconnected
[2013-02-25 20:08:31.887988] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562886 (xid=0x304050x)
[2013-02-25 20:08:31.887999] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888020] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562903 (xid=0x304052x)
[2013-02-25 20:08:31.888034] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888059] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562911 (xid=0x304053x)
[2013-02-25 20:08:31.888067] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888086] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562919 (xid=0x304054x)
[2013-02-25 20:08:31.888094] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888112] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562927 (xid=0x304055x)
[2013-02-25 20:08:31.888120] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888139] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562937 (xid=0x304056x)
[2013-02-25 20:08:31.888146] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888165] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562948 (xid=0x304057x)
[2013-02-25 20:08:31.888172] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888191] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562957 (xid=0x304058x)
[2013-02-25 20:08:31.888199] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562965 (xid=0x304059x)
[2013-02-25 20:08:31.888225] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888243] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562970 (xid=0x304060x)
[2013-02-25 20:08:31.888256] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888276] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562973 (xid=0x304061x)
[2013-02-25 20:08:31.888284] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888302] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562977 (xid=0x304062x)
[2013-02-25 20:08:31.888310] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888328] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562981 (xid=0x304063x)
[2013-02-25 20:08:31.888336] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888354] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562986 (xid=0x304064x)
[2013-02-25 20:08:31.888362] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888380] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562992 (xid=0x304065x)
[2013-02-25 20:08:31.888388] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888407] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562995 (xid=0x304066x)
[2013-02-25 20:08:31.888414] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888433] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.562999 (xid=0x304067x)
[2013-02-25 20:08:31.888446] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888465] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.563002 (xid=0x304068x)
[2013-02-25 20:08:31.888473] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888491] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.563006 (xid=0x304069x)
[2013-02-25 20:08:31.888499] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888518] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPEN(11)) called at 2013-02-25 20:07:07.563009 (xid=0x304070x)
[2013-02-25 20:08:31.888526] W [client-handshake.c:913:client3_1_reopen_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888544] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(OPENDIR(20)) called at 2013-02-25 20:07:07.563013 (xid=0x304071x)
[2013-02-25 20:08:31.888552] W [client-handshake.c:1020:client3_1_reopendir_cbk] 0-adata-client-3: received RPC status error, returning ENOTCONN
[2013-02-25 20:08:31.888558] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:31.888998] I [socket.c:2315:socket_submit_request] 0-adata-client-3: not connected (priv->connected = 0)
[2013-02-25 20:08:31.889013] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-3: failed to submit rpc-request (XID: 0x304429x Program: GlusterFS Handshake, ProgVers: 2, Proc: 4) to rpc-transport (adata-client-3)
[2013-02-25 20:08:31.889022] W [client-handshake.c:443:client_set_lk_version_cbk] 0-adata-client-3: received RPC status error
[2013-02-25 20:08:31.889052] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.681520 (xid=0x304072x)
[2013-02-25 20:08:31.889065] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.889096] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.681549 (xid=0x304073x)
[2013-02-25 20:08:31.889115] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902077] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692704 (xid=0x304417x)
[2013-02-25 20:08:31.902087] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902113] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692720 (xid=0x304418x)
[2013-02-25 20:08:31.902123] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902148] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692735 (xid=0x304419x)
[2013-02-25 20:08:31.902159] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902185] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692750 (xid=0x304420x)
[2013-02-25 20:08:31.902201] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902229] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692799 (xid=0x304421x)
[2013-02-25 20:08:31.902239] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902265] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692824 (xid=0x304422x)
[2013-02-25 20:08:31.902276] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902301] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692850 (xid=0x304423x)
[2013-02-25 20:08:31.902312] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902337] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692875 (xid=0x304424x)
[2013-02-25 20:08:31.902348] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902374] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692910 (xid=0x304425x)
[2013-02-25 20:08:31.902384] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902409] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692937 (xid=0x304426x)
[2013-02-25 20:08:31.902428] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902455] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:07.692960 (xid=0x304427x)
[2013-02-25 20:08:31.902466] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:31.902521] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:49.851324 (xid=0x304428x)
[2013-02-25 20:08:31.902533] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-3: timer must have expired
[2013-02-25 20:08:31.902541] I [client.c:2090:client_rpc_notify] 0-adata-client-3: disconnected
[2013-02-25 20:08:31.903281] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-5:  (0d93f435-6b4a-4eac-a589-88cbb5317259) remote_fd is -1. EBADFD
[2013-02-25 20:08:31.903313] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-5:  (0d93f435-6b4a-4eac-a589-88cbb5317259) remote_fd is -1. EBADFD
[2013-02-25 20:08:33.858483] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-7: server 172.16.5.3:24012 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:08:33.858698] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2013-02-25 20:07:36.408297 (xid=0x330149x)
[2013-02-25 20:08:33.858719] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.858783] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 20:07:15.606532 (xid=0x330140x)
[2013-02-25 20:08:33.858799] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.858849] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:16.635597 (xid=0x330141x)
[2013-02-25 20:08:33.858863] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859021] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.044203 (xid=0x330142x)
[2013-02-25 20:08:33.859048] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859086] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:17.810082 (xid=0x330143x)
[2013-02-25 20:08:33.859096] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859129] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:21.397855 (xid=0x330144x)
[2013-02-25 20:08:33.859139] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859173] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:22.235088 (xid=0x330145x)
[2013-02-25 20:08:33.859184] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859216] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:07:26.230489 (xid=0x330146x)
[2013-02-25 20:08:33.859226] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859259] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(READ(12)) called at 2013-02-25 20:07:31.966983 (xid=0x330147x)
[2013-02-25 20:08:33.859269] W [client3_1-fops.c:2700:client3_1_readv_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.859327] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:31.971218 (xid=0x330148x)
[2013-02-25 20:08:33.859348] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:33.859504] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:48.446447 (xid=0x330150x)
[2013-02-25 20:08:33.859528] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:33.859723] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:07:51.851693 (xid=0x330151x)
[2013-02-25 20:08:33.859746] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-7: timer must have expired
[2013-02-25 20:08:33.859781] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:07:54.704086 (xid=0x330152x)
[2013-02-25 20:08:33.859800] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:33.859946] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:01.406084 (xid=0x330153x)
[2013-02-25 20:08:33.859967] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.860024] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:10.187996 (xid=0x330154x)
[2013-02-25 20:08:33.860046] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:33.860248] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:19.027746 (xid=0x330155x)
[2013-02-25 20:08:33.860274] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:08:33.860403] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:08:29.111820 (xid=0x330156x)
[2013-02-25 20:08:33.860425] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:08:33.860479] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:31.902734 (xid=0x330157x)
[2013-02-25 20:08:33.860503] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:08:33.860635] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.860653] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.860816] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:08:31.903449 (xid=0x330158x)
[2013-02-25 20:08:33.860832] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.860840] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:08:33.860852] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.860881] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.860896] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.860918] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.860934] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.860948] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964627: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.860970] I [client.c:2090:client_rpc_notify] 0-adata-client-7: disconnected
[2013-02-25 20:08:33.861764] W [client3_1-fops.c:5306:client3_1_finodelk] 0-adata-client-7:  (e5dd47af-45bc-48ff-97e0-f531457854ae) remote_fd is -1. EBADFD
[2013-02-25 20:08:33.861828] W [client3_1-fops.c:4098:client3_1_flush] 0-adata-client-7:  (e5dd47af-45bc-48ff-97e0-f531457854ae) remote_fd is -1. EBADFD
[2013-02-25 20:08:33.866184] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.866212] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.866566] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.866581] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.866713] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 4159901438
[2013-02-25 20:08:33.866729] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.866740] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964632: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.866772] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.866785] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.866928] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.866941] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.866966] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.866974] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.866982] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964634: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.867359] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.867372] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867395] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.867401] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.867403] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867417] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.867427] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964636: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.867593] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.867605] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867629] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.867637] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867645] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964637: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.867744] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.867755] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867778] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.867786] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867793] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964638: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.867895] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.867907] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867931] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.867939] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.867946] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964639: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.867954] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:08:33.867968] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 6 subvolumes have unrecoverable errors
[2013-02-25 20:08:33.868099] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.868111] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.868135] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.868142] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.868150] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964640: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.868209] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.868221] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.868252] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:33.868260] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.868268] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964641: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.868961] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 4159901438
[2013-02-25 20:08:33.868973] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.868981] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964643: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:33.869247] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.869262] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.869287] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:33.869295] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:33.869303] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964644: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:34.587340] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:34.587367] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:34.587400] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:34.587408] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:34.587417] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964656: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:34.587686] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:34.587710] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:34.587753] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:34.587766] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:34.587774] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964657: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:36.577804] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:36.578002] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:36.578866] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:36.578929] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:36.578973] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:36.579007] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:36.579229] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.579244] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.579272] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.579291] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.579300] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964663: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:36.579424] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.579436] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.579459] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.579466] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.579473] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964664: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:36.975863] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.975882] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.975912] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.975920] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.975930] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964665: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:36.975995] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.976007] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.976030] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:36.976037] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:36.976044] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964666: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:37.375675] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.375700] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.375725] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.375732] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.375740] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964667: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:37.375956] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.375976] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.376016] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.376029] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.376042] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964668: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:37.837165] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:37.837352] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:37.837422] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:37.837465] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:37.837493] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:37.837518] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:37.837738] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.837754] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.837781] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.837789] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.837797] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964672: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:37.837946] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.837959] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.837981] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:37.837989] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:37.837996] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964673: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:38.235818] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.235843] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.235877] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.235885] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.235893] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964674: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:38.236159] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.236182] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.236225] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.236241] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.236249] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964675: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:38.675819] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.675841] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.675874] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.675883] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.675891] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964676: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:38.676110] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.676131] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.676163] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:38.676181] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:38.676189] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964677: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:39.092531] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:39.092885] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:39.092919] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:39.092947] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:39.092973] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:39.093338] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:39.093578] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.093602] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.093639] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.093647] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.093654] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964681: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:39.093889] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.093910] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.093950] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.093962] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.093969] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964682: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:39.496018] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.496032] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.496056] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.496064] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.496071] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964683: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:39.496269] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.496284] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.496315] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 667601132
[2013-02-25 20:08:39.496325] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:39.496333] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964684: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:52.749214] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:52.749415] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:52.749479] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:52.749509] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:52.749561] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:52.749593] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:53.262222] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:53.262250] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:53.262293] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:53.262301] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:53.262309] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964708: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:53.262376] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:53.262388] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:53.262411] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:08:53.262418] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:08:53.262425] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964709: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:08:53.830332] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:53.830371] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:53.830414] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:53.830441] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:53.830469] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:53.830511] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:55.687876] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:55.687957] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:55.688017] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:55.688047] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:55.688073] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:55.688098] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:57.786402] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:08:57.786471] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:08:57.786532] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-9 for /files returned -1 (Input/output error)
[2013-02-25 20:08:57.786592] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:08:57.786620] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-6 for /files returned -1 (Input/output error)
[2013-02-25 20:08:57.786646] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:08:59.546851] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-5: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.547428] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-5: Connected to 172.16.5.3:24011, attached to remote volume '/var/gfs/sdd/adata-s003-3'.
[2013-02-25 20:08:59.547456] I [client-handshake.c:1454:client_setvolume_cbk] 0-adata-client-5: Server and Client lk-version numbers are same, no need to reopen the fds
[2013-02-25 20:08:59.592235] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-21: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.592731] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-21: Connected to 172.16.5.3:24019, attached to remote volume '/var/gfs/sdl/adata-s003-11'.
[2013-02-25 20:08:59.592759] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-21: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:08:59.592774] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-21: 16 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:59.594806] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-21: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:59.596639] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-21: Server lk version = 1
[2013-02-25 20:08:59.603088] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-11: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.603531] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-11: Connected to 172.16.5.3:24014, attached to remote volume '/var/gfs/sdg/adata-s003-6'.
[2013-02-25 20:08:59.603555] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-11: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:08:59.603567] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-11: 1 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:59.604003] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-11: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:59.606031] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-19: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.606404] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-19: Connected to 172.16.5.3:24018, attached to remote volume '/var/gfs/sdk/adata-s003-10'.
[2013-02-25 20:08:59.606427] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-19: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:08:59.606441] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-19: 10 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:59.608115] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-19: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:59.608691] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-19: Server lk version = 1
[2013-02-25 20:08:59.611960] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-15: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.612625] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-15: Connected to 172.16.5.3:24016, attached to remote volume '/var/gfs/sdi/adata-s003-8'.
[2013-02-25 20:08:59.612666] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-15: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:08:59.612679] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-15: 2 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:59.613282] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-15: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:59.613669] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-15: Server lk version = 1
[2013-02-25 20:08:59.616049] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-13: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.617038] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-13: Connected to 172.16.5.3:24015, attached to remote volume '/var/gfs/sdh/adata-s003-7'.
[2013-02-25 20:08:59.617061] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-13: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:08:59.617073] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-13: 1 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:59.617591] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-13: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:59.617886] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-13: Server lk version = 1
[2013-02-25 20:08:59.620736] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-7: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.621364] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-7: Connected to 172.16.5.3:24012, attached to remote volume '/var/gfs/sde/adata-s003-4'.
[2013-02-25 20:08:59.621386] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-7: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:08:59.621403] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-7: 35 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:08:59.622857] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-11: Server lk version = 1
[2013-02-25 20:08:59.679340] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-7: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:08:59.707318] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-3: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:08:59.707431] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.714388] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.717041] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-7: Server lk version = 1
[2013-02-25 20:08:59.722312] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.734390] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.734774] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.735023] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.735205] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.735324] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:08:59.735496] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.738613] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.738675] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.738785] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.738854] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.738957] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.739032] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.739129] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.739210] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.739297] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.739380] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:09:59.739466] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:10:05.934021] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-adata-replicate-7: no missing files - /files/random-file.dat. proceeding to metadata check
[2013-02-25 20:11:05.882270] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-11: server 172.16.5.3:24014 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:11:05.882646] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:06.005670 (xid=0x301443x)
[2013-02-25 20:11:05.882681] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.882769] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:07.857239 (xid=0x301444x)
[2013-02-25 20:11:05.882782] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.882816] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:08.109129 (xid=0x301445x)
[2013-02-25 20:11:05.882828] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.882954] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:11.553436 (xid=0x301446x)
[2013-02-25 20:11:05.882967] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.883001] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:13.604996 (xid=0x301447x)
[2013-02-25 20:11:05.883016] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.883063] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:14.636508 (xid=0x301448x)
[2013-02-25 20:11:05.883079] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.883130] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:14.910060 (xid=0x301449x)
[2013-02-25 20:11:05.883166] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.883217] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:15.024052 (xid=0x301450x)
[2013-02-25 20:11:05.883237] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.883290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:09:15.499863 (xid=0x301451x)
[2013-02-25 20:11:05.883310] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.883362] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:22.893056 (xid=0x301452x)
[2013-02-25 20:11:05.883384] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.883593] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:09:41.871289 (xid=0x301453x)
[2013-02-25 20:11:05.893014] I [socket.c:2315:socket_submit_request] 0-adata-client-11: not connected (priv->connected = 0)
[2013-02-25 20:11:05.893041] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-11: failed to submit rpc-request (XID: 0x301468x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-11)
[2013-02-25 20:11:05.893057] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-11: timer must have expired
[2013-02-25 20:11:05.893060] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:05.893126] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:47.278094 (xid=0x301454x)
[2013-02-25 20:11:05.893159] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.893290] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:47.537436 (xid=0x301455x)
[2013-02-25 20:11:05.893310] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.893437] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:47.572644 (xid=0x301456x)
[2013-02-25 20:11:05.893459] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.893551] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:09:53.114476 (xid=0x301457x)
[2013-02-25 20:11:05.893567] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.893599] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-11: failed to submit rpc-request (XID: 0x301469x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-11)
[2013-02-25 20:11:05.893617] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:05.893667] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2013-02-25 20:09:59.549213 (xid=0x301458x)
[2013-02-25 20:11:05.893681] W [client3_1-fops.c:881:client3_1_flush_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.893766] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-11: failed to submit rpc-request (XID: 0x301470x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (adata-client-11)
[2013-02-25 20:11:05.893783] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.893813] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:10:01.194085 (xid=0x301459x)
[2013-02-25 20:11:05.893833] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.893920] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:10:09.287970 (xid=0x301460x)
[2013-02-25 20:11:05.893935] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.893960] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:27.460190 (xid=0x301461x)
[2013-02-25 20:11:05.894003] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.894029] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-11: failed to submit rpc-request (XID: 0x301471x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-11)
[2013-02-25 20:11:05.894046] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:05.894097] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:45.767322 (xid=0x301462x)
[2013-02-25 20:11:05.894115] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.894206] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:49.300578 (xid=0x301463x)
[2013-02-25 20:11:05.894225] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.894308] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:53.218770 (xid=0x301464x)
[2013-02-25 20:11:05.894326] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.894411] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:01.304498 (xid=0x301465x)
[2013-02-25 20:11:05.894427] W [rpc-clnt.c:1496:rpc_clnt_submit] 0-adata-client-11: failed to submit rpc-request (XID: 0x301472x Program: GlusterFS 3.1, ProgVers: 330, Proc: 27) to rpc-transport (adata-client-11)
[2013-02-25 20:11:05.894455] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:05.894456] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:05.894492] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:02.752691 (xid=0x301466x)
[2013-02-25 20:11:05.894506] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.894586] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-11: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:04.011684 (xid=0x301467x)
[2013-02-25 20:11:05.894615] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-11: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:05.894683] I [client.c:2090:client_rpc_notify] 0-adata-client-11: disconnected
[2013-02-25 20:11:05.894795] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895037] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895158] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895289] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895388] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895515] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895601] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:11:05.895721] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:11:23.884968] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-7: server 172.16.5.3:24012 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:11:23.885201] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:10:01.194048 (xid=0x330256x)
[2013-02-25 20:11:23.885235] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.885281] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:10:09.287920 (xid=0x330257x)
[2013-02-25 20:11:23.885304] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.885350] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:27.460124 (xid=0x330258x)
[2013-02-25 20:11:23.885363] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:23.885459] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:10:41.879319 (xid=0x330259x)
[2013-02-25 20:11:23.885478] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-7: timer must have expired
[2013-02-25 20:11:23.885507] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:45.767254 (xid=0x330260x)
[2013-02-25 20:11:23.885518] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:23.885611] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:49.300517 (xid=0x330261x)
[2013-02-25 20:11:23.885626] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:23.885725] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:10:53.218693 (xid=0x330262x)
[2013-02-25 20:11:23.885745] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:23.885860] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:01.304448 (xid=0x330263x)
[2013-02-25 20:11:23.885875] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.885900] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:02.752623 (xid=0x330264x)
[2013-02-25 20:11:23.885920] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:23.885970] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:04.011625 (xid=0x330265x)
[2013-02-25 20:11:23.885989] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:11:23.886056] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.883063 (xid=0x330266x)
[2013-02-25 20:11:23.886068] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886082] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:11:23.886113] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.893543 (xid=0x330267x)
[2013-02-25 20:11:23.886124] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886136] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:11:23.886168] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.893975 (xid=0x330268x)
[2013-02-25 20:11:23.886179] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886191] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:11:23.886244] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.894374 (xid=0x330269x)
[2013-02-25 20:11:23.886260] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886274] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-3 for /files returned -1 (Input/output error)
[2013-02-25 20:11:23.886365] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.895904 (xid=0x330270x)
[2013-02-25 20:11:23.886382] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886427] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.896460 (xid=0x330271x)
[2013-02-25 20:11:23.886440] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886479] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.897034 (xid=0x330272x)
[2013-02-25 20:11:23.886490] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:11:23.886528] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:15.616353 (xid=0x330273x)
[2013-02-25 20:11:23.886538] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.886561] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:15.725902 (xid=0x330274x)
[2013-02-25 20:11:23.886570] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.886592] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:16.173143 (xid=0x330275x)
[2013-02-25 20:11:23.886601] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.886623] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-7: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:19.809782 (xid=0x330276x)
[2013-02-25 20:11:23.886632] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-7: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:11:23.886652] I [client.c:2090:client_rpc_notify] 0-adata-client-7: disconnected
[2013-02-25 20:11:59.723830] I [client-handshake.c:1636:select_server_supported_programs] 0-adata-client-7: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)
[2013-02-25 20:11:59.724253] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-7: Connected to 172.16.5.3:24012, attached to remote volume '/var/gfs/sde/adata-s003-4'.
[2013-02-25 20:11:59.724281] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-7: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:11:59.724299] I [client-handshake.c:1282:client_post_handshake] 0-adata-client-7: 35 fds open - Delaying child_up until they are re-opened
[2013-02-25 20:11:59.738929] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:11:59.739024] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:11:59.739339] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:11:59.739456] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-3: remote operation failed: Transport endpoint is not connected. Path: /files/random-file.dat (00000000-0000-0000-0000-000000000000)
[2013-02-25 20:11:59.739589] I [client-handshake.c:1433:client_setvolume_cbk] 0-adata-client-3: Connected to 172.16.5.3:24010, attached to remote volume '/var/gfs/sdc/adata-s003-2'.
[2013-02-25 20:11:59.739603] I [client-handshake.c:1445:client_setvolume_cbk] 0-adata-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2013-02-25 20:11:59.739924] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-3: Server lk version = 1
[2013-02-25 20:11:59.891022] I [client-lk.c:601:decrement_reopen_fd_count] 0-adata-client-7: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2013-02-25 20:12:23.895902] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-adata-client-21: server 172.16.5.3:24019 has not responded in the last 42 seconds, disconnecting.
[2013-02-25 20:12:23.896144] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:01.304654 (xid=0x381521x)
[2013-02-25 20:12:23.896164] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.896210] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:02.752875 (xid=0x381522x)
[2013-02-25 20:12:23.896223] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:12:23.896317] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:04.011846 (xid=0x381523x)
[2013-02-25 20:12:23.896343] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:12:23.896433] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.893319 (xid=0x381524x)
[2013-02-25 20:12:23.896450] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.896465] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:12:23.896526] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.893803 (xid=0x381525x)
[2013-02-25 20:12:23.896539] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.896552] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:12:23.896624] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.894216 (xid=0x381526x)
[2013-02-25 20:12:23.896639] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.896653] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:12:23.896706] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:12:23.896717] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.894643 (xid=0x381527x)
[2013-02-25 20:12:23.896721] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.896750] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.896774] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:12:23.896776] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:12:23.896792] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.896802] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964856: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:12:23.896839] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.896186 (xid=0x381528x)
[2013-02-25 20:12:23.896874] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.896894] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.896903] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.896989] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2503125722
[2013-02-25 20:12:23.897002] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.897011] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.896739 (xid=0x381529x)
[2013-02-25 20:12:23.897025] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2503125722
[2013-02-25 20:12:23.897035] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.897030] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.897044] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964858: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:12:23.897076] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.897088] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.897143] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3312595588
[2013-02-25 20:12:23.897154] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:05.897308 (xid=0x381530x)
[2013-02-25 20:12:23.897155] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.897172] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.897181] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964859: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:12:23.897197] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.897205] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.897283] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:15.616474 (xid=0x381531x)
[2013-02-25 20:12:23.897326] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.897369] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:15.726011 (xid=0x381532x)
[2013-02-25 20:12:23.897380] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.897426] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:16.173259 (xid=0x381533x)
[2013-02-25 20:12:23.897438] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.897481] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:19.809892 (xid=0x381534x)
[2013-02-25 20:12:23.897500] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.897551] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:23.887037 (xid=0x381535x)
[2013-02-25 20:12:23.897574] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.897598] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.897610] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.897725] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:23.887610 (xid=0x381536x)
[2013-02-25 20:12:23.897749] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.897792] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.897808] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.897917] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:23.888079 (xid=0x381537x)
[2013-02-25 20:12:23.897937] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.897972] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.897986] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.898079] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:23.888674 (xid=0x381538x)
[2013-02-25 20:12:23.898102] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: /files (c38f1652-e476-4e96-b248-21f6d240944a)
[2013-02-25 20:12:23.898126] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=4 overlaps=0
[2013-02-25 20:12:23.898142] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 4 subvolumes have unrecoverable errors
[2013-02-25 20:12:23.898200] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 430221643
[2013-02-25 20:12:23.898219] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.898227] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:24.646400 (xid=0x381539x)
[2013-02-25 20:12:23.898249] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.898262] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 430221643
[2013-02-25 20:12:23.898278] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:12:23.898286] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:30.934457 (xid=0x381540x)
[2013-02-25 20:12:23.898307] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:12:23.898293] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964864: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:12:23.898418] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:33.105449 (xid=0x381541x)
[2013-02-25 20:12:23.898436] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.898472] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:11:33.516992 (xid=0x381542x)
[2013-02-25 20:12:23.898491] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.898535] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-02-25 20:11:41.888437 (xid=0x381543x)
[2013-02-25 20:12:23.898553] W [client-handshake.c:275:client_ping_cbk] 0-adata-client-21: timer must have expired
[2013-02-25 20:12:23.898587] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:49.707555 (xid=0x381544x)
[2013-02-25 20:12:23.898607] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:12:23.898724] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:11:53.175778 (xid=0x381545x)
[2013-02-25 20:12:23.898749] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:12:23.898898] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:12:01.386473 (xid=0x381546x)
[2013-02-25 20:12:23.898920] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.898954] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(STATFS(14)) called at 2013-02-25 20:12:06.750001 (xid=0x381547x)
[2013-02-25 20:12:23.898969] W [client3_1-fops.c:763:client3_1_statfs_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected
[2013-02-25 20:12:23.899003] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x7f640cb38ec8] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x7f640cb38b80] (-->/usr/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7f640cb385ee]))) 0-adata-client-21: forced unwinding frame type(GlusterFS 3.1) op(LOOKUP(27)) called at 2013-02-25 20:12:17.756296 (xid=0x381548x)
[2013-02-25 20:12:23.899021] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-adata-client-21: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
[2013-02-25 20:12:23.899074] I [client.c:2090:client_rpc_notify] 0-adata-client-21: disconnected
[2013-02-25 20:12:59.830181] I [client-handshake.c:453:client_set_lk_version_cbk] 0-adata-client-7: Server lk version = 1
[2013-02-25 20:12:59.833387] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:12:59.833404] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 3 subvolumes have unrecoverable errors
[2013-02-25 20:12:59.903588] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:12:59.903617] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 3 subvolumes have unrecoverable errors
[2013-02-25 20:12:59.903747] I [dht-layout.c:593:dht_layout_normalize] 0-adata-dht: found anomalies in /files. holes=3 overlaps=0
[2013-02-25 20:12:59.903764] W [dht-selfheal.c:882:dht_selfheal_directory] 0-adata-dht: 3 subvolumes have unrecoverable errors
[2013-02-25 20:13:59.869981] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.870515] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.870742] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.875587] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.875805] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.876051] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.876144] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.876327] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.876450] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.876638] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.876824] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.877196] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.877421] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-0 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.877573] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-8 for /files returned -1 (Input/output error)
[2013-02-25 20:13:59.877611] I [dht-common.c:543:dht_revalidate_cbk] 0-adata-dht: subvolume adata-replicate-10 for /files returned -1 (Input/output error)
[2013-02-25 20:14:00.029411] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.029430] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.029456] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.029464] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.029472] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964946: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.029628] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.029641] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.029664] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.029672] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.029679] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964948: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.029905] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2503125722
[2013-02-25 20:14:00.029918] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.029943] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2503125722
[2013-02-25 20:14:00.029950] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.029958] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964951: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.030186] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2503125722
[2013-02-25 20:14:00.030202] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.030237] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 2503125722
[2013-02-25 20:14:00.030252] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.030265] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964952: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.030696] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.030709] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.030733] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.030741] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.030749] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964956: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.030927] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.030942] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.030967] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.030975] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.030982] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964959: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.031188] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.031201] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.031224] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.031232] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.031239] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964962: LOOKUP() /files/random-file.dat => -1 (Invalid argument)
[2013-02-25 20:14:00.031297] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.031308] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.031331] W [dht-layout.c:186:dht_layout_search] 0-adata-dht: no subvolume for hash (value) = 3211401705
[2013-02-25 20:14:00.031339] E [dht-common.c:1372:dht_lookup] 0-adata-dht: Failed to get hashed subvol for /files/random-file.dat
[2013-02-25 20:14:00.031346] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 3964963: LOOKUP() /files/random-file.dat => -1 (Invalid argument)