[Lustre-discuss] I/O error on clients

Wed Jul 7 17:48:21 PDT 2010

Here is a dump of a log of some kind from the client at the time of the eviction.

Does this shed any further light on the problem?

00000100:00000400:7:1278392906.021560:0:6677:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1340041377934503 sent from AC3-OST0006-osc-ffff81043e754c00 to NID 172.16.16.2 at tcp 11s ago has timed o
ut (11s prior to deadline).
  req at ffff81043f31dc00 x1340041377934503/t0 o101->AC3-OST0006_UUID at 172.16.16.2@tcp:28/4 lens 296/544 e 0 to 1 dl 1278392906 ref 1 fl Rpc:/0/0 rc 0/0
00000100:02000400:7:1278392906.021598:0:6677:0:(import.c:159:ptlrpc_set_import_discon()) AC3-OST0006-osc-ffff81043e754c00: Connection to service AC3-OST0006 via nid 172.16.16.2 at tcp was lost; in progress operations using this service will wait for recovery to complete.
00000100:00080000:7:1278392906.021611:0:6677:0:(import.c:167:ptlrpc_set_import_discon()) ffff81042018c000 AC3-OST0006_UUID: changing import state from FULL to DISCONN
00000100:00080000:7:1278392906.021615:0:6677:0:(import.c:403:ptlrpc_fail_import()) AC3-OST0006_UUID: waking up pinger00000100:00080000:4:1278392906.021634:0:3573:0:(recover.c:67:ptlrpc_initiate_recovery()) AC3-OST0006_UUID: starting recovery
00000100:00080000:4:1278392906.021639:0:3573:0:(import.c:608:ptlrpc_connect_import()) ffff81042018c000 AC3-OST0006_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:4:1278392906.021645:0:3573:0:(import.c:470:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4379111012
00000100:00080000:4:1278392906.021645:0:3573:0:(import.c:470:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4379111012
00000100:00080000:4:1278392906.021651:0:3573:0:(import.c:544:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: import ffff81042018c000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcp
00000100:00000400:7:1278392906.030554:0:6687:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1340041377934535 sent from AC3-OST0006-osc-ffff81043e754c00 to NID 172.16.16.2 at tcp 11s ago has timed o
ut (11s prior to deadline).
  req at ffff81031686b000 x1340041377934535/t0 o101->AC3-OST0006_UUID at 172.16.16.2@tcp:28/4 lens 296/544 e 0 to 1 dl 1278392906 ref 1 fl Rpc:/0/0 rc 0/0
00000100:00080000:7:1278392906.030567:0:6687:0:(import.c:182:ptlrpc_set_import_discon()) osc: import ffff81042018c000 already not connected (conn 177, was 176): CONNECTING
00000100:00000400:5:1278392907.030414:0:6686:0:(client.c:1463:ptlrpc_expire_one_request()) @@@ Request x1340041377934536 sent from AC3-OST0008-osc-ffff81043e754c00 to NID 172.16.16.2 at tcp 12s ago has timed o
ut (12s prior to deadline).
  req at ffff8102b439bc00 x1340041377934536/t0 o101->AC3-OST0008_UUID at 172.16.16.2@tcp:28/4 lens 296/544 e 0 to 1 dl 1278392907 ref 1 fl Rpc:/0/0 rc 0/0
00000100:02000400:5:1278392907.030434:0:6686:0:(import.c:159:ptlrpc_set_import_discon()) AC3-OST0008-osc-ffff81043e754c00: Connection to service AC3-OST0008 via nid 172.16.16.2 at tcp was lost; in progress ope
rations using this service will wait for recovery to complete.
00000100:00080000:5:1278392907.030449:0:6686:0:(import.c:167:ptlrpc_set_import_discon()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from FULL to DISCONN
00000100:00080000:5:1278392907.030452:0:6686:0:(import.c:403:ptlrpc_fail_import()) AC3-OST0008_UUID: waking up pinger
00000100:00080000:4:1278392907.030465:0:3573:0:(recover.c:67:ptlrpc_initiate_recovery()) AC3-OST0008_UUID: starting recovery00000100:00080000:4:1278392907.030470:0:3573:0:(import.c:608:ptlrpc_connect_import()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:4:1278392907.030475:0:3573:0:(import.c:470:import_select_connection()) AC3-OST0008-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 437911101200000100:00080000:4:1278392907.030480:0:3573:0:(import.c:544:import_select_connection()) AC3-OST0008-osc-ffff81043e754c00: import ffff810420d1a000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcp
00000400:00000100:3:1278392907.213541:0:3560:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 4 match 1340041377934536 offset 192 length 352: 200000400:00000100:3:1278392907.213550:0:3560:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 4 match 1340041377934503 offset 192 length 352: 2
00000400:00000100:3:1278392907.213554:0:3560:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 4 match 1340041377934535 offset 192 length 352: 200000100:02020000:0:1278392907.213815:0:3572:0:(client.c:975:ptlrpc_check_status()) 11-0: an error occurred while communicating with 172.16.16.2 at tcp. The ost_connect operation failed with -16
00000100:00080000:0:1278392907.213828:0:3572:0:(import.c:1091:ptlrpc_connect_interpret()) ffff81042018c000 AC3-OST0006_UUID: changing import state from CONNECTING to DISCONN00000100:00080000:0:1278392907.213831:0:3572:0:(import.c:1137:ptlrpc_connect_interpret()) recovery of AC3-OST0006_UUID on 172.16.16.2 at tcp failed (-16)
00000100:02020000:0:1278392907.213839:0:3572:0:(client.c:975:ptlrpc_check_status()) 11-0: an error occurred while communicating with 172.16.16.2 at tcp. The ost_connect operation failed with -1600000100:00080000:0:1278392907.213841:0:3572:0:(import.c:1091:ptlrpc_connect_interpret()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from CONNECTING to DISCONN
00000100:00080000:0:1278392907.213843:0:3572:0:(import.c:1137:ptlrpc_connect_interpret()) recovery of AC3-OST0008_UUID on 172.16.16.2 at tcp failed (-16)
00000100:00080000:6:1278392907.226136:0:3571:0:(client.c:1145:ptlrpc_send_new_req()) @@@ req from PID 0 waiting for recovery: (FULL != DISCONN)  req at ffff8101b4588c00 x1340041377955935/t0 o3->AC3-OST0008_UUI
D at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
00000100:00080000:6:1278392907.226192:0:3571:0:(client.c:1145:ptlrpc_send_new_req()) @@@ req from PID 0 waiting for recovery: (FULL != DISCONN)  req at ffff81042061b800 x1340041377955949/t0 o4->AC3-OST0008_UUI
D at 172.16.16.2@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
00000100:00080000:6:1278392907.226202:0:3571:0:(client.c:1145:ptlrpc_send_new_req()) @@@ req from PID 0 waiting for recovery: (FULL != DISCONN)  req at ffff81042061a400 x1340041377955951/t0 o4->AC3-OST0008_UUI
D at 172.16.16.2@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
00000100:00080000:6:1278392907.226207:0:3571:0:(client.c:1145:ptlrpc_send_new_req()) @@@ req from PID 0 waiting for recovery: (FULL != DISCONN)  req at ffff8104201e5000 x1340041377955955/t0 o4->AC3-OST0006_UUID at 172.16.16.2@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
00000100:00080000:6:1278392907.226211:0:3571:0:(client.c:1145:ptlrpc_send_new_req()) @@@ req from PID 0 waiting for recovery: (FULL != DISCONN)  req at ffff81042030f800 x1340041377955956/t0 o3->AC3-OST0008_UUI
D at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/000000100:00080000:6:1278392907.226219:0:3571:0:(client.c:1145:ptlrpc_send_new_req()) @@@ req from PID 0 waiting for recovery: (FULL != DISCONN)  req at ffff81042061a000 x1340041377955959/t0 o4->AC3-OST0008_UUI
D at 172.16.16.2@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
00000100:00080000:4:1278392917.022748:0:3573:0:(recover.c:67:ptlrpc_initiate_recovery()) AC3-OST0006_UUID: starting recovery00000100:00080000:4:1278392917.022753:0:3573:0:(import.c:608:ptlrpc_connect_import()) ffff81042018c000 AC3-OST0006_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:4:1278392917.022759:0:3573:0:(import.c:470:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4724612378
00000100:00080000:4:1278392917.022766:0:3573:0:(import.c:544:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: import ffff81042018c000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcp00000100:00080000:0:1278392917.023419:0:3572:0:(import.c:868:ptlrpc_connect_interpret()) reconnected to AC3-OST0006_UUID at 172.16.16.2@tcp after partition
00000100:00080000:0:1278392917.023425:0:3572:0:(import.c:891:ptlrpc_connect_interpret()) ffff81042018c000 AC3-OST0006_UUID: changing import state from CONNECTING to RECOVER
1 dl 1278392951 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392917.023440:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377933846 new x1340041377973369
 1 dl 1278392951 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392917.023452:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377934227 new x1340041377973370
 1 dl 1278392952 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392917.023459:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377934618 new x1340041377973371
 1 dl 1278392952 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392917.023466:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377934964 new x1340041377973372
 1 dl 1278392955 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392917.023472:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377940508 new x1340041377973373
00000100:00080000:0:1278392917.023475:0:3572:0:(import.c:1319:ptlrpc_import_recovery_state_machine()) ffff81042018c000 AC3-OST0006_UUID: changing import state from RECOVER to FULL
00000100:00080000:6:1278392917.023480:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377973369 new x1340041377973374
00000002:00080000:0:1278392917.023492:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:02000000:0:1278392917.023497:0:3572:0:(import.c:1327:ptlrpc_import_recovery_state_machine()) AC3-OST0006-osc-ffff81043e754c00: Connection restored to service AC3-OST0006 using nid 172.16.16.2 at tcp.
00000100:00080000:6:1278392917.023513:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377973370 new x1340041377973375
ns 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
 lens 296/544 e 0 to 1 dl 1278392906 ref 1 fl Rpc:X/0/0 rc 0/0
 lens 296/544 e 0 to 1 dl 1278392906 ref 1 fl Rpc:X/0/0 rc 0/0
00000100:00080000:6:1278392917.023543:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377973371 new x1340041377973376
00000002:00080000:0:1278392917.023544:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:00080000:6:1278392917.023572:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377973372 new x1340041377973377
00000100:00080000:6:1278392917.023596:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377973373 new x1340041377973378
00000100:00080000:4:1278392919.644883:0:3573:0:(recover.c:67:ptlrpc_initiate_recovery()) AC3-OST0008_UUID: starting recovery
00000100:00080000:4:1278392919.644889:0:3573:0:(import.c:608:ptlrpc_connect_import()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:4:1278392919.644894:0:3573:0:(import.c:470:import_select_connection()) AC3-OST0008-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4724613386
00000100:00080000:4:1278392919.644900:0:3573:0:(import.c:544:import_select_connection()) AC3-OST0008-osc-ffff81043e754c00: import ffff810420d1a000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcp
00000100:00080000:0:1278392919.645477:0:3572:0:(import.c:868:ptlrpc_connect_interpret()) reconnected to AC3-OST0008_UUID at 172.16.16.2@tcp after partition
00000100:00080000:0:1278392919.645482:0:3572:0:(import.c:891:ptlrpc_connect_interpret()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from CONNECTING to RECOVER
00000100:00080000:0:1278392919.645486:0:3572:0:(import.c:1314:ptlrpc_import_recovery_state_machine()) reconnected to AC3-OST0008_UUID at 172.16.16.2@tcp
 1 dl 1278392952 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392919.645495:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377933615 new x1340041377977349
 1 dl 1278392952 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392919.645505:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377933659 new x1340041377977350
 1 dl 1278392952 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278392919.645511:0:3572:0:(client.c:1969:ptlrpc_resend_req()) resend bulk old x1340041377933703 new x1340041377977351
00000100:00080000:0:1278392919.645514:0:3572:0:(import.c:1319:ptlrpc_import_recovery_state_machine()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from RECOVER to FULL
00000100:00080000:6:1278392919.645518:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377977350 new x1340041377977353
00000002:00080000:0:1278392919.645531:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:02000000:0:1278392919.645536:0:3572:0:(import.c:1327:ptlrpc_import_recovery_state_machine()) AC3-OST0008-osc-ffff81043e754c00: Connection restored to service AC3-OST0008 using nid 172.16.16.2 at tcp.
 lens 296/544 e 0 to 1 dl 1278392907 ref 1 fl Rpc:X/0/0 rc 0/0
ns 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
ns 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
ns 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
ns 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
ns 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
ns 448/592 e 0 to 1 dl 1278392952 ref 2 fl Rpc:S/0/0 rc -11/0
00000100:00080000:6:1278392919.645591:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377977351 new x1340041377977354
00000002:00080000:0:1278392919.645593:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:00080000:6:1278392919.645729:0:3571:0:(client.c:1332:ptlrpc_check_set()) resend bulk old x1340041377977349 new x1340041377977355
00000100:02020000:2:1278393017.800317:0:3571:0:(client.c:975:ptlrpc_check_status()) 11-0: an error occurred while communicating with 172.16.16.2 at tcp. The ost_write operation failed with -107
00000100:00080000:2:1278393017.800332:0:3571:0:(recover.c:196:ptlrpc_request_handle_notconn()) import AC3-OST0006-osc-ffff81043e754c00 of AC3-OST0006_UUID at 172.16.16.2@tcp abruptly disconnected: reconnecting
rations using this service will wait for recovery to complete.
00000100:00080000:2:1278393017.800343:0:3571:0:(import.c:167:ptlrpc_set_import_discon()) ffff81042018c000 AC3-OST0006_UUID: changing import state from FULL to DISCONN
00000100:00080000:2:1278393017.800350:0:3571:0:(import.c:608:ptlrpc_connect_import()) ffff81042018c000 AC3-OST0006_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:2:1278393017.800357:0:3571:0:(import.c:470:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4724623378
00000100:00080000:2:1278393017.800366:0:3571:0:(import.c:544:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: import ffff81042018c000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcpt 0 o8->AC3-OST0006_UUID at 172.16.16.2@tcp:28/4 lens 368/584 e 0 to 1 dl 1278393031 ref 1 fl Interpret:RN/0/0 rc 0/0
00000100:00080000:0:1278393017.801096:0:3572:0:(import.c:911:ptlrpc_connect_interpret()) ffff81042018c000 AC3-OST0006_UUID: changing import state from CONNECTING to EVICTED
00000100:02020000:0:1278393017.801099:0:3572:0:(import.c:1249:ptlrpc_import_recovery_state_machine()) 167-0: This client was evicted by AC3-OST0006; in progress operations using this service will fail.
00000100:00080000:0:1278393017.801106:0:3572:0:(import.c:1253:ptlrpc_import_recovery_state_machine()) evicted from AC3-OST0006_UUID at 172.16.16.2@tcp; invalidating
00000002:00080000:0:1278393017.801150:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:00080000:0:1278393017.801188:0:6779:0:(import.c:1215:ptlrpc_invalidate_import_thread()) thread invalidate import AC3-OST0006-osc-ffff81043e754c00 to AC3-OST0006_UUID at 172.16.16.2@tcp
00000100:00080000:0:1278393017.801190:0:6779:0:(import.c:194:ptlrpc_deactivate_and_unlock_import()) setting import AC3-OST0006_UUID INVALID
00020000:01000000:0:1278393017.801201:0:6779:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0006_UUID inactive
00000002:00080000:0:1278393017.801206:0:6779:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00020000:00080000:7:1278393017.810055:0:6430:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00020000:00080000:5:1278393017.827252:0:3095:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00020000:00080000:5:1278393017.827298:0:3095:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00020000:00080000:5:1278393017.827325:0:3095:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00020000:00080000:7:1278393017.833456:0:3095:0:(lov_request.c:1099:lov_prep_destroy_set()) lov idx 6 inactive
00000080:00020000:7:1278393017.833461:0:3095:0:(namei.c:1176:ll_objects_destroy()) obd destroy objid 0x18542c4 at 0x0 error -5
00020000:00080000:4:1278393017.912169:0:6430:0:(lov_obd.c:2372:lov_join_lru()) lov idx 6 inactive
00020000:00080000:4:1278393017.913654:0:6419:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00000080:00020000:4:1278393017.913660:0:6419:0:(file.c:995:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
00020000:00080000:4:1278393017.922360:0:6419:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00010000:00020000:1:1278393018.325707:0:6779:0:(ldlm_resource.c:518:ldlm_namespace_cleanup()) Namespace AC3-OST0006-osc-ffff81043e754c00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
00010000:00020000:1:1278393018.325727:0:6779:0:(ldlm_resource.c:523:ldlm_namespace_cleanup()) Resource: ffff810117997500 (8432287/0/0/0) (rc: 1)
00020000:00080000:7:1278393018.325761:0:6687:0:(lov_obd.c:1727:lov_brw_check()) lov idx 6 inactive
00000080:00020000:7:1278393018.325781:0:6687:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5
00020000:00080000:7:1278393018.327514:0:6687:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00020000:00080000:2:1278393018.332887:0:6782:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00000080:00020000:2:1278393018.332895:0:6782:0:(file.c:995:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
00000100:00080000:1:1278393018.333028:0:6779:0:(import.c:1224:ptlrpc_invalidate_import_thread()) ffff81042018c000 AC3-OST0006_UUID: changing import state from EVICTED to RECOVER
00000100:00080000:1:1278393018.333032:0:6779:0:(import.c:1314:ptlrpc_import_recovery_state_machine()) reconnected to AC3-OST0006_UUID at 172.16.16.2@tcp
00000100:00080000:1:1278393018.333035:0:6779:0:(import.c:1319:ptlrpc_import_recovery_state_machine()) ffff81042018c000 AC3-OST0006_UUID: changing import state from RECOVER to FULL
00020000:01000000:1:1278393018.333042:0:6779:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0006_UUID active
00000002:00080000:1:1278393018.333051:0:6779:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:02000000:1:1278393018.333056:0:6779:0:(import.c:1327:ptlrpc_import_recovery_state_machine()) AC3-OST0006-osc-ffff81043e754c00: Connection restored to service AC3-OST0006 using nid 172.16.16.2 at tcp.
00000100:02020000:4:1278393020.758895:0:20660:0:(client.c:975:ptlrpc_check_status()) 11-0: an error occurred while communicating with 172.16.16.2 at tcp. The ost_read operation failed with -107
g
erations using this service will wait for recovery to complete.
00000100:00080000:4:1278393020.758919:0:20660:0:(import.c:167:ptlrpc_set_import_discon()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from FULL to DISCONN
00000100:00080000:4:1278393020.758925:0:20660:0:(import.c:608:ptlrpc_connect_import()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:4:1278393020.758928:0:20660:0:(import.c:470:import_select_connection()) AC3-OST0008-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4724626000
00000100:00080000:4:1278393020.758935:0:20660:0:(import.c:544:import_select_connection()) AC3-OST0008-osc-ffff81043e754c00: import ffff810420d1a000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcp
16.16.2 at tcp:6/4 lens 448/592 e 0 to 1 dl 1278393085 ref 1 fl Rpc:RS/0/0 rc 0/-107
UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:W/0/0 rc 0/0
t0 o8->AC3-OST0008_UUID at 172.16.16.2@tcp:28/4 lens 368/584 e 0 to 1 dl 1278393038 ref 1 fl Interpret:RN/0/0 rc 0/0
00000100:00080000:0:1278393020.770920:0:3572:0:(import.c:911:ptlrpc_connect_interpret()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from CONNECTING to EVICTED
00000100:02020000:0:1278393020.770923:0:3572:0:(import.c:1249:ptlrpc_import_recovery_state_machine()) 167-0: This client was evicted by AC3-OST0008; in progress operations using this service will fail.
00000100:00080000:0:1278393020.770930:0:3572:0:(import.c:1253:ptlrpc_import_recovery_state_machine()) evicted from AC3-OST0008_UUID at 172.16.16.2@tcp; invalidating
00000002:00080000:0:1278393020.771032:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:00080000:0:1278393020.771066:0:6784:0:(import.c:1215:ptlrpc_invalidate_import_thread()) thread invalidate import AC3-OST0008-osc-ffff81043e754c00 to AC3-OST0008_UUID at 172.16.16.2@tcp
00000100:00080000:0:1278393020.771068:0:6784:0:(import.c:194:ptlrpc_deactivate_and_unlock_import()) setting import AC3-OST0008_UUID INVALID
2 at tcp:6/4 lens 448/592 e 0 to 1 dl 1278393085 ref 1 fl Rpc:RES/0/0 rc -4/-107
00020000:01000000:0:1278393020.771084:0:6784:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0008_UUID inactive
00000002:00080000:0:1278393020.771087:0:6784:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000080:00020000:4:1278393020.771103:0:20660:0:(rw.c:122:ll_brw()) error from obd_brw: rc = -4
00010000:00020000:0:1278393020.777506:0:6784:0:(ldlm_resource.c:518:ldlm_namespace_cleanup()) Namespace AC3-OST0008-osc-ffff81043e754c00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
00010000:00020000:0:1278393020.777518:0:6784:0:(ldlm_resource.c:523:ldlm_namespace_cleanup()) Resource: ffff81041604b9c0 (5521743/0/0/0) (rc: 1)
00010000:00020000:0:1278393020.777531:0:6784:0:(ldlm_resource.c:518:ldlm_namespace_cleanup()) Namespace AC3-OST0008-osc-ffff81043e754c00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
00010000:00020000:0:1278393020.777533:0:6784:0:(ldlm_resource.c:523:ldlm_namespace_cleanup()) Resource: ffff810266786e00 (8127131/0/0/0) (rc: 1)
00020000:00080000:5:1278393020.777583:0:6686:0:(lov_obd.c:1727:lov_brw_check()) lov idx 8 inactive
00000080:00020000:5:1278393020.777624:0:6686:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5
00000100:00020000:2:1278393020.777678:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061a800 x1340041378196315/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 512/624 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/01068:0:6784:0:(import.c:194:ptlrpc_deactivate_and_unlock_import()) setting import AC3-OST0008_UUID INVALID
00000100:00020000:2:1278393020.778942:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061a000 x1340041378196323/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 448/608 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/01084:0:6784:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0008_UUID inactive
00020000:00080000:5:1278393020.779394:0:6686:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 8 inactive
00000100:00020000:2:1278393020.780168:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff8101f525e000 x1340041378196326/t0 o3->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/07506:0:6784:0:(ldlm_resource.c:518:ldlm_namespace_cleanup()) Namespace AC3-OST0008-osc-ffff81043e754c00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
00000100:00020000:2:1278393020.781246:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff8101f189fc00 x1340041378196328/t0 o3->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/07531:0:6784:0:(ldlm_resource.c:518:ldlm_namespace_cleanup()) Namespace AC3-OST0008-osc-ffff81043e754c00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
00000100:00020000:2:1278393020.782563:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061b400 x1340041378196329/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 496/616 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/07583:0:6686:0:(lov_obd.c:1727:lov_brw_check()) lov idx 8 inactive
00000100:00020000:2:1278393020.783344:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff8104212fc400 x1340041378196334/t0 o3->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00020000:2:1278393020.783559:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061ac00 x1340041378196337/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 448/608 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/09394:0:6686:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 8 inactive
00000100:00020000:2:1278393020.783727:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061c800 x1340041378196341/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 496/616 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00020000:00080000:7:1278393020.783778:0:6785:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 8 inactive
00000080:00020000:7:1278393020.783782:0:6785:0:(file.c:995:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
00000100:00020000:2:1278393020.784052:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061bc00 x1340041378196347/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 448/608 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00020000:2:1278393020.784197:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061cc00 x1340041378196354/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 480/616 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/03782:0:6785:0:(file.c:995:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
00000100:00020000:2:1278393020.784325:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061b000 x1340041378196360/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 496/616 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00020000:2:1278393020.784436:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81042061b800 x1340041378196369/t0 o4->AC3-OST0008_UUID at 172.16.16.2@tcp:6/4 lens 464/608 e 0 o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00080000:0:1278393020.792064:0:6784:0:(import.c:1224:ptlrpc_invalidate_import_thread()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from EVICTED to RECOVER
00000100:00080000:0:1278393020.792068:0:6784:0:(import.c:1314:ptlrpc_import_recovery_state_machine()) reconnected to AC3-OST0008_UUID at 172.16.16.2@tcp
00000100:00080000:0:1278393020.792071:0:6784:0:(import.c:1319:ptlrpc_import_recovery_state_machine()) ffff810420d1a000 AC3-OST0008_UUID: changing import state from RECOVER to FULL
00020000:01000000:0:1278393020.792078:0:6784:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0008_UUID active
00000002:00080000:0:1278393020.792087:0:6784:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:02000000:0:1278393020.792092:0:6784:0:(import.c:1327:ptlrpc_import_recovery_state_machine()) AC3-OST0008-osc-ffff81043e754c00: Connection restored to service AC3-OST0008 using nid 172.16.16.2 at tcp.
00000400:00000100:4:1278393020.806533:0:3561:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 8 match 1340041378188715 offset 0 length 1048576: 2
00000400:00000100:4:1278393020.814069:0:3561:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 8 match 1340041378188716 offset 0 length 1048576: 2
00000400:00000100:4:1278393020.828140:0:3561:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 8 match 1340041378188729 offset 0 length 1048576: 2
00000400:00000100:4:1278393020.834691:0:3561:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 8 match 1340041378188730 offset 0 length 1048576: 2
00000400:00000100:4:1278393020.841916:0:3561:0:(lib-move.c:1818:lnet_parse_put()) Dropping PUT from 12345-172.16.16.2 at tcp portal 8 match 1340041378188736 offset 0 length 1048576: 2
00000100:02020000:2:1278394241.028730:0:7780:0:(client.c:975:ptlrpc_check_status()) 11-0: an error occurred while communicating with 172.16.16.2 at tcp. The ldlm_enqueue operation failed with -107
00000100:00080000:2:1278394241.028748:0:7780:0:(recover.c:196:ptlrpc_request_handle_notconn()) import AC3-OST0006-osc-ffff81043e754c00 of AC3-OST0006_UUID at 172.16.16.2@tcp abruptly disconnected: reconnecting
rations using this service will wait for recovery to complete.
00000100:00080000:2:1278394241.028763:0:7780:0:(import.c:167:ptlrpc_set_import_discon()) ffff81042018c000 AC3-OST0006_UUID: changing import state from FULL to DISCONN
00000100:00080000:2:1278394241.028769:0:7780:0:(import.c:608:ptlrpc_connect_import()) ffff81042018c000 AC3-OST0006_UUID: changing import state from DISCONN to CONNECTING
00000100:00080000:2:1278394241.028772:0:7780:0:(import.c:470:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: connect to NID 172.16.16.2 at tcp last attempt 4724724153
00000100:00080000:2:1278394241.028783:0:7780:0:(import.c:544:import_select_connection()) AC3-OST0006-osc-ffff81043e754c00: import ffff81042018c000 using connection 172.16.16.2 at tcp/172.16.16.2 at tcp
6.16.2 at tcp:28/4 lens 296/544 e 0 to 1 dl 1278394306 ref 1 fl Rpc:RS/0/0 rc 0/-107
g
00000100:00080000:2:1278394241.029211:0:22741:0:(import.c:182:ptlrpc_set_import_discon()) osc: import ffff81042018c000 already not connected (conn 180, was 179): CONNECTING
00010000:00020000:2:1278394241.029215:0:22741:0:(ldlm_request.c:1025:ldlm_cli_cancel_req()) Got rc -107 from cancel RPC: canceling anyway
00010000:00020000:2:1278394241.029225:0:22741:0:(ldlm_request.c:1587:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -107
00000100:00080000:2:1278394241.029446:0:3572:0:(import.c:908:ptlrpc_connect_interpret()) @@@ evicting (not initial connect and flags reconnect/recovering not set: 4)  req at ffff8102e5404400 x1340041383549552/t0 o8->AC3-OST0006_UUID at 172.16.16.2@tcp:28/4 lens 368/584 e 0 to 1 dl 1278394268 ref 1 fl Interpret:RN/0/0 rc 0/0
00000100:00080000:2:1278394241.029451:0:3572:0:(import.c:911:ptlrpc_connect_interpret()) ffff81042018c000 AC3-OST0006_UUID: changing import state from CONNECTING to EVICTED
00000100:02020000:2:1278394241.029454:0:3572:0:(import.c:1249:ptlrpc_import_recovery_state_machine()) 167-0: This client was evicted by AC3-OST0006; in progress operations using this service will fail.
00000100:00080000:2:1278394241.029460:0:3572:0:(import.c:1253:ptlrpc_import_recovery_state_machine()) evicted from AC3-OST0006_UUID at 172.16.16.2@tcp; invalidating
00000002:00080000:2:1278394241.029546:0:3572:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:00080000:2:1278394241.029564:0:7781:0:(import.c:1215:ptlrpc_invalidate_import_thread()) thread invalidate import AC3-OST0006-osc-ffff81043e754c00 to AC3-OST0006_UUID at 172.16.16.2@tcp
00000100:00080000:2:1278394241.029567:0:7781:0:(import.c:194:ptlrpc_deactivate_and_unlock_import()) setting import AC3-OST0006_UUID INVALID
00000100:00080000:2:1278394241.029594:0:7780:0:(client.c:2098:ptlrpc_queue_wait()) @@@ "mv" awake: (EVICTED == FULL or 1/0 == 1)  req at ffff81012b338800 x1340041383514587/t0 o101->AC3-OST0006_UUID at 172.16.16.2@tcp:28/4 lens 296/544 e 0 to 1 dl 1278394306 ref 1 fl Rpc:RES/0/0 rc -4/-107
00000100:00020000:2:1278394241.029668:0:7780:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81012b338800 x1340041383549557/t0 o101->AC3-OST0006_UUID at 172.16.16.2@tcp:28/4 lens 296/544 e 0 to 1 dl 0 ref 1 fl Rpc:/0/0 rc 0/0
00020000:00020000:2:1278394241.029685:0:7780:0:(lov_request.c:209:lov_update_enqueue_set()) enqueue objid 0x18ea266 subobj 0x80ab49 on OST idx 6: rc -108
00020000:01000000:2:1278394241.030074:0:7781:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0006_UUID inactive
00000002:00080000:2:1278394241.030079:0:7781:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:00020000:7:1278394241.052192:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81032a347800 x1340041383549562/t0 o3->AC3-OST0006_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00020000:7:1278394241.052453:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff810436b9ac00 x1340041383549566/t0 o3->AC3-OST0006_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00020000:7:1278394241.052749:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff8101767fe000 x1340041383549569/t0 o3->AC3-OST0006_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00000100:00020000:7:1278394241.053042:0:3571:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req at ffff81012a81ac00 x1340041383549573/t0 o3->AC3-OST0006_UUID at 172.16.16.2@tcp:6/4 lens 448/592 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
o 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
00020000:00080000:7:1278394241.221422:0:7301:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00000080:00020000:7:1278394241.221429:0:7301:0:(file.c:995:ll_glimpse_size()) obd_enqueue returned rc -5, returning -EIO
00020000:00080000:6:1278394241.232801:0:7640:0:(lov_request.c:325:lov_prep_enqueue_set()) lov idx 6 inactive
00000100:00080000:2:1278394241.435957:0:7781:0:(import.c:1224:ptlrpc_invalidate_import_thread()) ffff81042018c000 AC3-OST0006_UUID: changing import state from EVICTED to RECOVER
00000100:00080000:2:1278394241.435965:0:7781:0:(import.c:1314:ptlrpc_import_recovery_state_machine()) reconnected to AC3-OST0006_UUID at 172.16.16.2@tcp
00000100:00080000:2:1278394241.435968:0:7781:0:(import.c:1319:ptlrpc_import_recovery_state_machine()) ffff81042018c000 AC3-OST0006_UUID: changing import state from RECOVER to FULL
00020000:01000000:2:1278394241.435975:0:7781:0:(lov_obd.c:567:lov_set_osc_active()) Marking OSC AC3-OST0006_UUID active
00000002:00080000:2:1278394241.435983:0:7781:0:(mdc_request.c:1465:mdc_init_ea_size()) updating max_mdsize/max_cookiesize: 288/320
00000100:02000000:2:1278394241.435989:0:7781:0:(import.c:1327:ptlrpc_import_recovery_state_machine()) AC3-OST0006-osc-ffff81043e754c00: Connection restored to service AC3-OST0006 using nid 172.16.16.2 at tcp.
ut (7s prior to deadline).
  req at ffff810346c34c00 x1340041411931387/t0 o101->AC3-OST0002_UUID at 172.16.16.4@tcp:28/4 lens 296/544 e 0 to 1 dl 1278419851 ref 1 fl Rpc:/0/0 rc 0/0
erations using this service will wait for recovery to complete.

Peter Kitchener
Systems Administrator
Capital Markets CRC Limited (CMCRC)
Telephone: +61 2 8088 4223
Fax: +61 2 8088 4201
www.cmcrc.com 

Capital Markets CRC Ltd - Confidential Communication
The information contained in this e-mail is confidential.  It is intended for the addressee only.  If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates.  If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100708/049ef3b6/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 13489 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/attachments/20100708/049ef3b6/attachment.png>