[lustre-discuss] Whacka-mole

[rodger]

Dear All,

I feel like I'm trying to hit the elusive mole on the head! Every time I 
fix something more weirdness pops up!

Background: Upgrade from 1.8.9 to 2.10

Test upgrades in the lab worked however I broke index values in the real 
upgrade, started an lfsck and only after it had run for some time got to 
fixing the index values. After much seemingly random weirdness with 
individual OSTs not seeming to heed writeconf, hanging on mount etc. I 
did get the whole system to come up with errors in the logs that I have 
posted a separate message about.

At the moment:
I can ls to the second level in the tree.
I can cat the few files at this level. Most actual data is deeper in the 
tree.
I can lfs find and lfs getstripe all files seemingly reliably.

An ls on a directory lower in the tree e.g.
ls /exports/terra/data/gfs
hangs
HOWEVER
strace ls /exports/terra/data/gfs
works!

Likewise 'cat' on some arbitrary text file fails while 'strace cat' on 
the same file works!

Apart from the name of a good psych facility does anyone have advice on 
how to proceed?

We are pretty desperate here!

Regards,
Rodger