A *lot* of it is knowing many application internals deeply. The better tools will give you access to... well, too much, almost. Knowing how to pull off things like:

hmm, if he says he sent no mail on the 18th, then there should be nothing over here... and -- woah! that 00110001100110001100 pattern means he used Bobs-wipes-U-disk on the free space

. . . is the big differentiator between decent tool user and wowza expert. I know our guy does training; I'll ask if he has any good resources other than raw experience and research.