I think that fandom has its own parallel "legal system" that operates in a (usually) non-cash way. So I agree with everyone who says that for profic, an audiobook is a derivative work that requires consent of the copyright holder. AFAIK a fan chooses to record a podfic because ze likes the story and wants it to have a wider audience (or be experienced by the audience in different ways) so why NOT ask first? The podfic artist usually isn't saying "I have a right of reply," but "Your story is awesome and I want people to hear it!"

Performance certainly makes a difference--you can't really say you've "seen Hamlet" because it matters who's playing the roles, whether it's set in 14th century Croatia, what the sets and costumes look like...