I don't think gender in general is inescapable. I think that we, as already gendered beings, are likely to imagine a unisex world as something more in line with one or the other of the genders our society already gives us--the way your geek utopia just erases, say, jocks. It's a failure of imagination which is the result of our already being molded by systemic injustice.

I'm going to go out on a limb and assume that in your household where everyone wore similar clothing, the men didn't wear skirts (although if they did, awesome!); instead, most likely, the "unisex" effect was achieved by the elimination of traditionally female clothing--everyone dressed "like men." Do you see why I would find that problematic?