A friend who is very good at Scrabble says he swaps tiles if at any point in the first half of the game can't find a play for more than 15 points. I am not good enough at Scrabble to follow this rule.
Usually when I have no vowels and no real options that don't open up a triple word score for the opponent.
all vowels; very late in the game (1-3 tiles left in the bag) when I've got letters I'm unlikely to be able to use given the board configuration
It's obviously situtation-dependent to some extent, but generally, I'll swap if my best play would 1) dispense with 3 or fewer letters, 2) for very few points (I consider anything under 15 a bad play, but that's not a hard-and-fast rule), and 3) leave me with a bad rack for my next turn. If I can play 4 letters, I usually prefer that to a swap without regard to points, unless the 3 letters I'd be left with are clearly untenable (e.g. UUV).