If I am passing someone who I think is unaware of me, and could quickly move to the left in the time that it takes to pass them, I say, "On your left...Thanks"
The one issue (that you mentioned) I have with this is a surprising number of people do not know which side is there left. I have almost had accidents from saying "on your left!" and having pedestrians jump to the left quickly, or a group of people just scattering all over.
With all of the less-than-experienced/educated bike riders in Oakland, I try to let other people on bikes know that I am passing too (Mainly because many don't keep a line). I feel that it is polite, but I once had a guy look back at me and almost hit a parked car.