Unless I'm exercising, I find I prefer temps to be about 82-85 F, which is much warmer than anyplace air conditioned near me. Layers are my friend. Sleeveless tank in my living room or yard; if I have to put on the A/C so my fuzzbutts don't heatstroke, I'll throw a button-down shirt over it, usually open. I'll take it or something like it if I'm going to have to be in A/C for any length of time, even if I'm carrying it in transit. And if I know from experience that someplace I'm going will be COLD!!! I'll carry another layer.
It can be a pain, but not worse than carrying an umbrella or raincoat, and better than shivering.