I teach the Self-Defense Forces guys in my area for a speech contest, and they have nothing but good things to say about US servicemen. Hard-working, crazy beer drinkers, friendly, etc. Locals around the Fukushima area have praise for the military recently because of the 311 disaster relief efforts they gave (this news from a step-relative who was there in the service).
Like Mike wrote, nobody will even know you served unless you wear a uniform, so I don't see why you would be concerned.