If you visit for less than three months, you are expected to be able to produce your passport when asked by the police. I'm guessing that the gaijin card serves this same purpose, and lets the longer-term visitors/residents leave the passport at home.