Formal is good. Formal shows respect. Formal shows professionalism. As an American, we tend to think that everyone wants to be friendly and wants to do away with formalities and break down unseen barriers between people, but in Japan this kind of familiarity is just awkward. Don't worry about making friends with your contact...a solid business relationship doesn't need to be casual or super friendly.
Cordial and formal, or cordial and "relaxed formal" as you become more comfortable with each other is a good thing. Be punctual, acknowledge mails or shipments, do everything you can to make the other party's job easier, obviously pay on time. Take responsibility - don't say "sorry I didn't respond sooner, my car broke down" or whatever the excuse is. Japanese companies don't want to hear the excuse. They just want you to take responsibility. "Sorry I didn't respond sooner. I apologize for the inconvenience I caused" is the correct response.
Emailing on weekends is fine (but don't expect a response). If you do get a response, you can say, "thank you for responding so quickly" (recognizing their extra effort).
I don't know how you address your partner, or if they have indicated some preference for how you should address them, but here formality is also obviously good. Mr/Mrs Familyname, unless otherwise advised.
All basic stuff, I guess, but Americans sometimes find it hard to suppress the urge to be casual/funny/jokey, because they find formality somewhat cold and awkward, i guess, but in Japan "formal" is preferred, unless you have a genuine close relationship (but even here, business correspondence tends to be formal).
Sending gifts? This is a tough one. In Japan, some
gift-giving used to be standard among partner companies - especially at New Year's and mid-summer. Nowadays, companies are refraining from this more and more, partly for cost-cutting, and partly because the custom can be problematic from a compliance point of view. But old fashioned companies still exchange gifts - and department stores package up items especially for
gift-giving: boxes of crackers or biscuits or canned juice, etc. As a foreign company I wouldn't spend too much time wondering whether I should send a gift. Sending it and getting it through customs might be more trouble than its worth AND it might make your partner feel obligated to reciprocate. So, my advice would be to not worry about gifts. Being aware of Japanese holidays, on the other hand, would be good. "Dear Mr/Mrs Suzuki, I hope you enjoyed your Golden Week holidays. ...". But that is about as far as I would go.