Taking kids to Japan?

My wife and I would like to go back to Japan next year only this time we will have a 3 year old child with us. Is this a bad idea? I know the flight will be a nightmare but will we be able to find hotels that will take the 3 of us and are happy with infants? What about going to restaurants, will we have problems?
Any advice would be much appreciated!
Thanks.

traveling with a toddler can be tough, but I wouldn't think Japan would be any more difficult than any other place. If the kid is potty trained that makes things that much easier.