Rick,
let me start by saying that we are by no means historians. Furthermore, I assume you have been to cinema recently, I am astonished to see how much controversy the movie "Pearl Harbour" stirs up.
Your question is interesting indeed and has already filled an abundant number of books. We all know the historical events from our textbooks (Japanese militarism and expansion in Asia, "China Incident" etc.), but - in order to cut it short - I would like to list a few issues/unfriendly acts that might have led to anti-American sentiment in Japan and to a significant deterioration of bilateral relations:
=> Large expropriations of farmers of Japanese descent by the State of California in 1913
=> Disarmament Conference 1921/22: Japan was refused to be treated "inter pares" and walked out of the conference
=> Cable Act, 1922: branding Japanese (among others) as "aliens ineligible for citizenship"
=> Exclusion Act, 1924: prohibiting all Asians to immigrate to the U.S.
=> Plan Orange: created after the turn of the century, amended in 1937 by the U.S., depicting Japan as hypothetical enemy in the Pacific, carried out in WWII
=> Oil Boycott in 1941, as a reaction to Japan's aggression in Asia
All of these events, of course, should not hide the fact that Japan was the clear aggressor in the Pacific War.