Even though English isn't the easiest language on Earth to learn, it isn't incredibly hard for Europeans to learn it. It is Indo-European, so it has much in common with both Romance languages (especially since it was heavily influenced by French) and German (a sister-tongue).
But English is tough for Japanese speakers to learn. We have phenomes that they don't... many phenomes. We have about 15 distinct vowel sounds, and regional accents may add 1 or 2 more. Japan has 5 vowel sounds. In addition, English (being an Indo-European language) has definite and indefinite articles, and a very lacksidaisacal utilization of them (especially when compared to German, French, ancient Greek, etc), while Japanese doesn't. English has plural and singular forms of nouns and verbs. And English also has more verb tenses (I think, but I may be wrong). Add to that, English has an enormous vocabulary drawn from 1000 years of loan-words from French, Greek, Latin, German, Celtic, etc.
If anything, Japanese is probably easier to learn how to speak than English.