Hmmm this is getting a little confusing for me @.@... just my take on it... Miyavi is probably the most difficult J-music star to say this about! because he does different things - music, acting in movies, and of course so much photoshoots you can almost say that he is a model as well! ^^ And like you pointed out, I'm sure he couldn't choose which aspect of what he does he likes the most or would want other people to be most interested in... well he said himself he cannot choose.
My view is simply because music is the most important thing for me, so looking from my perspective, if I don't much like someone's music then I will not consider myself a 'fan' of them; that's only because music is my main interest (for example, I don't have any movie actors I'm a fan of, because I don't have so much interest in movies). That's my personal viewpoint and nothing to do with objectiveness or what Miyavi might/mightn't want.
I think it's ok to be a 'fan' of only one aspect of what someone does providing that it's clear that they do have these separate aspects - for example, an actor who does modelling; you could say he is a very good model but a lousy actor, or vice versa, and still be a 'fan'. ^^
With Miyavi, I view his music as the major part of what he does because that is the route by which he started off, on his career, and what he built his success on, and still the primary foundation of his career and 'persona'; the 'modelling' that he does is still based around promoting his music (through his personality), and although he acted, in a movie, it is not so much that it's a significant other branch of his career.
What I am driving at... is... Me, I am not comfortable to call myself a fan, I would never call myself a 'Miyavi fan' because although I think he is very good-looking, I don't like his music enough to consider calling myself a 'fan' of his. Don't get me wrong, I like his music fine enough on balance, but not enough to call myself a 'fan' I think. BUT I am quite happy to hear other people calling themselves 'fans' for WHATEVER reason that they want, whether it's the music or just his looks alone. BUT I have to add that it can be annoying if people go on and on about his looks and how hot he is, but dislike his music or know nothing about it... it just strikes me as a bit disrespectful... because although Myv himself had said he can't choose in most important being music or other things, but he does work significantly on his music and it's disrespectful in my view to disregard that... I'm not saying everyone has to like the music but they should at least listen to it and have an opinion on it... since that is part of what makes Miyavi who he is...
Sorry for long post.