The line between confidence and arrogance is ambiguous. Doing anything bold inevitably leads to some people viewing you as arrogant, when in reality it's merely an act of self-confidence.
The definition of arrogance is somewhat subjective. What one person views as arrogant, another might see as self-confidence. This is something I've personally experienced.
When I was considering leaving school to pursue my own path I shared this with a few people. Some of them claimed I was arrogant on the basis that I believed I was "the exception to the rule".
But what they viewed as arrogance I merely saw as self-confidence. I didn't think I could do this because I was the exception to the rule; I just realised it was possible for anyone and therefore had the confidence to do it myself.
You'll inevitably encounter resistance when pursuing anything audacious. When people say things like this they're projecting their self-belief, values, and self-confidence onto you.
Society has a huge influence on our beliefs. Their judgments are based on their view of what's sensible and what's not: following the norm, playing by the rules, and playing it safe. This means anyone who steps out of this is likely considered arrogant from their point of view, even though it's technically not true.
Practically every person who has achieved great things did the opposite. They ignored nonsense paradigms, believed in themselves, and achieved great things because of it. Who do you think you should listen to?
You'll disappoint someone no matter what you do. You could be the most incredible person in existence and some people will still dislike or even hate you. Therefore you should just be yourself, believe in yourself, and do what you want.