The line between confidence and arrogance is ambiguous. Doing anything bold inevitably leads to some people viewing you as arrogant, when in reality it's merely an act of self-confidence.
The definition of arrogance is somewhat subjective. What one person views as arrogant, another might see as self-confidence. This is something I've personally experienced.
When I was considering leaving school to pursue my own path I shared this with a few people. Some of them claimed I was arrogant on the basis that I believed I was "the exception to the rule".
But what they viewed as arrogance I merely saw as self-confidence. I didn't think I could do this because I was the exception to the rule; I just believed in myself enough to have the confidence to walk my own path.
You'll inevitably encounter resistance when pursuing anything audacious. When people say things like this they're projecting their self-belief, values, and self-confidence onto you.
Their judgments are based on their view of what's sensible and what's not. Their negative worldview is developed by society pushing us to do what everyone is doing, follow the rules, and play it safe.
Practically every person who has achieved great things did the opposite. They ignored nonsense paradigms, believed in themselves, and achieved great things because of it. Who do you think you should listen to?
You'll disappoint someone no matter what you do. You could be the most incredible person in existence and some people will still dislike or even hate you. Therefore you should just be yourself, believe in yourself, and do what you want.