I believe that most people are self-conscious about their physical appearance because they have zero control over the external judgments made by others. If you are in the world - you are being judged - like it or not. In contrast, your internal personality is something you can hide or parse out to others as you see fit. If you’re silent and your body language is quiet there is nothing to judge - it can only be imagined in the mind of someone else. You can choose whether or not to open your mouth. You are in control.
If you’ve never played golf this next thought might not make sense to you but I would submit that there is no better way to get to know someone than to spend 4 hours with them on a golf course. There is no way to hide from yourself when you play golf. You can learn everything you need to know about someone by how they carry themselves through a round of golf. Are they honest or a cheat? How do they handle adversity? How do they handle things they cannot control? Are they selfish and egotistical or courteous and thoughtful? Are they humble? How do they treat others? All of this and much more is revealed after 70-120 swings of a golf club.