I ask this early, but nobody answered it.
Why exactly do you feel it rude to wear a baseball cap at the dinner table? In what way does it exactly offend you?
Removing your hat has more to do with age old tradition that live on today than being a respectful thing.
For instance, they say you don't wear white after labor day. 99% of people do not even actually know why that faux pas even exists in the first place.
For those that don't know, the do not wear white after labor day faux pas has nothing to do with wearing white shirts or pants, it has to do with wearing white dress shoes. The whole point behind not wearing them wasn't because it was incorrect, it was due to the fact that they were inappropriate to wear in winter weather! You didn't wear white dress shoes because you didn't want to get them dirty. Wearing white anything else was because people that didn't know what they were talking about, talked about it confusing the reasoning.
The basic fact is, removing your hat when you are indoors is an act of tradition, not an act of being rude or disrespectful. Not everyone follows everyone else traditions. In that case, you SHOULD remove your hat when you enter someones how who DOES hold that tradition as not doing so would then be a show of disrespect of the mans traditions in his own house. If you are in a public place (even if it's a 5 Star restaurant) you don't have to remove it unless the establishment requests that you do. The reason is, this is America and you don't have to respect someone else's traditions in a public place like you should when you enter their home.