You mean in his movies where he was acting? In reality the Duke was a huge fan of hats indoors. He was often seen in his neighborhood Applebees sporting his Allman Brothers trucker hat, shirt tail out, and elbows on the table. If you were to confront him about it he would simply tell you "I'll have you spread-eagled on a wagon wheel".
