I never felt unsafe when I went to New Orleans for the Saints game 2 years ago. Before the game there was minimal trash talking and we were absolutely embarrassed during the game, so there was no risk in them fighting me, but I have a pretty cool story for you.
On the Saturday night before the game, me and the group I was with were partying on Bourbon Street up on a balcony above one of the many bars. We saw Jerry Jones (surrounded by a posse) stroll down the street as he was on his way to a late dinner (my immature friend actually threw a set of beads around his neck, which was, low-key, very impressive). Later that night, on the same balcony and after a few chardonnays, I struck up a convo with a guy on the balcony right next to me. It happened to be a high ranking Cowboys ticket salesman and I realized all of the Cowboys higher-ups were right up there with him and I. They mostly stuck to themselves, but I can be a little overly-friendly when I am intoxicated, and luckily, apparently, so is Steven Jones. I called his name and he came stumbling over and was talking trash to my friend next to me who had a Giants jersey on. I actually bought him a shot of Fireball (which he nearly vomited on me). Yes, I bought a billionaire a shot.
Moral of the story: definitely go to Bourbon Street on the Saturday night before the game and keep your eyes peeled!