It's also just not that fun to be a Duke fan, all else equal. The bulk of fans and media are against you, and in ways that would seem more fucked up if we weren't used to it.
There's this "sorta tongue in cheek sorta not" idea that Duke = inherently immoral and evil that is generally accepted. If they win, it's discreted due to officiating, flopping, whatever. If they lose, they're still evil, but also lame, dorky, white, soft, dirty, etc., and deserve any pain and ridicule coming their way - and NEVER any empathy, since they are subhuman. This includes games against UNC, where it's all magnified.
It's fine for an entire team to skip handshakes to run up on Duke's student fans who just absorbed a brutal loss and talk a bunch of aggressive shit, and if a couple of the fans throw water bottles, the focus should simply be on how that's disgusting and indicative of their evil nature. It's fine to rush the court and steamroll Duke's players, but the player yelling in pain as a result the ensuing injury deserves ridicule for probably lying and being a pussy and deserving it anyway so on.
I know there's some degree of dehumanizing that always happens with sports fandom, but I really think it happens to an extreme level (and degree of general acceptance of it by the zeitgeist) with Duke basketball. There's zero good-natured "sportsmanship" left in it for most people, and not even much underlying competitive respect, where it's fun to "hate" but they recognize where it comes from and keep context. I genuinely think that if Duke's whole team got murdered on the court, there's a good chunk of the country that would celebrate and have no doubt that they deserved it, simply because Duke.
It's a pretty fucked up dynamic to have to constantly deal with for "fun", and the gleefully-socially-accepted/encouraged kicking while down makes the low points more frustrating than they are for other fanbases, IMO.