I've had the strange pleasure of being sent to the Nine Hells of Baator in two separate campaigns by two separate DMs. I guess having me for a player brings that out in a lot of people. Regardless, comparing and contrasting the approaches taken by my old DMs can be a really interesting case-study on how to effectively approach Hell, and maybe even how to approach DMing in general.
DM: The Optimizer
Both times I was sent to Hell, we were playing D&D 3.5, an edition well-known for being a power-gamer's paradise. Different DMs handled this reality differently, but my first DM was definitively an optimizer. He believed in reciprocity between the DM and the players, and would optimize his NPCs and monsters one step beyond any level we had managed to accomplish with our characters. Naturally, this reflected closely in his DMing style: I remember spending almost two months (meeting once a week for about four hours) in a single dungeon, routinely spending each session in one combat encounter at a time. There was very little storytelling and very little NPC interactions that we could leverage to our benefit.
So Hell was more appropriately a combat hell, filled with unwinnable encounters and battles so large that our PCs could barely affect them. Regardless, we could, and did, stroll into almost every combat we were presented with. Thanks to a few very powerful PCs at the table, we almost always scraped out of things in one piece.
This was very true on the first layer, a plane filled with battle between the demons of the Abyss and the Devils of Hell. My character was a bruiser efreet (using the Savage Species rules from that edition) and was far from optimized, despite my best efforts. In the every first session, I failed a stealth check, which alerted a large army of devils, who attacked up on sight. Foolishly, we charged into the fray and spent the next two sessions in a single, massive combat that did nothing to progress the campaign.
Dis was up next, which we made it to via Tiamat's Tower (where I distinctly remember my DM playing her as a seductress, just to completely weird out one of the players.) This layer, dominated by the idea of bureaucracy, was an extended, multi-session game of finding ways to optimize the forgery skill of the party to sneak by dozens of checkpoints and devil patrols.
We spent the majority of our time in the first two layers of Hell, but ended up migrating down the River Styx on a devil boat we commandeered from the villains of the campaign to get between layers. Our joyride down the evil river stopped on the third level, where things became too shallow and swampy to continue. We abandoned the boat and set off on foot, getting lost in the endless fog, and eventually (by the luck of a d100 die toss) found an NPC which rescued us from Hell for plot reasons.
DM: The Storyteller
The second DM I've persisted with though 5th edition. Though his DMing style has been refined over the years, he was largely a storyteller when we went to Hell. I recall that he poured over 2nd edition Planescape rulebooks, and even published adventures, which remain the most comprehensive sources for detailed descriptions of the multiverse.
This time, the trip to Hell was clearly motivated: we were lost in the multiverse after a BBEG's death tossed us to the Outlands, and we knew there was a way to Sigil through Hell. Problem was, my PC, a highly optimized spellsword type, was sent across first, which happened to be weeks early, because of the convoluted nature of time in the planes. Most of the trip was side-focused on tracking down that PC (now NPC), and discovering clues that he left behind though the layers of Hell, which revealed that he was reconciling a life of wrongdoing with his time alone in Hell. For him, it was a psychological journey; for the rest of us, it was tracking quest.
The pacing in Hell was far better this time around, and, though the combats weren’t accurate to the over-the-top setting, there were still plenty of memorable ones to go around. That being said, I don't remember any point where I felt like Hell was going to kill us, drive us mad, or consume us whole. We stopped after only two layers, bailing out when we tracked down the character in Dis.
If I ever do Hell, I'm going to try a synthesis of these two approaches. If you confront your players with dozens of unwinnable combats, you either devalue the horror of the setting or party-wipe them, neither of which make for a good game. But if you simply allow the characters to progress mostly unharmed, it doesn't really embody the setting. The trick, I think, is to bring hell in ways other than combat.
I loved following the story of someone reconciling their misdeeds in a place designed to punish them for it, but I think it would have been far more engaging if my current character was going through it. If only one of my DMs had sent a devil hunting party after us, who were dedicated to punishing us for our specific sins in life, that would have been far more personally effecting.
Also, Hell should risk driving you mad. I think if I ever do a hell campaign, I'm going to end a session with the party escaping through a portal to Sigil, then begin the next session with them waking up caged and tortured by devils; they only imagined escape. I'd be putting illusions and disguised NPCs all over the place to really keep the players on their toes.
Lastly, I would have at least one really scary, explicitly 'no-win' combat each session. The characters can avoid this through stealth or panicked escape, but it's very wise to remind them that, in Hell, everything is dangerous, and everything can kill you. Without that type of reminder, things can feel too safe.