That’s quite the underestimation of what would happen. DM here with an overactive imagination for NPC villainy, a few possible scenarios unfolding:
1: You’ve heard of gangs, right? When one is betrayed and they’re too weak and/or cowardly to do the vengeance themselves, a group becomes stronger than the individual and overwhelms defenders.
2: You’ve heard of stalkers and arsonists, right? The revenge need not occur immediately. Either wait until after hours to isolate the target or leave come back with an arsenal and set fire to the location.
3: He’s not the only victim of that charm spell. If there’s any others who have shared a similar fate and saw what just happened, they might join in the revenge tale. After level ups, they might be more capable than the first time.

As the charmed fool isn’t a spellcaster himself, magical options are out of the question … but this is where the story could go, if it wanted a darker twist.
Considering the Darkest Dungeon character and aberration-type horrors introduced in previous chapters, that’s certainly a possibility where this goes.

That’s why bars have bouncers of high level – if you’re high enough level to beat the bouncer, you’re high enough level to resist the charm attempt.