CD Projekt Red lead quest designer Paweł Sasko once said that a game designer’s job is to create fun, not take it away.
In The Witcher 3, players found a way to make easy money by repeatedly killing cows and selling their hides. Instead of removing the exploit, the developers added a powerful chort monster that would appear if players abused it.
The monster was much harder to defeat, but its hide sold for even more money. This turned a simple exploit into a tougher challenge with better rewards.