This isn't even a hot take at all, it's literally canon Valyrian history. Back in the Freehold, the Targaryens were minor, bottom tier nobility who only survived the Doom because Daenys the Dreamer had a vision and they fled.
Once you strip away the dragons, their power base in Westeros is completely pathetic. Dragonstone is just a cold, rocky island with no massive armies or wealth. They don't have Lannister gold, Tyrell crops, or Stark territory, without their dragons, the major lords would have laughed them off the throne.