Once you take over the Battle Tower you have signed a death warrant for all those poor people of Broken valley. The Poison stays and your only access is flying back in from the Fjords.(or teleporting)There are a few people left that are involved in quests but all are accessable thru dragonform and dropping into safe areas.
Its a tossup for me on which is easiest to clear first (a)broken valley, or (b)Stones Fortress...