You need to be able to transform into a dragon to reach the other mine entrance, so it will have to wait until you take over the battle tower (late teens to early twenties for character level).
From the quarry tower, the other entrance is a short flight to the south east (in the direction of one of the walls), and should be marked on the map.
You don't need to save the people in the tower, just get the key.
Welcome to the forum.