Captains!
Click the "Edit" button in the top right-hand panel. You can move all of your structures, but only Fortifications and Improvements can be permanently deleted once they have been constructed. Remember that you can expand your Haven if you find yourself running out of space. You can also upgrade Fortifications to higher levels.
Be safe out there scourin' the seven seas, ye scallywags!