No, they can't. You lose only Crew and Resources if you fail to defend your Haven. The stronger the enemy force is, the more units and Resources you will lose during a defeat. Your buildings will be untouched. You can re-position all buildings and improvements on your Haven, but the only things you may permanently destroy are Fortifications (to make room for higher-level replacements).
Be safe out there scourin' the seven seas, ye scallywags!