No, they can't. You lose only Troops and Resources if you fail to defend your Castle. The stronger the enemy force is, the more units and Resources you will lose in case of a defeat.
Please note that your Altar of Weor can be destroyed if a successful attack occurs.