You must beat your opponent by having the main part of the territory. There are three fortification levels,to obtain the control over one square you must have more influence on it than your adversary. If you have more influence than your opponent on a square occupied by them, the castle is destroyed.