Dont know if with other castles the problem is the same, but when you go in to your castle throught portal you cant back to markut/keter.

I put photos so u can understand better