I don't have enough population to upgrade the Castle.

You should be able to do the following (at least one option):
- buy more dwellings in the Shop (Hovel, Burger's House, Merchant's House, Aristocratic Mansion);
- upgrade the dwellings you have - each upgrade gives you more citizens (click on the house and then on the arrows in the lower left corner of the house's menu);
- reach the level when you'll be able to buy more dwellings (it usually happens when you are on 10th level and you need to reach 11th level to buy next available dwelling).