You can do this with the Domino Administrator client. You can set quotas and warning thresholds for a specific mail database file. If the quota is reached, then pending messages are held in the Mail.box until more space is created. The threshold setting enables you to notify the user before their mail file gets to the quota.
That’s it in a nutshell. The Domino Administrator Help has much more detail, specific to the release you are using.