You don’t say if you are talking about a process or a user limit. If you are talking about a process you can
use ulimit to restrict file sizes that are created. I don’t believe that you can restrict this to a particular folder though.
If you are talking about a user you can set up user quotas. However this is going to work as a total limit for
given file system, not on individual files.
In either case you can find out more (it’s too much info for this type of post) by seeing the man page for quota or
info page for ulimit.