You can go into the properties of the folder in your Outlook if you have the needed permissions. Go to the permissions tab and set the “default” permission level to none, and uncheck the “folder visible” checkbox on default as well. Then remove all users from the permissions to that folder. You can also simply delete the folder if you want although you did mention that is not what you wanted to do.
If you are unable to edit the permissions on this public folder, you can use PFDAVAdmin tool to do so. Download version 2.8 and install it on your computer, not the exchange server.