Short answer, No. Doing so would not be advisable.
Long answer, you could accomplish this task with a lot of planning (more than I’ve put into here.) This solution would <i><b>not be recomended</b></i>, because it will cause any number of situations to arise that could create you problems with the server and break much of the windows management model with respect to the server’s managability.
You could modify the default membership of the local administrators group to <b>not </b> include the domain admins, enterprise admin’s group, and the user’s ID. This would in essence not allow the user to log in. Then go back and add the people who you need to be able to log onto the server into the administrators group on the server. <i>(<b>Caution</b>: This will mean that any time any of these people ever leave or change roles in your organization, in addition to modifying their group membership in Active Directory– you will need to do so here as well.)</i>
I would highly recommend trying this on a test server first, preferably one you can rebuild if it doesn’t go well with out much pain. Partly, because you may end up in advertently removing rights and priveledges from domain admins everywhere that “Administrators” is specified in the local security policy of the system.
I’m certain I’m forgetting some other things that would need to be done as well to accomplish this, anyone else feel free to add to “the pile”.