Did you issue the “tell http refresh” or “tell http restart” command after setting your realm ?
Realm is a way to secure access so something else is wrong. I would check the following.
The realm applies if the HTTP query contains the path defined in your realm. If the realm is not in the HTTP query, the default protection applies.
Also, make sure that your servlet is not accessed through another mean.
For instance, you included your servlet into an HTML file and restricted access to the servlet. Since authentication occurs only for the HTML file (which is unprotected), your servlet is unprotected.
I assume it works the same when accessing the servlet through a db. In this case though, you could restrict access to the servlet by restricting accesss to the note containing the servlet.
Lastly, You could also create a file protection document for you specific servlet.