There is a program called <a href="http://www.faronics.com/">DeepFreeze</a> which enables public workstations to revert back to their original state after reboot. If you can't find a suitable freeware or shareware option, it might be worth buying some licenses for this product. Optionally, you might look at <a href="http://www.microsoft.com/windows/products/winfamily/sharedaccess/default.mspx">Microsoft's Steady State</a>. It's free and does exactly what you want it to do.
It's fully configurable. I locked down a banking kiosk with the Steady State predecessor, Microsoft's Shared Computer Toolkit; no matter what anyone tried to do to it, when I rebooted, it went back to the way I set it up. I'm implementing this on the public computers at a local crisis center where, unfortunately, some of the clients like to visit porn sites. Doesn't matter; reboot and all history, bookmarks, cookies, trojans, viruses, and crap are gone.
Without any software, You can set your user shell to iexplore.exe in Group policy, so that when they login the desktop is internet explorer instead of the regular windows desktop. That gives you internet explorer with NO desktop, NO start bar, nothing else. But isn't terribly secure or make it rever back when rebooted by itself. You could then set the do not save settings at exit in group policy to prevent users from modifying the desktop environment. Additionally, there are a number of questions answered here on group policy--- specifically with respect to internet explorer.
Since the machines are older, you could look into a linux OS since it is free and might actually run better than Windows on these machines. but if you are not very familiar with them then you would have to find out how to lock it down too. so you are back in the same boat.
Aside from that, you could perhaps use a Knoppix liveCD, which is a bootable Linux operating system with apps. However if I'm not mistaken Knoppix comes pretty well loaded with apps, so to trim it down you might explore creating your own LiveCD from your favorite Linux build (there are cookbook-style guides on how to do this, search for "Linux LiveCD").