Yes you should be able to do this. Just make sure to disable the DHCP Server.
Do not use the WAN port (Internet connection) for cross connecting between the old router and your new router. Use just the four (4) switch ports. Depending on the system you may need a crossover cable or adapter to connect the two routers together. You can test without a crossover connection and see if it will work. If not, a crossover cable/adapter is required.
In the past, a crossover cable was used to chain switches together. Newer switches may auto detect if a port is chained to another switch and act as a crossover adapter. Some switches only do this with a specific port or have a button to manually change the mode of a specific port.