The amount of time that it takes to get a recordset back will basically depend on three things.
What driver you use to connect
How fast the network is that you connect over
How fast the application which is getting the recordset can process the response from the SQL Server.
If you look at the waits which are being reported while the query is running and the data is transferring I bet that you would see either network waits or the thread goes to sleep while it waits for the client to accept more data.