Are the tables actually identical in format? Particularly variable (or fixed!) length character fields. If a field is defined as a fixed length field in SQL, most drivers will download all the trailing blanks in the field, taking up additional network time.
Verify that all fields are in fact identical in type and length (or true equivalents, since there are types that are not exactly identical).

Most ODBC drivers (or other “open” client database drivers) tend to return ONE row per IP packet. If you are using a native Microsoft Access client, it may pack multiple rows in a single packet and save a lot of network round trips. (It may also pack/discard trailing blanks in transmission.)

This can be particularly bad if the average data row size returned is just a little over 1300 bytes (or just over 1/2 that – 670+ bytes). Many systems are configured with an MTU size of 1400, which means a maximum data payload of about 1380 bytes in most packets. If your average row size is just over that size, each row can wind up taking two packets, with two network round-trip overheads. You can try increasing your MTU size and see if that speed up the response. (However, if elsewhere in your network, a router changes to a smaller MTU size, it can actually re-fragment, causing additional overhead.)
If your average row length is just over 1/2 the available payload, the same problem occurs – one row gets sent, wasting possibly half the packet space. Network round-trips are very expensive in time.

Lastly, how are you measuring the difference? Is this the time until the first “page” of results are displayed on the screen, or is this the total time to retrieve all the results to a file or other storage on the client? Access may quickly show the 1st page, while SQL Server may be retrieving ALL the results before displaying.