The packet inspection done by the ASA does take some time, and does delay the packets. You could try putting it back in, and then remove the ‘inspect’ commands one by one, and see which one is causing the most problem, probably the http inspection. Also you may have configured Java inspection or blocking, this also has an effect.
If there is a need to firewall within the network, then this delay is the price that is paid for this security. Lan-to-Lan firewalling is almost always going to introduce some delays, dispite what the marketing brochures say !
Otherwise, just firewall on the edge of the network, and ensure that only trusted user equipemtn is connected, and that you have up to date virus and trojan checking installed, and also personal firewalls on the PCs. That should give you a good degree of protection. If you need to restrict access to some areas of the network, you could use VLANs and access control to do this.