It is not recommended to compact the disk every time you remove a file, as a small number of holes does not matter much in read/write operation, it is only recommended on a periodic basis depending on how frequently you remove file from your disk. All speeds being taken into consideration, it is worth doing the operation as if you compare the overall time taken to defrag/compact your disk viz a viz the speed enhancement you get in return for your next cycle of operations is still worth.
I believe that it is not advisable to compact the disk every time a file is removed. This is because this would possibly affect the efficiency of the disk and could cause hardware failure. It would be better if you perform the compacting or defragmenting of files at once in every predefined time.