At the very least, you need to add some attribute to the source data to know whether that record has been processed or not. Without some indicator, the process will have to re-run on all 20 million records each time, even if there are only a few thousand new ones to be processed.

Beware adding a simple “flag” to a large table (i.e., a yes/no type of value). Most database engines don’t like to use indexes on columns that are “non-discriminant” (have few distinct values). When I have had to deal with this kind of situation, I usually add a numeric column and a sequence. Set new values to 0 or -1 as an indicator of records to be processed, and then set the processed records to the sequence number when done.

You can then use a compound index on the new flag column + the serial number and get the records to be processed with “select serialno … where newdata = 0″. This is extremely efficient with a two-column index – it can actually retrieve the serialno value from the index without even touching the actual data records.

Tom

Tom