batch job – performance improvement

5 pts.
Tags:
Batch jobs
Shell scripts
Hi, I have a batch job which has to read each and every record (file has millions of records), to check for some data. This is talking the whole night. Is there any way so that i can complete my job in half time.. that it is talking today?

Answer Wiki

Thanks. We'll let you know when a new response is added.

It is kind of hard to say without more information. But a few thoughts are :
> if you are using an interpretive language, it might be faster with a compiled language.
> better, more efficient coding – tighter loops, remove unneeded processing
> does your operating system allow checkpoints? You might consider taking a checkpoint so that you could restart the job.
> ensure you are not re-reading the same records. (I have seen a program where it read the first record and processed it. Then it read the first record and the second record and processed it. Then it read the first record, then the second record, then the third record and processed it.)
> have you tested this with a hundred records? a thousand? 10,000? Are the timings appropriate for the number of records when compared to the total? I had one programmer who was “only reading” but the more records the program read, the slower it got. Well, it turned out that he also doing an INSERT into a DB2 table and later was going back to SELECT from that table. As the table grew from zero rows to hundreds of thousands, the program slowed down. (the DB2 optimizer was counting on zero rows in that table).

just a few random thoughts.

Discuss This Question: 1  Reply

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
  • carlosdl
    There is no way to provide help without more details.
    67,785 pointsBadges:
    report

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Thanks! We'll email you when relevant content is added and updated.

Following