Faster File copy mechanism- Want to copy a huge data from one AS/400 file to another AS/400 file.

5 pts.
Tags:
AS/400
CPYF
SAVF
I want to copy a huge data from one AS/400 file to another AS/400 file. This is copying on the same machine. I tried a CPYF command first. It takes about 45 minutes to copy that file. Next, I tried the SAVE to a SAVF and restore from this SAVF. It takes about 20 minutes now. I would like to know if there is any faster mechanism of copying this big data file. This file has about 200,000,00 records(20 million).

Answer Wiki

Thanks. We'll let you know when a new response is added.

If the target file exists then indexes and journals on that file will slow the process.
Other jobs can compete for the drives and other resources.

Phil

Hi,

Do you need an exact copy of your file? In that case have you tried CRTDUPOBJ?

Regards,

Martin Gilbert.

===============================================================

Try FTP. It is reasonably fast (compared to some other methods) between different systems. It should have similar speed advantages going to the same system.

Tom

Discuss This Question: 12  Replies

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
  • bvining
    CRTDUPOBJ across systems? I don't think so :-)
    6,620 pointsBadges:
    report
  • Rickmcd
    Have you tried the CPYF with changing the FROMRCD to 1 instead of default *Start. If the file is keyed and you use default it will copy in Key Order if you change to 1 it will copy sequentially much faster. But 20 Nillion record depending on hardware is a huge file to do anything with,
    1,630 pointsBadges:
    report
  • Gilly400
    The copy is from one AS/400 file to another - not across systems ?
    23,730 pointsBadges:
    report
  • bvining
    Sorry. For some reason I thought LPAR was in the question somewhere, but I see that it isn't...
    6,620 pointsBadges:
    report
  • Mid11
    How much of the data changes regularly? Could you use Journaling or maybe better yet triggers on the file to write/apply changes to the copy of the file.
    80 pointsBadges:
    report
  • Splat
    Fastest method I've used is a simple RPG program:
    FInput     Ip   e             Disk
    FOutput    O    e             Disk
                                      
                                      
    IInputfmt      01                 
                                      
                                      
    OOutputfmt D    01                
    O                       *all
    7,625 pointsBadges:
    report
  • DanTheDane
    Nice to see Splat's input primary solution, which is a very fast method to do sequential read of data. Having that you need an efficient way for the read-program to get rid of the records (ie. do output). Output to a dataqueue is a very fast way trunk to output data to. I'll call this the -get data- program. Another -put-data- progrm shall read data from the datqueue and add records to the target file. This solution is somewhat complex, BUT starting the -get data- program and the -put data- program at the same time, will make the two processes run parallel. Of cause the -put-data- program has to wait for records to arrive in the dataqueue before the output to the target file can take place (via the -put-data- program). Since the -get-data- program will execute and put records into the dataqueue very fast, the -put-data- program will be kept busy. I developed this solution for a client recieving approx 30K+ records per hour as a single file, which had to be emptied fast. My first shot was to add directly to the targetfile (Splat's solution) , but since the targetfile has approx 12-15 index'es attached and contains 600+ million records, this proces took too mush resource from the very limited resources available. The solution to this problem was using a dataqueue as described above, letting the -put-data- program execute in batch with LOW priority. The solution now executes very fast with with next-to-no impact on interactive applications. DanF
    2,555 pointsBadges:
    report
  • BigKat
    Dan, Doesn't this leave you open to the "put" program bogging down, and some process triggered by the "get" program ending processing an incomplete file?
    8,350 pointsBadges:
    report
  • DanTheDane
    BigCat, The full solution includes functionality in the 'put' program to submit itself for repetitive execution after n minutes within a time frame. In actual case the -put- pgm submits itself for execution again 3 minutes after *inLr was set on; - this repetitive self-submit is limited to working hours + 1 hour end the ends. The practical experience though, is very poitive - very good performance, no break-downs in the proces, and no lost data, through more than 3 years of use. I admit that I may not have understood the full content of your comment, so I actually do not know if my comments explains anything to you. Eventually clarify your thoughts on my solution, and I will comment back. DanF
    2,555 pointsBadges:
    report
  • BigKat
    was just wondering if you could end up with data stuck in the queue and some other process expecting that data to be in the "replicated" file runs over an incomplete file
    8,350 pointsBadges:
    report
  • MurrayInfoSys
    How come there is no mention of DDM? We use it regularly for large file transfers hourly. I will admit that we have a T1 for the comm. line. Maybe , the real problem is the speed of the comm line? Phil
    940 pointsBadges:
    report
  • Splat
    MurayInfoSys, the original question noted that both files were on the same system (and, it was assumed, the same partition). DDM doesn't seem to offer any benefits in this scenario.
    7,625 pointsBadges:
    report

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Thanks! We'll email you when relevant content is added and updated.

Following