20 pts.
 Script to extract files information from ftp server
Hi Wonder if you could help me with the following please. I would like to construct a script (in any language whichever it's easiest) which when run, it will log on to a ftp server using a generic account and password, then list the files in a specific folder that begins with the string "abc". The file names along with their creation date are then to be extracted on a csv file (or similar). Can this be easily done? Many thanks in advance!! Ken

Software/Hardware used:
ASKED: September 22, 2008  9:34 PM
UPDATED: March 19, 2009  8:51 AM

Answer Wiki:
You will need to download the files. FTP does not support launching an extract command on the remote server. <a href="http://www.columbia.edu/kermit/ftpscripts.html">This site</a> shows how to automate getting files from an FTP Server. ************************* FTP Clients can show the user a directory listing from the remote server, so it can be done. We do something similar (but without getting the creation date), but we don't do it with a script, but with a small VB application. **************************** Expect would work well for this. Check out the following for an introduction. http://www.gnulamp.com/expect.html http://expect.nist.gov/
Last Wiki Answer Submitted:  September 23, 2008  2:59 pm  by  Peterp   270 pts.
All Answer Wiki Contributors:  Peterp   270 pts. , carlosdl   63,535 pts. , Denny Cherry   64,520 pts.
To see all answers submitted to the Answer Wiki: View Answer History.


Discuss This Question:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _


 

Hi,

I don’t know if your FTP server has this functionality, but on the AS400 you can use a “dir (DISK” command for example :-

dir abc* (DISK

which will create a file with a list of the directory contents, which you can then retrieve using an FTP get command.

Regards,

Martin Gilbert.

 23,625 pts.

 

One method we use is to have the script execute FTP twice. The first time it makes a connection to the ftp server, it issues the “dir” command and puts the output in a file on your local server. The script then closes the ftp session and checks the contents of the local file for anything that it should be picked up from the remote server. The script then calls ftp a second time to do the actual transfer. For each file transferred, you can then check the size of the local file to the information about the file that was obtained on the first ftp call. If they are not the same, then the file was not transferred successfully.
For example:
#set a variable with the wildcard of files you are looking for
FILEWC=”S*”
These two variables hold the value for the file name where I hold information from the ftp sessions
RFINFO=”$LDIR/remote_directory_info.dif”
RFINFO2=”$LDIR/remote_directory_info2.dif”
FTPLOG=”$LDIR/ftplog.log”
FTPHISTORY=”$LDIR/ftp_history.log”
# other variables are also used to store the name/address of the ftp server and the connection #information. This makes it easy to modify the script to handle multiple ftp requirements:

ftp -nciv > $FTPLOG 2>&1 <<EOT
open $FTPSERVER
user $RUSER $RPWD
cd $RDIR
dir $FILEWC $RFINFO
EOT
FTPSTATUS=$?
if [ $FTPSTATUS -ne 0 ] ; then
return $FTPSTATUS
fi
NumToTransfer=`cat $RFINFO | /usr/bin/wc -l `
if [ $NumToTransfer -eq 0 ]; then
echo “$CURDATE – No files to transfer” >>$FTPHISTORY
else
echo “$CURDATE – Files to transfer: $NumToTransfer” >>$FTPHISTORY
FLIST=`cat $RFINFO | awk -f $LDIR/awkname.dat`
fi

#in this code snippet, it builds a list of files to retrieve into a variable named FLIST

There is then another piece of our script that actually makes the second connection, for each file to be transferred, and retrieves the files. The variabel ftpmode is used to pass the command to set the transfer mode of either BIN or ASC.

for eachfile in $FLIST;
do
ftp -nicv >$FTPLOG 2>&1 <<EOT
open $FTPSERVER
user $RUSER $RPWD
cd $RDIR
$FTPMode
dir $eachfile $RFINFO2
get $eachfile
quit
EOT
FTPSTATUS=$?
if [ $FTPSTATUS -ne 0 ] ; then
return $FTPSTATUS
fi
# Check the file size that was actually transfered
FSIZE=`/usr/bin/ls -l $eachfile | /usr/bin/awk ‘{print $5}’`
RFSIZE=`cat $RFINFO |grep $eachfile | awk -f $LDIR/awksize.dat`
RFSIZE2=`cat $RFINFO2 |grep $eachfile | awk -f $LDIR/awksize.dat`
if [ $FSIZE -ne $RFSIZE2 ] ; then
echo “$CURDATE – FILE $eachfile SIZE NOT EQUAL – $FSIZE – $RFSIZE2″ >>$FTPHISTORY
echo “File size not equal!! Local: $FSIZE, Remote: $RFSIZE2″ >> $FTPLOG
FTPINCOMPLETE=1
return
fi
echo “$CURDATE – File $eachfile received – Size: $FSIZE” >>$FTPHISTORY
let NumTransferred=$NumTransferred+1
if [ $NumTransferred -ge $FTPLIMIT ] && [ $FTPLIMIT -ne 0 ] ; then
break
fi

I may have missed documenting some of the variables here, but I hope this helps.
Regards,

Larry

 20 pts.

 

Hi HolmanSAP
What is awkname.dat and awksize.dat ?
Can you please help me with this?
The FLIST is not being created correctly in my script.

Please help.

 10 pts.