Enterprise IT Consultant Views on Technologies and Trends

Dec 5 2011   2:15AM GMT

Integration using Bulk Data Transfer



Posted by: Sasirekha R
Tags:
bulk
file transfer
Integration

Integration using Bulk Data Transfer

While most of the aspects discussed so far are related to online integration, it is worthwhile to mention that 80% of today’s integration relies on basic technologies like file transfers. File Transfer based integration though has the negative factor of High Latency, brings in lots of benefits and hence should be the preferred choice wherever possible.

File Transfer Make the data available by transporting a file – to any format like xls, csv, xml, flat file – that is an extract from the application’s database so that other applications can load the data from the files.

The plus points are:

  • Local copy of the data exists – and high performance (for both applications)
  • Maximum level of Isolation – each application can be independent of each other – and only communicate on a planned basis for data replication
  • Security constraints can be enforced at file usage level
  • More efficient as it is getting chunk of information at one go
  • Can be used in both Push/Pull mode, depending on the business requirement
  • Preferred for Updates – as the Validations and data integrity checks can be built in before bulk updates

The downsides are:

  • High Latency – needs Latency Tolerance
  • Data replication and Synchronization to be handled – if the communication is two-way

 Other factors are:

  • Isolation possible – using data transformation
  • Need to have Simple Conflict resolution Logic – involving human intervention, if required.
  • Medium Complexity to build

Conclusion

  • Use Wherever possible – within the context of data integration
  • Exploit application’s Import and Export facility
  • Ideal if there is latency tolerance (if considered from pure business perspective, this becomes applicable in large number of real-life situations).

 Comment on this Post

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: