I have around 100 GB of data that is generated in multiple files (around 250-280) over a period of 5 minutes. I need this data to be inserted in elastic search (that what I'm using) which is then available via search through the API. All this has to be achieved within 5 minutes.
I am using kafka, elastic search at present, is it the good enough to handle this.
If yes how exactly should the architecture be? How many instances of ES etc?
How do I achieve the read write of so much data within a window of every 5 minutes.