Okay, so coming to the data pipeline, what we do is like, we pull the data, I told you earlier, we pull the data from the different kind of APIs, how we do pull the data is like, we have a poller running, that is a Python script on ECS, Amazon Container Services, that goes to that API, hit with a certain amount of time, start time, end time, we get the data, okay, as a response, we put that data to Amazon Kinesis, and then that based on the trigger, trigger lambda, so from there, the Kinesis, the lambda has a trigger, okay, we have set up a trigger for that, so lambda handler will be invocated, that transforms the data and put that data to Elasticsearch, between this, this is the complete end-to-end pipeline, okay, transform that data into our own format, and also some kind of a nonsense we do, okay, so apart from this, in the same pipeline, what we do is like, we have to push this data to a Kafka topic, this is like an old pipeline running, okay, and we have to push this data to the Kafka, push this logs to the Kafka, basically, what kind of article, what kind of data we are processing, we do that logs and push that to the Kafka, so basically for each batch, 25 articles being processed, so we push 50 logs for that, one for the start of the, start of that particular, start of the processing of that particular article and when the article gets processed completely, so major challenge was here to optimize the performance, because what was happening, just without Kafka push, it was taking 300 to 350 milliseconds to process that data, 25 articles, but when it comes to Kafka push, okay, 50 records get increased, means 50 logs needs to be pushed to the Kafka, so it was taking around 2.5 seconds, so it has huge impact of the performance when it comes to the scale data, right, 2 seconds for the each batch, it is not a good timing, so what we did is like, we have optimized the performance to push all the records in a batches, not to push each of the log in a, in one by one go, not, no, that is not a good choice, okay, when we analyze that, we found that when pushing in a batches will create less latency, less like, you will get a performance, good performance, so what we will be doing, we will collect, we will collect, create a global variable or global connection to the Kafka, once the, once the logs started getting generated, okay, what will we do, we will put that into a list, started, ended, start of processing, end of processing the article, okay, we will put that into a list, and once the list is been completed or the processing of the article is completed, we will push all the Kafka logs in a one, in a batch, in a batch of 25 or a batch of 50, you can do that, so what will happen, you do not need to wait for your, basically, you do not, what will happen here, you will not push your data in a single, single, single, single run, so what will happen, you will flush all the 25 articles at a time, so Kafka will be reliable here and that is how we have optimized our performance and we have down the duration of 2.5 seconds to 400 milliseconds, only around 35 to 40 milliseconds of timing has been increased to process 50 records, basically, 50 logs, so there is a minimum and the performance is quite good right now, so we are, we have implemented that, that is how we have, that is the challenge that I face and that is how our pipelines are and a new pipeline that we are building based on the Kafka and Apache Beam will be a quite different, that will pull, that would not pull the data, that will directly, directly fetch the data or the source will directly put the data through a Kafka topic, from there the consumer will work, consumer will take the data each, like maybe in the batches, maybe, it is different for every pipeline, every kind of a data, in the batches or in the, in a single, single way, we transform the data, push the data to the elastic, and it will be a much faster, because it will be running, and the clusters and all those things, we will manage in the same network, so network latency will be very less, so that is how we are maintaining our pipeline.
