I would like to share my experience with a project focused on real-time data processing. At Onclusive, I managed real-time data and encountered several challenges throughout the process.

One major task involved processing data from Kinesis in near real-time. It was essential to ensure timely processing while maintaining high performance. We optimized the concurrency of AWS Lambda functions and implemented data batching to improve efficiency. For example, we processed 25 articles in a single Lambda function, which significantly reduced the overhead associated with launching multiple Lambdas.

Another challenge we faced was sending the processed data to Elasticsearch and Kafka. We generated logs for each article—one at the beginning and one at the end. By batching these logs and sending them to Kafka in groups of 25, we enhanced both efficiency and timeliness. This approach minimized connection times and ensured successful log submissions.

For Elasticsearch, we utilized bulk operations to push records. Instead of updating records one by one, we sent them in bulk, which decreased connection times and repetitive tasks.

By optimizing concurrency, batching data, and employing bulk operations, we effectively addressed the challenges of real-time data processing, leading to a more efficient and reliable system.