NAVIJO ON CLOUD

Understanding service latency is a critical aspect in designing and developing applications on AWS. In this blog we will look at the service latency and service limits of some of the commonly used data collection services in AWS.  Latency considerations for "Real time" data collection services: Amazon Kinesis Data Streams and Amazon SQS can be used for real time data collection. It is important to understand the limits of each of these service to determine how they fit in the application architecture. Amazon Data Streams limits – 1 MB/Sec or 1000 Messages/Sec Per Shard for Write. 2 MB/Sec Per Shard for Read for all consumers. 5 API calls /Sec Per Shard for Read for all Consumers. This means Amazon Data Streams have a latency of about 200ms. Go with Amazon Data Streams when you are looking for " real time " data at scale with low latency .  However remember that there is no auto scaling of shards in Amazon Kinesis Data Streams. You have to pre-calculate your sha...

This blog is the second in the series on Job scheduling in AWS. The first one focused on scheduling jobs using AWS native services. This blog focuses on an architecture to extend the on premise job scheduler to AWS.  (Link to the first blog - https://ideasforcloud.blogspot.com/2021/06/job-scheduling-on-aws.html) A common requirement when you have Hybrid infrastructure (On premise data center + AWS Cloud), is to have a single job scheduler  to schedule jobs on both the on premise and AWS Cloud environments. (Note - As I had indicated in the Part 1 blog, if your requirement is to source the data from an on premise data source, AWS Data Pipeline offers capabilities which helps you do that. You can install an AWS Data Pipeline Task Runner on your on premise server that can help manage your on premise data source).  Here are the steps you can follow to use your on premise scheduler to schedule jobs on AWS. I have outlined the steps for setting this on 1 AZ. However d...