Posts

A case for Narrow - Purpose built LLM’s

Image
The days of using weather as conversation starter seems to have long gone, at least within the tech circles. Generative AI has not only overtaken weather as the conversation starter but is often coming out to be the only conversation topic. Everyone wants to know what everyone else is building using Gen AI, what proof of concept (POC) are being considered etc. Large Language Model (LLM) names and their capabilities are talked with the excitement trailing even the Thanksgiving deals. Yet, a new report about Gen AI adoption and usage published last week seems to contradict this weather replacing hype. The report indicated a decrease in traffic to ChatGPT and other Gen AI services, for the first time in several months. This traffic report further showed that this decreasing trend was in both new traffic, as well as traffic from existing users of these Gen AI services. What might be causing this gap between the hype and the actual usage? When Gen AI opened for public, it was that shi...

Accessing Data from an AWS data lake using AWS Lake Formation – Part 1 - Data Filtering based access control

Image
A data lake is a repository to store your data. Like a database, a data lake is expected to have the data in an organized manner, provide tools for data processing and data access, and have well defined methods for authentication and authorization. And unlike a database, data lake is expected to hold structured, semi-structured or unstructured data, and are envisioned to hold this data forever.   It is these subtle differences that make data lakes fit more for analytic needs such as deriving patterns, detailed comparisons, build an exhaustive story etc. A database can also be used for analytic use cases, but they are not meant for storing large volumes of data indefinitely, or store unstructured data, and this in turn puts a limit on how detailed of a result it can provide. Whether you need a data lake is a question that depends on the volume of data in your organization. If your organization data is small; you do not have much history or you do not have a need to store large histo...

AWS Instance Role and Instance Profile

 In this blog, let us try to understand what are Instance Profile and Instance Roles in AWS. An Instance profile is an identifier for an Amazon EC2 instance while an Instance Role defines what can the user (in this case the Amazon EC2 which assumes this role) accomplish. In some way, think of the Instance Profile as the designation – Architect, which the Instance Role defines the roles and responsibilities of the person who holds that designation. You may now be wondering, Why is this understanding and the distinction between these two important. The answer lies in how AWS services create these; or expects us to create these. If you have always used the AWS console to create the Instance Role, then there is a chance you may not know what an Instance Profile is, as the console automatically creates the Instance Profile too and gives it the same name as the name you used for the Instance Role. Now when you use the console to create and launch the Amazon EC2 attaching the role you...

Amazon Redshift and the “Low/Zero-Code - Low/Zero-ETL” narrative

Image
Amazon Redshift is a column-oriented, fully managed, petabyte-scale data warehouse that makes it easy and cost-effective to analyze all your data. It achieves efficient query performance through a combination of massively parallel processing, columnar data storage, data compression, and ML powered system optimizations.  In this blog let's look at how Amazon Redshift helps the “Low/Zero Code – Low/Zero-ETL” narrative.  Before we get started, let's first understand what ETL means and why we do ETL.  ETL stands for Extract, Transform and Load, and it means exactly what each of these terms sounds like. We Extract data from a source, Transform the data based on the need and Load it to a target. For example - You can use a Cobol program to Extract data from a GDG, Transform the data in a series of Cobol code and Load it into a DB2 table. Another example can be where you use an ETL tool such as Informatica or Talend to extract data from a table or a file, Transform the data and ...

5G: What does it mean to me?

Image
A lot of different things come to mind when I think of 5G. The capabilities it brings forth in terms of enhanced bandwidth, lower latency, higher speeds, higher reliability; the different 5G enabled services that Cloud providers have launched; and even the recent news about how air traffic can be affected by 5G, owing to the capabilities of 5G to handle radio bands of much broader frequency range compared to 4G; etc. The capabilities of 5G further takes me to think about specific use cases: Think of a hospital with thousands of connected health monitoring 5G enabled devices. The higher speeds and lower latency can help transmit the information collected by these devices to a processing unit which can all help take faster decisions to save lives. Think of an industrial machine shop using connected 5G enabled sensors which can detect variations in temperature, alignment of a drill bit, or any other change that can affect the machining operation. 5G capabilities of these sensors can hel...

Implementing AWS Outposts to solve real world problems

Image
This blog is the second part in the series on AWS Outposts.  If you have not worked on AWS Outposts, I suggest read my previous blog where I have provided an overview of AWS Outposts.   As I have indicated in my previous blog, AWS Outposts is tailored made for scenarios such as local data storage, low latency processing and Hybrid Cloud use cases. In this blog let us look at how we can use AWS Outposts for solving some of these problems using 2 use cases.   Use Case 1 : Customer has a local data storage requirement i.e. No data at rest or no databases on Cloud.   Solution: Let us look at how we can setup a simple 3-tier architecture (consisting of a Presentation layer, an application layer and a data layer) using AWS Outposts and Cloud, to address this use case. Infrastructure setup - For the purpose of this blog, we will consider a simple AWS setup with 1 region and 1 AZ. Our VPC consists of 3 subnets, 2 private and 1 public. AWS Outposts sits on one of th...