Exploring Databricks Lakeflow SDP & AUTO CDC INTO for implementing SCD Type 2 with full history tracking. Modern incremental CDC pipelines without complex MERGE logic. #Databricks#Lakeflow#CDC#DataEngineering#data https://t.co/NJiaNhzyrF
Databricks Delta Sharing enables secure cross-org data collaboration across clouds & BI tools. Important: consumers get READ-ONLY access, no DML/DDL operations. #Databricks#DeltaSharing#data https://t.co/Er7Fi2pEr6
AWS IDENTITY AND ACCESS MANAGEMENT (IAM)
INTRODUCTION
AWS Identity and Access Management (IAM) is a security service that helps you securely control access to AWS resources. IAM allows you to define who can access resources, what actions they can perform, and under which conditions access is granted.
WHAT IS IAM?
โ IAM is a global AWS service used for authentication and authorization
โ Helps manage users, permissions, and secure access to AWS services
โ Enables fine-grained access control across the AWS environment
CORE COMPONENTS OF IAM
IAM USERS
โ Represents a person or application needing AWS access
โ Each user can have unique credentials
โ Supports passwords and access keys
IAM GROUPS
โ Collection of IAM users
โ Simplifies permission management
โ Permissions assigned to a group apply to all users inside it
IAM ROLES
โ Temporary access identities with specific permissions
โ Used by AWS services, applications, and external users
โ No permanent credentials attached
IAM POLICIES
โ JSON documents defining permissions
โ Specify what actions are allowed or denied
โ Can be attached to users, groups, or roles
HOW IAM WORKS
โ User/Application sends request to AWS
โ AWS authenticates identity credentials
โ IAM policies are evaluated
โ Access is either Allowed or Denied
โ Requested AWS resource responds accordingly
TYPES OF IAM POLICIES
MANAGED POLICIES
โ Predefined or reusable policies
โ Can be attached to multiple identities
INLINE POLICIES
โ Directly embedded into a single user, group, or role
โ Tight one-to-one relationship
AWS MANAGED POLICIES
โ Created and maintained by AWS
โ Covers common use cases
CUSTOMER MANAGED POLICIES
โ Created and managed by the organization
โ Provides custom access control
IAM SECURITY FEATURES
MULTI-FACTOR AUTHENTICATION (MFA)
โ Adds extra security layer beyond passwords
โ Requires a second verification factor
PASSWORD POLICIES
โ Enforces strong password requirements
โ Controls expiration and complexity rules
ACCESS ANALYZER
โ Identifies resources shared externally
โ Helps maintain least privilege access
IAM ACCESS KEYS
โ Used for programmatic access via CLI or SDKs
โ Should be rotated regularly for security
BEST PRACTICES FOR IAM
โ Follow the Principle of Least Privilege
โ Enable MFA for all users
โ Avoid using the root account for daily tasks
โ Rotate credentials regularly
โ Use IAM Roles instead of sharing access keys
โ Audit permissions frequently
REAL-WORLD IAM ARCHITECTURE FLOW
โ Developer โ IAM User โ IAM Group โ Policies โ AWS Services
โ EC2 Instance โ IAM Role โ Temporary Credentials โ S3 Access
โ Application โ STS Token โ Limited-Time Permissions
COMMON IAM USE CASES
DEVELOPER ACCESS
โ Grant developers controlled access to AWS resources
APPLICATION AUTHENTICATION
โ Allow applications to securely interact with AWS services
CROSS-ACCOUNT ACCESS
โ Share access between AWS accounts securely using IAM Roles
TEMPORARY ACCESS MANAGEMENT
โ Provide short-term credentials for secure operations
WHY IAM IS IMPORTANT
โ Protects AWS resources from unauthorized access
โ Simplifies permission management
โ Enhances cloud security posture
โ Supports compliance and governance requirements
TIP
AWS IAM is a foundational security service that enables secure and scalable access management across AWS environments. By properly configuring users, groups, roles, and policies, organizations can maintain strong security while ensuring efficient resource access.
Grab the AWS EBOOK
https://t.co/G3pdEOq4DK
2000: Data modeling = ER diagrams + 3NF
2010: + Star schemas + Dimensions
2015: + Big Data + Schema-on-read + NoSQL
2020: + Data Vault + Modern Data Stack
2026: + Iceberg + OBT + Data Contracts + LLM-aware schemas + ...
โ Today's "design a schema" interview
Most candidates still jump to a star schema.
The bar didn't just rise. It multiplied.
Here's what "data modelling" actually means in 2026 ๐
๐ฆ ๐ง๐ต๐ฒ ๐ฏ ๐น๐ฎ๐๐ฒ๐ฟ๐
โ Conceptual (entities + relationships)
โ Logical (tables, columns, keys, normalization)
โ Physical (indexes, partitions, storage strategy)
๐ฉ ๐ง๐ต๐ฒ ๐ฑ ๐ฝ๐ฎ๐ฟ๐ฎ๐ฑ๐ถ๐ด๐บ๐
โ 3NF / OLTP (transactional databases)
โ Dimensional / Kimball (analytics warehouses)
โ Data Vault (audit-heavy enterprise)
โ One Big Table (modern columnar)
โ Document / NoSQL (flexible schemas)
๐ง ๐ง๐ต๐ฒ ๐ถ๐ป๐๐ฒ๐ฟ๐๐ถ๐ฒ๐ ๐๐ถ๐ด๐ป๐ฎ๐น
When the interviewer says "design a schema":
โ Junior: jumps to a star schema
โ Senior: asks WHICH paradigm fits the use case
That ONE question separates levels.
Most candidates lose senior offers not because they can't draw a star schema.
They lose because they don't realize how much "data modeling" has expanded.
I'm dropping a complete Data Modeling Masterclass on May 16.
Inside:
โ All 5 paradigms compared (with code examples)
โ Worked example with Uber
โ Physical modeling (partitioning, indexing, costs)
โ Anti-patterns that fail interviews
โ Leveled expectations: junior / senior / staff+
FREE.
Stay tuned.
Data engineers, what's the first paradigm you think of when you hear "data modelling"? ๐
๐จ In 1993, Steve Jobs literally predicted the future of technology decades before it happened.
Most people still havenโt seen this.
Long before the iPhone, modern Internet, or AI boomโฆ
He was already describing it.
Watching it today feels unreal.
He talked about computers becoming personal companions not just tools, but extensions of how we think and live. Devices you carry, systems that understand you, and technology that feels almost human.
He imagined a world where everything is connected, information flows instantly, and software adapts to people not the other way around.
And his biggest insight? The future isnโt about machines.
Itโs about people.
Thatโs why this still hits hard.
Because while most people wait for the futureโฆ
A few can see it coming years before it arrives.
@dhanushkraja and gv sir weโve created a song with the help of AI tools and would truly value your time if you could give it a listen. Your feedback would mean a lot to us. @gvprakash https://t.co/7jrVMhiuBT
3 ways to scale your vector database ๐ง โ๏ธ
When your embeddings start to outgrow a single node, scaling isnโt optional itโs survival. Hereโs how to do it right:
๐น Sharding โ Split your data across multiple servers to scale storage and speed up parallel imports.
๐น Replication โ Create redundant copies to handle high query loads and ensure zero downtime.
๐น Vertical Scaling โ Add more CPU and memory for faster queries (but remember, hardware caps out eventually).
Scaling smartly isnโt just about performance itโs about reliability at scale.
๐ง ๐ธ How to Build Multimodal AI Agents from Scratch โ No Prior Experience Needed.
๐ง๐ต๐ถ๐ ๐ถ๐ ๐ฎ ๐ต-๐ฆ๐๐ฒ๐ฝ ๐๐๐ถ๐ฑ๐ฒ ๐ณ๐ฟ๐ผ๐บ ๐ฆ๐ฐ๐ฟ๐ฒ๐ฒ๐ป๐๐ต๐ผ๐ ๐๐ผ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด.
STEP 0: WHAT IS AN AI AGENT?
โน A system that uses an LLM to reason, plan, act, and adapt
โน Goes beyond prompts & RAG โ can use tools + learn from memory
โน Use when workflows are complex, multi-step, or need personalization
ใAgents โ demos. Theyโre decision-makers.
STEP 1: DEFINE THE AGENTโS ROLE AND GOAL
โน What will your agent do?
โน Who is it helping?
โน What kind of output will it generate?
ใExample: An agent that answers questions and explains graphs from reports
STEP 2: PREPARE THE MULTIMODAL CORPUS
โน Convert each page of your PDFs into screenshots
โน Preserve text + image layout together
โน Store image paths as metadata
ใWhy? So retrieval captures full document context
STEP 3: GENERATE UNIFIED EMBEDDINGS
โน Use Vision-Language Models (VLMs)
โน Avoid the โmodality gapโ of CLIP-style models
โน One encoder for both text + visuals
ใTools: YJI-VLM, Gemini 2, Claude Opus
STEP 4: STORE IN VECTOR DATABASE
โน Embed screenshots and attach metadata
โน Store in a retrievable format
โน Make retrieval page-level, not just paragraph-level
ใTools: MongoDB Atlas, Weaviate, ChromaDB
STEP 5: RETRIEVE CONTEXT VIA TOOL USE
โน Agent uses vector DB to fetch relevant pages
โน Returns image references, not raw text chunks
โน Agent fetches actual images based on those references
ใThink: tool = bridge to your multimodal memory
STEP 6: ADD MEMORY (SHORT-TERM)
โน Use session IDs to track conversations
โน Load past Q&A into LLM at each turn
โน Let agent reason over previous steps
ใLong-term memory optional โ short-term is a must
STEP 7: PLUG IN A MULTIMODAL LLM
โน Must support both images + text
โน Inputs = query + memory + retrieved images
โน Generates step-by-step plans or final answers
ใTools: Gemini 2.0, GPT-4o, Claude 3.5
STEP 8: DESIGN PLANNING AND FEEDBACK LOOPS
โน Chain-of-Thought for stepwise reasoning
โน ReAct for dynamic reasoning + tool calls
โน Prompt the LLM to decide: act or answer?
ใThis is where real agentic behavior kicks in
STEP 9: WRAP IT IN AN APP OR API (OPTIONAL)
โน Let users upload reports, images, or ask questions
โน Build a UI or expose your agent via API
โน This turns your agent into a product
ใTools: Gradio, Streamlit, FastAPI
https://t.co/zy8mvcqSrv
โฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃ
โซธ๊ Want to Build Real-World AI Agents?
Join My ๐๐ฎ๐ป๐ฑ๐-๐ผ๐ป ๐๐ ๐๐ด๐ฒ๐ป๐ ๐ฑ-๐ถ๐ป-๐ญ ๐ง๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด,
trusted by 1,500+ builders worldwide!
โ Build Geo, Audio, Video & Vision Agents
โ Master 5 Modules: ๐ ๐๐ฃ ยท LangGraph ยท PydanticAI ยท CrewAI ยท OpenAI Swarm
โ Deploy for Healthcare, Finance, Smart Cities & More
โ 9 Real-World Projects with Full Code
Only basic Python required.
๐ ๐๐ป๐ฟ๐ผ๐น๐น ๐ก๐ข๐ช (๐ฑ๐ฒ% ๐ข๐๐): https://t.co/5i2v1fIrhJ
New release from @PacktDataML >> Hands-On #AI for #IoT : Expert #MachineLearning and #DeepLearning techniques for developing smarter IoT systems [2nd Edition]
See it at https://t.co/36PCOv9cLc
๐๐ฎ๐ ๐๐ฎ๐ช๐ฝ๐พ๐ป๐ฎ๐ผ:
๐ดLeverage the power of Python libraries such as TensorFlow and Keras to work with real-time IoT data
๐ตEnhance your IoT solutions with advanced AI techniques, including deep learning, optimization, and generative adversarial networks
๐ขGain practical insights through industry-specific IoT case studies in manufacturing, smart cities, and automation
๐ดPurchase of the print or Kindle book includes a free PDF eBook