5 Data Engineering Projects for FREE (1 Million+ Views)📈
☁️ You will learn everything from AWS, GCP, and Azure by doing these projects
👨🏻💻 Kick-start your Career in Data Engineering by Doing these projects on weekends.
1. YouTube Data Analysis (End-To-End Data Engineering Project) - https://t.co/VoWVgzb614
What will you learn?
✅ Python and PySpark
✅ SQL
✅ How to understand the business problem
✅ AWS Services - Athena, Glue, Redshift, S3, IAM
✅ Building Data Pipeline and Scheduling it
2. Twitter Data Pipeline using Airflow - https://t.co/peLViDzonF
What will you learn?
✅ Python
✅ Basics of Airflow
✅ Working with Twitter Data and Package - Tweepy
✅ Python Package - Pandas
✅ Writing ETL job and storing data on S3
3. Stock Market Real-Time Data Analysis using Kafka, AWS, and Python - https://t.co/3H5rrBa4io
What will you learn?
✅ Build a Real-Time app using Python
✅ Understand the basics of Kafka
✅ Install Kafka on EC2
✅ Generate a real-time pipeline and
✅ Analyze Data in Real-Time
4. Uber Data Analytics Project On GCP
Video Link - https://t.co/H9jBKErD8i
Here's what you will learn:
✅ How to understand raw data
✅ Building Data Model (Lucid Chart)
✅ Writing ETL Script (Python)
✅ Modern Data Pipeline Tool (mage)
✅ SQL queries for analysis
5. Olympic Data Analytics | End-To-End Azure Data Engineering Project
Video Link - https://t.co/Y8bnAB5nae
Here's what you will learn:
✅ Extract Data from APIs
✅ Learn Azure Services DataBricks, DataFactory, and Synapse Analytics
✅ Writing Spark Code
✅ SQL queries for analysis
Tag someone who might find this helpful 👇🏻
IP Address in a nutshell:
1. What is an IP Address?
IP stands for Internet Protocol. An IP address is a unique numerical label assigned to each device on a computer network.
2. Types of IP Addresses:
- IPv4:
The most common version with addresses like 192.168.1.1.
- IPv6:
A newer version with longer addresses like 2001:0db8:85a3:0000:0000:8a2e:0370:7334.
3. IPv4 Address Format:
Consists of four sets of numbers separated by dots (e.g., 192.168.1.1).
Each set can range from 0 to 255 (8 bits).
4. IPv4 Classes:
- Class A:
Used for large networks, with the first octet reserved for network identification.
Range: 1.0.0.0 to 126.0.0.0.
- Class B:
Suitable for medium-sized networks. First two octets for network ID.
Range: 128.0.0.0 to 191.255.0.0.
- Class C:
Used for small networks, with the first three octets for network ID.
Range: 192.0.0.0 to 223.255.255.0.
- Class D: Reserved for multicast groups.
- Class E: Reserved for experimental purposes.
5. IPv6 Address Format:
Composed of eight groups of four hexadecimal digits separated by colons (e.g., 2001:0db8:85a3:0000:0000:8a2e:0370:7334).
6. IP Address Types:
- Public IP Address:
Identifies a device on the internet.
Usually provided by your Internet Service Provider (ISP).
- Private IP Address:
Used within a local network (e.g., home or office).
Defined by RFC 1918, with common ranges like 192.168.x.x, 10.x.x.x, and 172.16.x.x.
7. Subnetting:
- Subnet Mask:
A subnet mask defines the network and host portions of an IP address.
Common subnet masks include 255.255.255.0 for a Class C network.
8. CIDR Notation:
Classless Inter-Domain Routing.
Allows specifying network prefixes using a slash notation (e.g., 192.168.1.0/24).
9. Special IP Addresses:
Loopback Address:
127.0.0.1 for IPv4 (localhost).
::1 for IPv6.
10. Broadcast Address:
Used to send data to all devices on a local network (e.g., 192.168.1.255).
11. Default Gateway:
The device that connects your local network to the internet.
Typically ends with .1 (e.g., 192.168.1.1).
12. Reserved Addresses:
Some IP addresses are reserved for special purposes (e.g., 0.0.0.0, 255.255.255.255).
13. IP Address Assignment:
- DHCP (Dynamic Host Configuration Protocol):
Automatically assigns IP addresses to devices on a network.
- Static IP Address:
Manually configured and doesn't change unless you modify it.
- NAT (Network Address Translation):
Allows multiple devices in a local network to share a single public IP address.
14. IP Version Transition:
Transition from IPv4 to IPv6:
Due to IPv4 address exhaustion, IPv6 adoption is increasing.
IPv6 provides a vastly larger address space.
15. Common Private IPv4 Address Ranges:
Class A: 10.0.0.0 to 10.255.255.255
Class B: 172.16.0.0 to 172.31.255.255
Class C: 192.168.0.0 to 192.168.255.255
𝗗𝗲𝘃𝗢𝗽𝘀 𝗮𝘀 𝗮 𝗕𝘂𝗿𝗴𝗲𝗿 (𝗗𝗮𝗮𝗕)
What should the DevOps engineer roadmap look like:
𝟭. 𝗟𝗲𝗮𝗿𝗻 𝗮 𝗽𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 (Python, Go, ...) to write automation scripts
𝟮. 𝗠𝗮𝘀𝘁𝗲𝗿 𝗼𝗻𝗲 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 ����𝘆𝘀𝘁𝗲𝗺 (Linux) and its command-line interface (CLI)
𝟯. 𝗠𝗮𝘀𝘁𝗲𝗿 𝘀𝗲𝗿𝘃𝗲𝗿 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁, web servers, including proxies such as Nginx or IIS
𝟰. 𝗘𝘅𝗽𝗹𝗼𝗿𝗲 𝗰𝗼𝗻𝘁𝗮𝗶𝗻𝗲𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘂𝘀𝗶𝗻𝗴 𝗗𝗼𝗰𝗸𝗲𝗿
𝟱. 𝗖𝗵𝗲𝗰𝗸 𝗰𝗼𝗻𝘁𝗮𝗶𝗻𝗲𝗿 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀
𝟲. 𝗨𝗻𝗹𝗼𝗰𝗸 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗮𝘀 𝗖𝗼𝗱𝗲 (IaC) using tools like Terraform, Ansible, Chef, or Puppet for provisioning and configuration management
𝟳. 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝗻𝗲𝘁𝘄𝗼𝗿𝗸 𝗽𝗿𝗼𝘁𝗼𝗰𝗼𝗹𝘀: DNS, IP addresses, ports, and the OSI model
𝟴. 𝗘𝗺𝗯𝗿𝗮𝗰𝗲 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻/𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁 (𝗖𝗜/𝗖𝗗) 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 for automating app delivery and deployment stages
𝟵. 𝗠𝗮𝘀𝘁𝗲𝗿 𝗺𝗼����𝗶𝘁𝗼𝗿𝗶𝗻𝗴 𝘁𝗲𝗰𝗵𝗻𝗶𝗾𝘂𝗲𝘀 for real-time oversight of applications, services, and infrastructure
𝟭𝟬. 𝗚𝗮𝗶𝗻 𝗵𝗮𝗻𝗱𝘀-𝗼𝗻 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗖𝗹𝗼𝘂𝗱 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗿𝘀 like AWS and Azure.
To learn more about it, check out my complete DevOps roadmap for 2023 in the comments.
_______
If you like my posts, please follow me, @milan_milanovic, and hit the 🔔 on my profile to get a notification for all my new posts.
Grow with me 🚀!
#technology #softwareengineering #programming #techworldwithmilan #devops