THESE TRUMPS RECIPROCAL TARIFFS ARE NUTS 😳
49% on Cambodia
46% on Vietnam
36% on Thailand
34% on China
32% on Taiwan
32% on Indonesia
31% on Switzerland
26% on India
25% on South Korea
24% on Japan
20% on EU
What is DevSecOps?
DevSecOps emerged as a natural evolution of DevOps practices with a focus on integrating security into the software development and deployment process. The term "DevSecOps" represents the convergence of Development (Dev), Security (Sec), and Operations (Ops) practices, emphasizing the importance of security throughout the software development lifecycle.
The need for DevSecOps arose from the recognition that traditional security approaches, which often involve late-stage security testing or manual security reviews, were insufficient to address the growing complexity and pace of modern software development. By integrating security practices early and continuously throughout the development lifecycle, DevSecOps aims to enhance the security posture of software applications while maintaining agility and innovation.
The diagram below shows the important concepts in DevSecOps.
1 . Automated Security Checks
Use tools to automate security checks and scans. These include Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST) and Dependency Scanning.
2 . Continuous Monitoring
Ensure real-time monitoring of applications to detect and respond to threats. This includes monitoring system logs, user activity and network traffic for any suspicious activity.
3 . CI/CD Automation
Continuous Integration and Continuous Deployment (CI/CD) pipelines ensure that code changes are automatically tested, built, and deployed before deployment. Integrating security checks into these pipelines ensures that vulnerabilities are detected and resolved before deployment.
4 . Infrastructure as Code (IaC)
Use code and automation to manage and configure infrastructure. Tools like Terraform and Ansible can be used for this, ensuring that security best practices are followed in these scripts.
5 . Container Security
As containerization becomes more prevalent, it is critical to ensure container image and runtime security. This includes scanning container images for vulnerabilities and ensuring runtime security.
6 . Secret Management
Ensure that sensitive data like API keys, passwords, and certificates are securely stored and managed. Tools like HashiCorp Vault can help securely manage and access secrets.
7 . Threat Modeling
Regularly assess and model potential threats to your application. This proactive approach helps to understand potential attack vectors and mitigate them.
8. Quality Assurance (QA) Integration
Embed quality checks and tests throughout the development cycle, not just in the post-development phase.
9 . Collaboration and Communication
Facilitates effective communication and collaboration between development, operations and security teams.
10 . Vulnerability Management
Go beyond scanning to systematically manage, prioritize, and remediate discovered vulnerabilities.
What are the most common 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀 𝗳𝗼𝗿 𝗞𝗮𝗳𝗸𝗮?
We have covered lots of concepts around Kafka already. But what are the most common use cases for The System that you are very likely to run into as a Data Engineer?
𝗟𝗲𝘁’𝘀 𝘁𝗮𝗸𝗲 𝗮 𝗰𝗹𝗼𝘀𝗲𝗿 𝗹𝗼𝗼𝗸:
𝗪𝗲𝗯𝘀𝗶𝘁𝗲 𝗔𝗰𝘁𝗶𝘃𝗶𝘁𝘆 𝗧𝗿𝗮𝗰𝗸𝗶𝗻𝗴.
➡️ The Original use case for Kafka by LinkedIn.
➡️ Events happening in the website like page views, conversions etc. are sent via a Gateway and piped to Kafka Topics.
➡️ These events are forwarded to the downstream Analytical systems or processed in Real Time.
➡️ Kafka is used as an initial buffer as the Data amounts are usually big and Kafka guarantees no message loss due to its replication mechanisms.
𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗥𝗲𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻.
➡️ Database Commit log is piped to a Kafka topic.
➡️ The committed messages are executed against a new Database in the same order.
➡️ Database replica is created.
𝗟𝗼𝗴/𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻.
➡️ Kafka is used for centralized Log and Metrics collection.
➡️ Daemons like FluentD are deployed in servers or containers together with the Applications to be monitored.
➡️ Applications send their Logs/Metrics to the Daemons.
➡️ The Daemons pipe Logs/Metrics to a Kafka Topic.
➡️ Logs/Metrics are delivered downstream to storages like ElasticSearch or InfluxDB for Log/Metrics discovery respectively.
➡️ This is also how you would track your IoT Fleets.
𝗦𝘁𝗿𝗲𝗮𝗺 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴.
➡️ This is usually coupled with ingestion mechanisms already covered.
➡️ Instead of piping Data to a certain storage downstream we mount a Stream Processing Framework on top of Kafka Topics.
➡️ The Data is filtered, enriched and then piped to the downstream systems to be further used according to the use case.
➡️ This is also where one would be running Machine Learning Models embedded into a Stream Processing Application.
𝗠𝗲𝘀𝘀𝗮𝗴𝗶𝗻𝗴.
➡️ Kafka can be used as a replacement for more traditional messaging brokers like RabbitMQ.
➡️ Kafka has better durability guarantees and is easier to configure for several separate Consumer Groups to consume from the same Topic.
❗️Having said this - always consider the complexity you are bringing with introduction of a Distributed System. Sometimes it is better to just use traditional frameworks.
--------
Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.
Also hit 🔔to stay notified about new content.
𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 👍, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁!
Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿.