Streamdal

Streamdal is an open-source platform for managing code-native data pipelines | Join us 🤝 🌟

[Streamdal @streamdal]

PII Protection Upgrade! 🛡️

We’re thrilled to unveil the latest feature for the Streamdal pipeline engine:

✨ Automatic Detection and Masking of PII ✨

Much less manual hassle! With a few clicks, safeguard sensitive info like email addresses or phone numbers automatically in your data pipelines. 🚀

🔐 Streamlined Security: Effortlessly obscure PII data to and from external systems.

🔧 Simplified Workflows: Greatly simplify custom steps; this feature detects PII dynamically in payloads!

🎯 Precision Matters: Tested rigorously to detect and mask PII, even in complex JSON structures.

📹 Curious? @jacobheric got a quick video walkthrough for you!
#DataSecurity #PII

[Prisma @prisma]

💡 Ecosystem highlight of the week: prisma-extension-streamdal

@streamdal is an open-source tool to embed privacy controls into app code. With this Prisma Client extension, you can code native data pipelines for Prisma Client queries.

👉 Check it out at https://t.co/T2wh14r8PO

[Streamdal @streamdal]

@prisma You can also check out a video walkthrough 🔗:
https://t.co/hfexVGNOCY

[Streamdal @streamdal]

@prisma This is awesome, thank you @prisma team for sharing 💜

[Streamdal @streamdal]

🔗: https://t.co/a0hC5Iv9Wd

[Streamdal @streamdal]

💡Streamdal is all about automating mundane and annoying tasks…

That's why we're excited to introduce our newest enhancement 🎉: a Terraform provider used for stamping down pipelines and various data rules - no more manual clicking around in a web UI!

👀Check out our blog post by our Lead Software Engineer, Mark Gregan (@marqueetag) for a step-by-step guide on getting your pipeline setup with our Terraform provider (the link is in the following thread) 🔗👇
#OpenSource #Terraform #CodeNative #PII #DataPrivacy

[Streamdal @streamdal]

🔍 Curious about how Streamdal's mission to democratize real-time data pipelines connects with our latest update?

🌐 Dive into our blog post to uncover the magic! ✨

🌟 Discover how our open-source shim for KafkaJS is revolutionizing data pipelines, empowering any application to seamlessly integrate sophisticated functionalities with just a few configuration tweaks 🚀💻

💡 Ready to streamline your data operations and unlock new possibilities? Check the links in the following thread 🔗 👇

#OpenSource #Kafka #JavaSkript #DataEngineering #DataQuality

[Streamdal @streamdal]

Your favorite libraries, but now with Streamdal!

We are on a roll with these, and plan to continue to pump these out regularly:

🌟streamdal/ibm-sarama
🌟streamdal/kafkajs
🌟streamdal/segmentio-kafka-go
🌟streamdal/confluent-kafka-python
🌟streamdal/confluent-kafka-go

[Streamdal @streamdal]

🔍 Curious about how Streamdal's mission to democratize real-time data pipelines connects with our latest update?

🌐 Dive into our blog post to uncover the magic! ✨

🌟 Discover how our open-source shim for KafkaJS is revolutionizing data pipelines, empowering any application to seamlessly integrate sophisticated functionalities with just a few configuration tweaks 🚀💻

💡 Ready to streamline your data operations and unlock new possibilities? Check the links in the following thread 🔗 👇

#OpenSource #Kafka #JavaSkript #DataEngineering #DataQuality

[Streamdal @streamdal]

Blog post 🔗: https://t.co/1oWFQK4Pql
YT video walkthrough 🔗: https://t.co/njxYjZsdsH

[Streamdal @streamdal]

🔗: https://t.co/jmLcxqK9JQ

[Streamdal @streamdal]

🔮 Imagine a dev world where data privacy and compliance changes are made immediately and directly in your app, development is empowered, and costs are slashed.

👀Check out the future of data privacy - our CTO Dan Selans (@skey) breaks down the code-native concept

Find the link in the thread to get an idea about this modern approach to dealing with data privacy-related issues 🔗 👇
#DataPrivacy #SoftwareDevelopment #SoftwareEngineering

[Streamdal @streamdal]

🔐Want to ensure data is protected in dev?

Our Director of Engineering, @fritzstauff shows you how to get powerful data workflows up and running quickly in your development environments 🚀

Check it out 👇
#Development #TechTutorial #DataPrivacy #PII #CodeNative

[Streamdal @streamdal]

🔗 https://t.co/kal5xbreGa

[Streamdal @streamdal]

🛡️Tired of playing catch-up with data security drama?

Streamdal CTO, Dan Selans (@skey), lays down the facts on Upstream Data Security. Learn what it’s all about and why you should care about it.

Dive into the insights 🔗👇
#DataPrivacy #PII #DataEngineering #Governance

[Streamdal @streamdal]

🚨 Video Walkthrough Alert

Join our lead engineer, @jacobheric, as he takes you through the newest feature on the Streamdal Code-Native platform: HTTP Requests.

Explore this exciting addition 👇
#CodeNative

[Streamdal @streamdal]

There are also some great libraries and shims we’ve created to help get started with Streamdal:
Kafkajs: https://t.co/JUhF4yUfk8
Prisma-Streamdal: https://t.co/iDsggAwQDP

[Streamdal @streamdal]

Ready for full Node mode? Whether you need to observe, operate on data, or are looking for something clean and fresh for your project, check out our Node SDK.

Similar to our other SDKs, it’s open-source and used to enable data observability and code-native data pipelines. You’ll find this tech in it:

✅Protobuf - Typescript/Javascript gRPC and protocol buffers for Node.
✅Vitest (@vitest_dev) & Sinon - Great for testing!

And of course, a lot more 🔗👇
#FOSS #FossFeb #opensource #Node #protobuf

[Streamdal @streamdal]

Check out the Node SDK on Github: https://t.co/5w5v5c4BUw

Other useful links 👇
Protobuf: https://t.co/u7ZRa2Bxv4
Sinon: https://t.co/dKEjBl0pBr
Vitest: https://t.co/fUa4EheYOT

[Streamdal @streamdal]

Thrilled to share a recent endeavor – a comprehensive video walkthrough showcasing the ease of extracting, truncating, and adding metadata with Streamdal 🎥✨

This powerful technique allows you to easily filter out messages or data that may not be intended for downstream endpoints such as Logstash, Snowflake, S3, Postgres, and more.

Check it out! 🚀
#DataEngineering #DevOps #DataManagement #Engineering

[Streamdal @streamdal]

@senordosplatos @golang 🙌🙌🙌

[Streamdal @streamdal]

More @golang inspiration for those developers young and old, learned or aspiring!
To visualize read/write, observe data, and sync rules for workflows and pipelines, we needed a way to communicate with the frontend.

We built that server functionality in Go.

Like all of our components, the server is open-source. Some particulars you’ll see:

✅gRPC (@grpcio) + Protobuf - We use both unary and streaming functionality for communication.
✅Redis (@Redisinc) - We specifically use redis for Pubsub and its traditional key/value functionality.
✅Go-director - This library enables us to declare for loops as injectable dependencies, which greatly simplifies testing.

The code 🔗👇
#FOSS #FossFeb #opensource #go #redis