If you have @SnowflakeDB Enterprise and you like lineage, I have good news for you. You can now easily send it to your favorite @MarquezProject or any other @OpenLineage compatible consumer. (Hint: check my bio)
Today starts the next stage of our evolution! I'm looking forward combining the best of data orchestration and data observability at @astronomerio, and helping grow the @ApacheAirflow and @OpenLineage communities.
Astronomer has acquired Datakin, the data lineage tool from founders of the @OpenLineage and @MarquezProject.
The move accelerates our shared goal of helping organizations build and manage reliable ecosystems that deliver trusted data. #blog
https://t.co/Ay1SsBrdgQ
This is a big milestone in the journey to bring true observability to the world of data. Thanks to all community members who have made this possible! #dataobservability#dataquality
Congrats to @OpenLineage, which has just become an @LFAIDataFdn sandbox project! Consistent collection of lineage metadata is hard, and requires a broad collaborative effort.
If you don’t know what you need to know when your pipeline is on fire, you are likely to get burned. See the difference #datalineage can make the next time you're on-call.
https://t.co/rAFBPAwxXI
We're excited to announce that the Marquez integrations for Airflow, Spark and DBT will be the official integrations for @OpenLineage! 🎉 To learn more about the high-level timeline of when the integration will be moved over, see: https://t.co/H8rgzmfJzD
Once we have data lineage, we can start to work with platforms that provide an operational degree of introspection into data logs, metrics and traces i.e. all the elements of data mechanics that give us a window into that thing we seek most in cloud - observability.
Me, 2021
@data_mesh_learn@montecarlodata No. A rigorous record is required, not notes. Provenance requirements. And we do document it. In code. And there is an execution plan with all of this information. All the information is in Spark. We just need to save it. Sounds like @OpenLineage and Marquez do exactly this.
There are a lot of similarities between the goals of @OpenTelemetry and @OpenLineage - this post (on the shiny new OpenLineage site!) explores them in detail
Data without context is just technical debt. That’s why lineage matters - if you haven’t spent much time thinking about how to track #datalineage, this podcast about @openlineage is a great place to start.
We are excited to launch @OpenLineage and work cross functionally with contributors of open source projects in the data ecosystem to enable the development of tools and solutions in terms of data operations, governance, and compliance.#DataOps#DataLineage
Are you looking for a "hello world" approach for #datagovernance?
Take a look at my latest article on available metadata solutions.
p.s. @MarquezProject is a great place to start!
https://t.co/cKM5xS36FK
You are human beings, I'm a human being; you want a happy life, and I want a happy life. But I think that aim will not be fulfilled only on the basis of material values; what we all want is affection and what we all need is to be more warm-hearted.