Learning the fundamentals of DevOps principles and processes may seem daunting or feel like a huge mountain to climb. I personally don’t think it should or needs to be that way. https://t.co/BuPfGemTzb the community interactions around this repo is mind blowing! #90DaysOfDevOps
Today we are going to be build an assistant. Everything I need bubbled into one place.
Written in golang
Runs local
Needs to hit Outlook, Jira, Confluence, newsletters, notion, some forums
@_ediri@gopherconuk I have found this event the last few years. The sessions have been really informative and interactive. Something you can’t get from the hallway (although always the best track)
For me it’s also educational and great to ask my beginner noob questions
@gopherconuk do you have an expected date for the agenda?
I want to attend but also want to make sure the sessions are useful for me before I purchase a ticket.
Welcome to the era of agentic data engineering.
Today at the Data + AI Summit, we are excited to announce the next major evolution of Databricks Lakeflow:
- 𝗟𝗮𝗸𝗲𝗳𝗹𝗼𝘄 𝗖𝗼𝗻𝗻𝗲𝗰𝘁: Connect to 100+ enterprise data sources across SaaS, databases, cloud storage, and more.
- 𝗭𝗲𝗿𝗼𝗯𝘂𝘀 𝗜𝗻𝗴𝗲𝘀𝘁: Stream high-volume event data via multiple interfaces, including OTEL and Kafka-compatible APIs.
- 𝗦𝗽𝗮𝗿𝗸 𝗗𝗲𝗰𝗹𝗮𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀: Batch and streaming, SQL and Python, and now real-time streaming with e2e latencies as low as 5ms.
- 𝗚𝗲𝗻𝗶𝗲 𝗖𝗼𝗱𝗲 & 𝗚𝗲𝗻𝗶𝗲 𝗭𝗲𝗿𝗼𝗢𝗽𝘀: Accelerate authoring and reduce operational toil with agentic development.
- 𝗟𝗮𝗸𝗲𝗳𝗹𝗼𝘄 𝗝𝗼𝗯𝘀: Orchestrate complex data pipelines and AI agents seamlessly with 50+ integrations.
https://t.co/tX3vhxDqzG
I am not against under 16s not being allowed on social media.
I am cautious and interested how a government imposes the identity checks.
It must be levels of airport and bank security which then means it’s in the hands of the Metas and Google?
Seems fishy to me?
@GaryWilliamsIT@TheAutisticTech I mean the government and skills not the kids. I am fully aware they can learn and will be steps ahead of whomever the government pays billions to try and implement something.
Awesome keynote by @alighodsi at the @databricks summit.
Some takeaways below. TLDR - Databricks is now a data processing platform, a data platform, an agents platform and an apps platform
Architecture updates
Iceberg / table formats
- v3 is now GA: the unified data layer. Delta & Iceberg files are laid out identically on disk, no rewriting to share across formats
- v4 (targeting Q4 '26) finishes the job by unifying metadata. After that, Delta and Iceberg are effectively one format end-to-end
Lakeflow ingestion (3 modalities, all GA):
- ZeroBus: kills the need for Kafka. Hit one API endpoint with bursty/tiny-row data at high rate, no buffering, no millions of small files
- Spark Real-Time Mode: sub-10ms latency in open-source Spark, closing the gap with Flink (Spark's old micro-batch floor was ~1s)
- Lakeflow Designer: Alteryx-style drag-and-drop; talk to Genie, it generates inspectable, version-controllable Spark under the hood
- 100+ connectors now (Salesforce, Workday, NetSuite, Meta, Google Analytics, and more)
Lakebase + LTAP:
- Lakebase: open-source Postgres on the lake. Serverless autoscaling to zero, plus branching: clone a petabyte DB in <1s via copy-on-write (agents love it)
- LTAP: unifies the transactional (Lakebase) and analytical (Lakehouse) layers, a breakthrough the industry has chased for 40 years
- Reyden: new query engine hitting tens-of-ms latency
Unity AI Gateway: one pane of glass for all AI
- Single entry point for every agent, harness & model
- Commit spend to Databricks, buy tokens directly from OpenAI / Anthropic / Gemini on any cloud
- Budgets + alerts down to the individual level
- Guardrails, auditing, identity management for every agent
- Register any MCP server, authenticate once
- Free + open source (part of Unity Catalog + MLflow)
Genie: The Agents Platform, powered by Ontology
Instead of agents for-looping through your data live (slow, expensive, inaccurate), Genie Ontology runs in the background and builds a knowledge graph of your most important assets across lakehouse + Drive + SharePoint + email + many other sources. It ranks importance using "OntoRank" - basically PageRank for enterprise assets. Databricks' own instance has 4.5M ontology snippets. That context feeds four agents:
Genie One: universal interface for any business user to ask questions across all data
Genie Agents: turn any Genie conversation into a deployable autonomous agent
Genie Code: coding agent that's elite at data engineering + ML/data science
Genie Zero Ops: monitors pipelines, fixes a 2am break in an isolated branch, pings you for one-click approval
New vertical apps
Lakewatch: agentic SIEM / "security lakehouse" (also acquiring Panther Labs)
Customer Lake: agentic CDP with LLM-powered identity dedupe + one-to-one "infinity campaigns"
@marius402 Why is everyone going on about Bluesky?
I would be more worried about the next social media for kids that is hidden away and easily accessible.