Rodrigo Santana

CTO | Estrategista Tecnológico | Liderança em Dados e Inteligência Artificial

[Zach Wilson @Zachly]

Someone from private equity offered me $10m for https://t.co/krtu8inEv2 today

Crazy to think I could just be done working forever today

[Santiago @svpino]

There are four stages in a RAG pipeline (very similar to an ETL system):

1. Ingestion: Where the pipeline loads the information from the data source.

2. Extraction: Where the pipeline processes the input data and decides how to retrieve the text contained inside them.

3. Transform: Where the pipeline chunks the data and generates document embeddings.

4. Load: Where the pipeline creates a search index in a vector database and loads the document embeddings.

But doing these with unstructured data is not simple:

• You must figure out how to refresh vector databases whenever the original data sources change.

• You must figure out how to extract content from complex documents containing tables, images, or cross-references.

• You must figure out the optimal strategy for chunking your data.

A few weeks ago, I recorded a video partnering with Vectorize. I built a simple RAG pipeline that solves the problems above.

They just went live on Product Hunt:

https://t.co/GQEs35hsTW

For those of you interested in building RAG applications, it's very easy to set up a free account and try it out.

[Santiago @svpino]

Data pipelines will put you in the top 1% of the market.

If you could only learn one skill for the next decade, I can't think of anything more critical than learning to move and process data at scale.

I like to tell people I'm a Machine Learning Engineer, but in reality, 90% of the value I produce comes from my ability to move data around consistently and correctly.

In the field, we like to use the term "orchestration" when talking about coordinating workflows that move and process data. At a high level, there are three main steps you need to worry about:

1. Getting the data from its source
2. Processing and cleaning that data
3. Delivering the cleaned data to the right place

You might have also heard about "ETL" (Extract, Transform, Load). That's how most people refer to the process above.

Of course, building a simple ETL system isn't complex; most developers can do it without too much trouble. The problem is designing resilient, scalable, and fault-tolerant systems.

You can't code your way to a production-ready orchestration platform (ask me how I know.) I started with AirFlow and eventually moved to @kestra_io because of its event-driven architecture.

Event-driven means you can kick off a workflow automatically based on different triggers. For instance, when somebody uploads a new file to a folder, an app updates a database table, or there's a new message in a queue.

It's hard to summarize everything you get from Kestra, but here are some of the highlights:

• Kestra is free and open-source
• You install it from a Docker container
• Workflows as Code using YAML <--- this is awesome
• Scales to millions of executions
• It integrates with every cloud platform you've seen
• Language agnostic (but I still like Python the most)

Here is a link to their GitHub repository:

https://t.co/h8mWoYcymb

Here are the three things I recommend:

1 - Take a look at their live demo in their GitHub repo
2 - Build a simple workflow (it will take 5 minutes)
3 - Talk to your boss. Where can you plug this into your company?

I started using Kestra at the height of the pandemic. It's an awesome tool, and I'm proud that they are sponsoring my writing. I hope you find it helpful as well.

[Mage @mage_ai]

🎉 We've hit another MAGER milestone in our slack community, reaching 5K members! 🚀

🙏 Thank you to our incredible community for being a part of this magical journey. Here's to more collaboration, learning, and growth together! 🧙‍♂️

#mage #mageai #community #dataengineering

[OSO @osodevops]

Deploying and Managing an NGINX Application on Google Kubernetes Engine (GKE) using Terraform… https://t.co/2howWKnLjm #kubernetes

[Ajay Yadav @ATechAjay]

OpenAI just launched a big upgrade. 🥳

ChatGPT is now 10x easier to use. 💹

Check out 7 new features of ChatGPT you shouldn't miss.

[Rodrigo Santana @rodrigofs10]

O maior problema da arquitetura #Lakehouse é o Spark...

Pensa bem, enquanto existir essa dependência ainda será "pouca" utilizada..

Muitos pipelines não usam Spark.

Spark é excelente, muito mesmo, porém nem todos precisam dele.

Faz sentido?

[Naval @naval]

If you can't code, write books and blogs, record videos and podcasts.

[Rodrigo Santana @rodrigofs10]

No último fim de semana eu e o Felipe Santana ministramos para mais uma turma o Bootcamp em Data Science, dessa vez em Campos dos Goytacazes - RJ.

Nessa turma tivemos pessoas de diversas áreas: Doutor em Engenharia de Produção, Doutor em Biologia, Econo…https://t.co/gICqkQgLqU

[Rodrigo Santana @rodrigofs10]

Olá pessoal, lançamos o Data Science do Zero, esse é novo projeto do blog minerando dados. #datascience #machinelearning #ia #python 😙https://t.co/zTqZnbxG6T

[Rodrigo Santana @rodrigofs10]

Named Entity Recognition: Aprenda como implementar usando Python e o Framework de NLP Spacy. =&gt; https://t.co/AvPpRpa2u3 #minerandodados #DataScience #NLP

[Rodrigo Santana @rodrigofs10]

Você conhece bem seu dado? Sabe o que está tentando fazer? =&gt;
https://t.co/8pTDPop4JT #MachineLearning #DataScience

[Rodrigo Santana @rodrigofs10]

Agrupamento de Dados com o Orange Data Mining, aprenda aplicar o algoritmo sem usar uma linha de comando. ;P
https://t.co/KFdQ9fJIlc #minerandodados #datascience #orangedatamining #kmeans

[Leticia Portella @leleportella]

Finally! We have launched v0.1 of Bottery, the future @djangoproject of bots!

Special thanks to @nicoddemus and @GabRF :) https://t.co/i6BFE19Z7M

[Leticia Portella @leleportella]

@fmasanori @marcorougeth @nicoddemus @JairoJair @PythonSul @jesstemporal Olha onde fica a minha sessão de gifs na resposta @fmasanori ! Dá uma olhada na sua ai :) Ai qdo você clica ali abre uma aba pra busca :)

[Leticia Portella @leleportella]

Finalmente consegui ter tempo e calma emocional para sentar e escrever um pouco sobre como foi a @PythonSul na minha visão. Espero que gostem e se sintam um pouco mais perto de tudo o que aconteceu:

https://t.co/2g4XpV8HJW

[Rodrigo Santana @rodrigofs10]

Artigo novo: Classificando Músicas do Spotify com SVM em Python. 🤖🐍 #python #MachineLearning #spotify #datascience https://t.co/I1eVUVXxA4

[Rodrigo Santana @rodrigofs10]

Data Science: Cursos, Livros, Video Aulas..
https://t.co/fkJLT74PSP #DataScience #mineracaodedados #MachineLearning #Python #BigData 😀

[Larissa Mourullo @larismourullo]

Mais um artigo da nossa querida @larienmf pro devtranslate. 😍
https://t.co/UPjXYjydlz

[Rodrigo Santana @rodrigofs10]

#ImpeachmentDoGM Urgente https://t.co/Qwy8QTXMYI

[Rose D Barros @rosedbarros]

#ImpeachmentDoGM Urgente
https://t.co/TzWtMYt3m6