Once an LLM application is deployed, the work is not finished.
Now you need to understand what happens when real users start using it:
- What questions do they ask?
- Which documents are retrieved?
- What context is passed to the model?
- What answer is generated?
- Was the answer useful?
- Where does the pipeline fail?
In the next LLM Zoomcamp workshop, we'll focus on monitoring.
๐ Wednesday, June 3, 2026
๐ 12:00-13:30 CET
๐ YouTube
We'll go through a hands-on example and look at how to:
- Collect traces from an LLM pipeline
- Instrument LLM calls and retrieval steps
- Store chat history and user interactions
- Collect user feedback on generated answers
- Track answer quality over time
- Use dashboards to inspect system behavior
- Run evaluations on production traces
Before deployment, we evaluate on test data. After deployment, we use traces, feedback, and production metrics to understand what actually happens in the system. This helps us detect issues in retrieval, generation, latency, cost, and answer quality.
Register here: https://t.co/fXhQliFWgM
CHEATCODE FOR LIFE:
1. Psalm 42 - when worry strikes
2. Psalm 112 - if you feel weak
3. Psalm 27 - when fear arises
4. Psalm 118 - if you lack confidence
5. Psalm 61 - if you need an answer
6. Psalm 41 - if you're battling sickness
7. Psalm 91 - when you need protection
๐ ๐๐ฟ๐ผ๐บ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ ๐๐ผ ๐ฃ๐ฟ๐ฎ๐ฐ๐๐ถ๐ฐ๐ฒ:
A new ๐ ๐ฒ๐ฑ๐ถ๐ฐ๐ฎ๐น ๐๐ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ shows end-to-end ๐๐ด๐ฒ๐ป๐๐ from data to reasoning and Iโll show you how you ๐ฏ๐๐ถ๐น๐ฑ ๐ถ๐ ๐๐ผ๐๐ฟ๐๐ฒ๐น๐ณ. ๐
๐ช๐ต๐ฎ๐ ๐๐ผ๐ ๐ฏ๐๐ถ๐น๐ฑ:
In hospitals, data comes as notes, scans, and spreadsheets โ fragmented and messy.
Youโll build an AI Agent system that turns raw hospital data into clean, private, interpretable predictions.
Powered by cooperating agents โ no manual work.
Fast. Transparent. Scalable.
๐ฆ๐๐ฒ๐ฝ๐:
ใ1ใ Ingestion Identifier Agent
Checks incoming data: spreadsheet, clinical note, or scan image.
Auto-detects type so the right workflow starts.
โ Think: file scanning + MIME detection + structured-data check.
โ Framework: implement inside an Ingestion Agent with CrewAI or LangGraph.
ใ2ใ Data Anonymizer Agent
Privacy first: removes names, IDs, record numbers.
Delivers fully anonymized data ready for analysis.
โ Think: regex filters + hashing + redaction layer.
โ Framework: a local PrivacyAgent for text & image anonymization.
ใ3ใFeature Extraction Agent
Understands what the data means.
Tables โ headers like Age, Gender, Diagnosis.
Images โ scan type & condition (e.g., colonoscopy โ polyp).
โ Think: header parsing + embedding similarity + lightweight vision labeling.
โ Framework: MetadataAgent with pandas (tables) or simple vision modules (scans).
ใ4ใ ModelโData Feature Matcher Agent
Chooses the best model automatically.
Text โ LLM for reading/reasoning.
Tabular โ structured model.
Imaging โ VLM for medical scans.
โ Think: feature matching + routing logic + model registry.
โ Framework: ModelSelectorAgent that switches between LLMs, VLMs, or structured models dynamically.
ใ5ใ Preprocessing Recommender & Implementor Agents
Recommend and apply cleaning/normalization before inference.
Tables scaled, missing values filled; images resized.
Everything optimized for the selected model.
โ Think: pandas.fillna(), StandardScaler(), torchvision.transforms.
โ Framework: PreprocessorAgent linking cleaning steps to model metadata.
ใ6ใ Model Inference Agent (with Explainability)
Runs the model and explains decisions.
Predicts (e.g., detect a polyp, estimate risk) and shows why (features or image regions).
โ Think: SHAP values + attention heatmaps + local explanations.
โ Framework: ExplainabilityAgent connected to model outputs.
Paper: https://t.co/KLiuT9GihG
๐ In my AI Agents Mastery training,
I teach you to build systems like thisโstep by step.
From Healthcare to Finance to Education,
youโll connect these building blocks into real, working, explainable AI agents.
โฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃโฃ
โซธ๊ Want to build Real-World AI Agents?
Join My ๐๐ฎ๐ป๐ฑ๐-๐ผ๐ป ๐๐ ๐๐ด๐ฒ๐ป๐ ๐ฑ-๐ถ๐ป-๐ญ ๐ง๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด !
โ Build Agents for Healthcare, Finance, Smart Cities & More
โ Master 5 Modules: ๐ ๐๐ฃ ยท LangGraph ยท PydanticAI ยท CrewAI ยท OpenAI Swarm
โ Includes 9 Full Projects
๐ ๐๐ป๐ฟ๐ผ๐น๐น ๐ก๐ข๐ช (๐ฑ๐ฒ% ๐ข๐๐):
https://t.co/5i2v1fIrhJ