Pooja Algikar @algikarpooja - Twitter Profile

Pinned Tweet

3 months ago

Built something I wished existed earlier: universal_uq is a model-agnostic uncertainty quantification wrapper focused on advanced model families, especially generative systems. You can wrap your model, choose compatible uncertainty methods, run inference, and get theory-backed: epistemic uncertainty aleatoric uncertainty OOD / novelty uncertainty calibration a deployment-facing aggregate score diagnosis of why uncertainty is high counterfactual suggestions for how to reduce it The main idea I focused around is most UQ tools are fragmented by task family, but in reality, we work across model types. So I wanted one wrapper and one dashboard that could work across: LLMs foundation / multimodal models generative models forecasting models graph models RL policies tabular classifiers/regressors One contribution I’m especially excited about is Foundation Uncertainty Score (FUS), a wrapper-level unified uncertainty metric. Such metric is essential because uncertainty quantification methods are scattered across: entropy-based, disagreement-based, novelty-based, calibration-like. FUS gives a common operating score across model families. Important caveat: FUS is: wrapper-level model type-dependent normalization-sensitive useful as a unified operating score, not yet a benchmark-calibrated standard also added: 1. recent UQ methods like semantic_density, uot_plan, and conformal forecasting variants 2. an uncertainty investigator that asks:why is uncertainty high here? 3. what information would reduce it? 4. what data should we collect? 5. counterfactual reruns to rank uncertainty-reduction suggestions by measured effects 6. latent uncertainty cartography for generative models a research dashboard with method-based plots and deployment decisions My big goal was more than computing uncertainty: “help someone understand whether to trust this model, why not, and what to do next” Repo: https://t.co/ErwTxMJquS If you work on LLM evals, foundation models, safety, calibration, or deployment, I’d love to hear what uncertainty views you wish existed.

algikarpooja's tweet photo. Built something I wished existed earlier: universal_uq is a model-agnostic uncertainty quantification wrapper focused on advanced model families, especially generative systems.

You can wrap your model, choose compatible uncertainty methods, run inference, and get theory-backed:

epistemic uncertainty
aleatoric uncertainty
OOD / novelty uncertainty
calibration
a deployment-facing aggregate score
diagnosis of why uncertainty is high
counterfactual suggestions for how to reduce it

The main idea I focused around is most UQ tools are fragmented by task family, but in reality, we work across model types. So I wanted one wrapper and one dashboard that could work across:

LLMs
foundation / multimodal models
generative models
forecasting models
graph models
RL policies
tabular classifiers/regressors

One contribution I’m especially excited about is Foundation Uncertainty Score (FUS), a wrapper-level unified uncertainty metric.

Such metric is essential because uncertainty quantification methods are scattered across: entropy-based, disagreement-based, novelty-based, calibration-like. FUS gives a common operating score across model families.

Important caveat:
FUS is:
wrapper-level
model type-dependent
normalization-sensitive
useful as a unified operating score, not yet a benchmark-calibrated standard

also added:
1. recent UQ methods like semantic_density, uot_plan, and conformal forecasting variants
2. an uncertainty investigator that asks:why is uncertainty high here?
3. what information would reduce it?
4. what data should we collect?
5. counterfactual reruns to rank uncertainty-reduction suggestions by measured effects
6. latent uncertainty cartography for generative models
a research dashboard with method-based plots and deployment decisions

My big goal was more than computing uncertainty:
“help someone understand whether to trust this model, why not, and what to do next”
Repo:
https://t.co/ErwTxMJquS

If you work on LLM evals, foundation models, safety, calibration, or deployment, I’d love to hear what uncertainty views you wish existed.

0

7

0

3

2K

Pooja Algikar

@algikarpooja

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users