Still using using XGBoost (substitute your favourite classifier) and think it outputs probabilities? Read on.
Probabilistic Prediction in scikit-learn.
'Adding confidence measures to predictive models should increase the trustworthiness, but only if the models are well-calibrated.
Historically, some algorithms like logistic regression, but also neural networks, have been considered to produce well-calibrated probability estimates off-the-shelf. Other techniques, like decision trees and Naive Bayes, on the other hand, are infamous for being significantly overconfident in their probabilistic predictions. In this paper, a large experimental study is conducted to investigate how well-calibrated models produced by a number of algorithms in the scikit-learn library are out-of-the-box, but also if either the built-in calibration techniques Platt scaling and isotonic regression, or Venn-Abers, can be used to improve the calibration.
The results show that of the seven algorithms evaluated, the only one obtaining well-calibrated models without the external calibration is logistic regression. All other algorithms, i.e., decision trees, adaboost, gradient boosting, kNN, naive Bayes and random forest benefit from using any of the calibration techniques. In particular, decision trees, Naive Bayes and the boosted models are substantially improved using external calibration.
From a practitioner’s perspective, the obvious recommendation becomes to incorporate calibration when using probabilistic prediction.'
https://t.co/uI5i4OgLte
#algorithms #neuralnetworks #uncertainty #machinelearning #conformalprediction
this paper's nuts. for sentence classification on out-of-domain datasets, all neural (Transformer or not) approaches lose to good old kNN on representations generated by.... gzip https://t.co/6eZiXlJxOX
Deep learning neural networks are notoriously miscalibrated and for several years people thought they can fix them using using Bayesian methods to quantify uncertainty.
And now there is complete disillusionment including by some of the leading bayesians. The way to fix it is not by creating methods that don’t have validity guarantees though but by using #conformalprediction that can be deployed as a reliable and safe UQ layer on top of any model including deep learning without any modification of the model itself. A powerful incentive that no other methods can ever offer.
”Assessing the predictive uncertainty of deep neural networks is crucial for safety- related applications of deep learning. Although Bayesian deep learning offers a principled framework for estimating model uncertainty, the approaches that are commonly used to approximate the posterior often fail to deliver reliable estimates of predictive uncertainty.”
#deeplearning #safety #neuralnetworks #conformalprediction #machinelearning
We've just released the first version of our Deep Learning Tuning Playbook! This is our attempt to distill our process for actually getting good results with deep learning. We emphasize hyperparameter tuning since it has been a large pain point. https://t.co/PjeJVWeOzS
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones
Jiawei Zhao, Florian Tobias Schaefer, Anima Anandkumar
https://t.co/FwExkRvshm
🎉 It's finally ready! I've been hacking on something for the past couple months and it's absurdly great. It's time for more powerful web apps.
Announcing "A future for SQL on the web": https://t.co/TrXcfMoHa6
This article made me dig a bit more into Zig's ability to cross compile C and C++. Honestly, it is hands down the best. The list of targets is _insane_. I think @andy_kelley did ship Clang/LLVM with all possible os/arch combinations. I'm really impressed. https://t.co/3Gi8aRDK80
New blog post: introduction to gradient-based optimization with constraints: reparametrization, projected gradient, cool connections in between.
🔮Geometry insights show that 🪞mirror descent = reparametrization with a "straight-through" sigmoid!
https://t.co/KQcq6mkM1i
@Sol_HSA See: https://t.co/mWjlJS2Me4 - that calls it round-robin iterative algorithm until a fixpoint. A variation is the worklist algorithm, where changed nodes are added back to the worklist.