Topics

A list of topics we will cover.

Conformal Prediction

  • Exchangeable Data: Full and Split Conformal Prediction
  • Distribution Shift and Time Series Data
  • Adaptive/Adversarial Conformal Prediction
  • Threshold Calibrated Multivalid Conformal Prediction

(Multi)Calibration

  • Proper Scoring Rules, Calibration, and Regret
  • Algorithms for offline (batch) multicalibration
  • Algorithms for online (adversarial) multicalibration
  • Moment Multicalibration
  • Applications of Multicalibration
    • Downstream Unconstrained Optimization (Omnipredictors)
    • Downstream Constrained Optimization
    • Proxies for Downstream Measurement
    • Distribution Shift

Tests and Predictors

  • Ignorantly Passing Tests: Possibility and Hardness
  • Outcome Indistinguishability

Other Topics

  • Smooth Calibration
  • Applications of Calibration in Mechanism Design
  • The Reference Class Problem

A list of papers we will draw from:

  1. Conformal Prediction
    1. A Gentle Introduction to Conformal Prediction
    2. A Tutorial on Conformal Prediction
    3. Predictive Inference with the Jackknife+
    4. Exact and Robust Conformal Inference Methods for Predictive Machine Learning with Dependent Data
    5. Mondrian Confidence Machines
    6. Adaptive Conformal Inference Under Distribution Shift
    7. Conformalized Online Learning: Online Calibration Without a Holdout Set
    8. Practical Adversarial Multivalid Conformal Prediction
  2. Calibration
    1. The Well Calibrated Bayesian
    2. Calibration Based Empirical Probability
    3. Asymptotic Calibration
    4. Calibration for the (Computationally Identifiable) Masses
    5. Moment Multicalibration for Uncertainty Estimation
    6. Online Multivalid Learning: Means Moments and Prediction Intervals
    7. Omnipredictors
    8. Universal adaptability: Target-independent inference that competes with propensity scoring
    9. Smooth calibration, leaky forecasts, finite recall, and Nash dynamics
    10. Multicalibrated Regression for Downstream Fairness
    11. Omnipredictors for Constrained Optimization
  3. Passing Distributional Tests Beyond Calibration
    1. Outcome Indistinguishability
    2. Good Randomized Sequential Probability Forecasting is Always Possible
    3. Falsifiability
    4. The Reproducible Properties of Correct Forecasters
  4. Calibration and Game Theory/Mechanism Design
    1. Calibrated Learning and Correlated Equilibrium
    2. Calibrated Incentive Contracts
    3. Prior Free Dynamic Allocation under Limited Liability
  5. The Reference Class Problem
    1. The Reference Class Problem is Your Problem Too
    2. On Individual Risk
    3. A Practical Solution to the Reference Class Problem
    4. Model Multiplicity: Opportunities, Concerns, and Solutions
    5. Reconciling Individual Probability Forecasts