A list of topics we will cover.

Conformal Prediction

• Exchangeable Data: Full and Split Conformal Prediction
• Distribution Shift and Time Series Data
• Adaptive/Adversarial Conformal Prediction
• Threshold Calibrated Multivalid Conformal Prediction

(Multi)Calibration

• Proper Scoring Rules, Calibration, and Regret
• Algorithms for offline (batch) multicalibration
• Algorithms for online (adversarial) multicalibration
• Moment Multicalibration
• Applications of Multicalibration
  • Downstream Unconstrained Optimization (Omnipredictors)
  • Downstream Constrained Optimization
  • Proxies for Downstream Measurement
  • Distribution Shift

Tests and Predictors

• Ignorantly Passing Tests: Possibility and Hardness
• Outcome Indistinguishability

Other Topics

• Smooth Calibration
• Applications of Calibration in Mechanism Design
• The Reference Class Problem

A list of papers related to the topic that we will draw from and/or that you might use as a starting point for a project:

1. Conformal Prediction
   1. A Gentle Introduction to Conformal Prediction
   2. A Tutorial on Conformal Prediction
   3. Predictive Inference with the Jackknife+
   4. Exact and Robust Conformal Inference Methods for Predictive Machine Learning with Dependent Data
   5. Mondrian Confidence Machines
   6. Adaptive Conformal Inference Under Distribution Shift
   7. Conformalized Online Learning: Online Calibration Without a Holdout Set
   8. Practical Adversarial Multivalid Conformal Prediction
   9. Batch Multivalid Conformal Prediction
2. Calibration
   1. The Well Calibrated Bayesian
   2. Calibration Based Empirical Probability
   3. Asymptotic Calibration
   4. Calibration for the (Computationally Identifiable) Masses
   5. Moment Multicalibration for Uncertainty Estimation
   6. Online Multivalid Learning: Means Moments and Prediction Intervals
   7. Smooth calibration, leaky forecasts, finite recall, and Nash dynamics
   8. Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration
   9. Low Degree Multicalibration
   10. “Calibeating”: Beating Forecasters at Their Own Game
   11. Online Minimax Multiobjective Optimization: Multicalibeating and Other Applications
   12. Multicalibration as Boosting for Regression
   13. The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation
3. Applications of Multicalibration
   1. Omnipredictors
   2. Multiaccurate Proxies for Downstream Fairness
   3. Universal adaptability: Target-independent inference that competes with propensity scoring
   4. Multicalibrated Regression for Downstream Fairness
   5. Omnipredictors for Constrained Optimization
   6. Loss Minimization Through the Lens of Outcome Indistinguishability
   7. Making Decisions Under Outcome Performativity
4. Passing Distributional Tests Beyond Calibration
   1. Outcome Indistinguishability
   2. Good Randomized Sequential Probability Forecasting is Always Possible
   3. Falsifiability
   4. The Reproducible Properties of Correct Forecasters
   5. Comparative Testing of Experts
5. Calibration and Game Theory/Mechanism Design
   1. Calibrated Learning and Correlated Equilibrium
   2. Calibrated Incentive Contracts
   3. Prior Free Dynamic Allocation under Limited Liability
6. The Reference Class Problem
   1. The Reference Class Problem is Your Problem Too
   2. On Individual Risk
   3. A Practical Solution to the Reference Class Problem
   4. Model Multiplicity: Opportunities, Concerns, and Solutions
   5. Reconciling Individual Probability Forecasts