A list of topics we will cover.

Conformal Prediction

• Exchangeable Data: Split Conformal Prediction
• Adaptive/Adversarial Conformal Prediction
• Group and Action Conditionally Valid Conformal Prediction
• Conformal Prediction Under Distribution Shift

(Multi)Calibration

• Calibration as “Trustworthiness” for decision making
• Proper Scoring Rules, Calibration, and Regret
• Algorithms for offline (batch) multicalibration
• Algorithms for online (adversarial) multicalibration
• Decision Calibration
• Applications of Multicalibration
  • Downstream Unconstrained Optimization (Omnipredictors)
  • Downstream Constrained Optimization
  • Proxies for Downstream Measurement
  • Distribution Shift
  • Ensembling
  • Agreement

Multigroup Optimality in Learning

• The distributional setting (via list update)
• The online adversarial setting (via multiobjective optimization)

Tests and Predictors

• Ignorantly Passing Tests: Possibility and Hardness
• Outcome Indistinguishability

Other Topics

• Smooth Calibration
• Applications of Calibration in Mechanism Design
• The Reference Class Problem

A list of papers related to the topic that we will draw from and/or that you might use as a starting point for a project:

1. Conformal Prediction
   1. A Gentle Introduction to Conformal Prediction
   2. A Tutorial on Conformal Prediction
   3. Predictive Inference with the Jackknife+
   4. Exact and Robust Conformal Inference Methods for Predictive Machine Learning with Dependent Data
   5. Mondrian Confidence Machines
   6. Adaptive Conformal Inference Under Distribution Shift
   7. Conformalized Online Learning: Online Calibration Without a Holdout Set
   8. Practical Adversarial Multivalid Conformal Prediction
   9. Batch Multivalid Conformal Prediction
2. Calibration
   1. The Well Calibrated Bayesian
   2. Calibration Based Empirical Probability
   3. Asymptotic Calibration
   4. Calibration for the (Computationally Identifiable) Masses
   5. Moment Multicalibration for Uncertainty Estimation
   6. Online Multivalid Learning: Means Moments and Prediction Intervals
   7. Smooth calibration, leaky forecasts, finite recall, and Nash dynamics
   8. Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration
   9. Low Degree Multicalibration
   10. “Calibeating”: Beating Forecasters at Their Own Game
   11. Online Minimax Multiobjective Optimization: Multicalibeating and Other Applications
   12. Multicalibration as Boosting for Regression
   13. The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation
   14. Oracle Efficient Online Multicalibration and Omniprediction
   15. A Unifying Theory of Distance From Calibration
   16. High Dimensional Prediction for Sequential Decision Making
   17. Forecasting for Swap Regret for All Downstream Agents
   18. Multicalibration for Confidence Scoring in LLMs
   19. Loss Minimization Yields Multicalibration for Large Neural Networks
   20. When is Multicalibration Post Processing Necessary?
   21. Calibrated Language Models Must Hallucinate
   22. A Unifying Perspective on Multicalibration: Game Dynamics and Multi-Objective Learning
   23. Breaking the T^{2/3} Barrier for Sequential Calibration
   24. An Elementary Predictor Obtaining 2T^{1/2} Distance to Calibration
3. Multigroup Optimal Learning
   1. Advancing Subgroup Fairness via Sleeping Experts
   2. Multigroup Agnostic PAC Learnability
   3. An Algorithmic Framework for Bias Bounties
   4. Simple and near-optimal algorithms for hidden stratification and multi-group learning
   5. Oracle Efficient Algorithms for Groupwise Regret
4. Applications of Multicalibration
   1. Omnipredictors
   2. Multiaccurate Proxies for Downstream Fairness
   3. Universal adaptability: Target-independent inference that competes with propensity scoring
   4. Multicalibrated Regression for Downstream Fairness
   5. Omnipredictors for Constrained Optimization
   6. Loss Minimization Through the Lens of Outcome Indistinguishability
   7. Making Decisions Under Outcome Performativity
   8. From Pseudo-Randomness to Multi-Group Fairness and Back
   9. Complexity Theoretic Applications of Multi-Calibration
   10. Omnipredictors for Regression and the Approximate Rank of Convex Functions
   11. Bridging Multicalibration and Out-of-Distribution Generalization Beyond Covariate Shift
   12. Orthogonal Causal Calibration
5. Passing Distributional Tests Beyond Calibration
   1. Outcome Indistinguishability
   2. Good Randomized Sequential Probability Forecasting is Always Possible
   3. Falsifiability
   4. The Reproducible Properties of Correct Forecasters
   5. Comparative Testing of Experts
6. Calibration and Game Theory/Mechanism Design
   1. Calibrated Learning and Correlated Equilibrium
   2. Calibrated Incentive Contracts
   3. Prior Free Dynamic Allocation under Limited Liability
   4. Efficient Prior-Free Mechanisms for No Regret Agents
7. The Reference Class Problem
   1. The Reference Class Problem is Your Problem Too
   2. On Individual Risk
   3. A Practical Solution to the Reference Class Problem
   4. Model Multiplicity: Opportunities, Concerns, and Solutions
   5. Reconciling Individual Probability Forecasts
   6. Reconciling Model Multiplicity for Downstream Decision Making