07B: Sample Hardness

Materials:

Date: Friday, 13-Sep-2024

Pre-work:

  1. AIC a criteria for model selection
  2. cords for a collection of works/implementations based on subset selection

In-Class

  1. Characterizing data difficulty or sample hardness.
  2. Look at some statistics like Relative Mahalanobis Distance (which some used to flag OOD, others used to measure sample hardness), Perplexity (cross entropy between two models, one model and data or between), Trust Scores
  3. Sample easiness on training performance and generalization error
  4. See this notebook where we walk through these concept on a toy dataset

Post-class

  1. [paper] Learning Sample Difficulty from Pre-trained Models for Reliable Prediction
  2. [paper] A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection
  3. [paper] Dissecting Sample Hardness: A fine-grained analysis of hardness characterization methods for data-centric AI
  4. [paper] To Trust or Not To Trust A Classifier

Additional Reading (optional)

  1. [paper] Understanding Dataset difficulty
  2. [tools] Pytorch-ood - a collection of techniques to detect OOD in PyTroch. Mostly image focussed.
  3. [tools] PyOD - a collection of anomaly detection techniques
  4. [tools] DEEL - a collection of OOD, XAI, and other techniques

Notes

  1. Not all examples are equal (in the eyes of the model). There can be many reasons.
  2. There can be outliers (in the feature space, in the label space or both in the feature and label space).
  3. Outliers affect the model performance in different ways.
  4. A suite of techniques, preferably model-agnostic, are needed to quantify sample hardness and make them available at dataset level (train set) and also at inference time.