Can context replace training?
Tabular foundation models make a compelling promise: instead of retraining a model for each dataset, provide examples in context and let a pretrained model infer the task. I wanted to test that claim on messy, real-world data rather than a polished benchmark alone.
The question was not only whether Nori could win on accuracy. I also wanted to understand how the accuracy changed as context grew and what that improvement cost in inference time.
Keep the test population fixed.
I used roughly 20,000 observations from the NYC 2015 Street Tree Census. The test set stayed fixed at 500 rows while I varied how much context Nori received. That made the comparison about context size rather than about different test samples.
- Fixed test set: 500 rows
- Context sizes: 250, 500, 1,000, 2,000, 4,000, 6,000
- Baselines: Random Forest, Gradient Boosting, HistGradientBoosting, Ridge
- Metrics: R², MAE, prediction time
Holding the test population fixed made the context-size curve much easier to interpret. It does not remove every source of variance, but it avoids confusing a context effect with a changing evaluation sample.
A small accuracy edge, a large runtime tradeoff.
Across the sweep, Nori maintained a modest R² edge over HistGradientBoosting. The system-level story was more interesting: the quality improved with additional context, but prediction time grew steeply while HGB remained sub-second.
Accuracy is not the whole benchmark.
A 0.04–0.07 R² edge can matter, but whether it is worth paying for depends on the application. If predictions are latency-sensitive and retraining is cheap, a classical model can still be the better engineering choice. If rapid adaptation without task-specific training matters more, the foundation-model approach becomes more compelling.
The benchmark made the model-selection problem feel much more like a systems decision: accuracy, context length, hardware support, latency, and operational simplicity all move together.
The benchmark changed the question.
- Experimental design matters as much as the headline metric.
- Foundation-model convenience does not remove infrastructure constraints.
- Small metric gains should be interpreted against latency, retraining burden, and error cost.
- A useful benchmark should make the deployment decision clearer, not just declare a winner.