XGBoost

XGBoost (eXtreme Gradient Boosting) is a machine learning algorithm based on gradient-boosted decision trees. While LLMs and deep learning dominate headlines, XGBoost remains arguably the most important algorithm for structured, tabular data—the kind of data that most businesses actually run on.

The algorithm works by building an ensemble of decision trees sequentially, where each new tree focuses on correcting the errors of the previous ones. This "boosting" approach, combined with XGBoost's aggressive optimization (regularization, tree pruning, handling of missing values, parallel computation), produces models that are fast to train, highly accurate, and remarkably robust. It was created by Tianqi Chen in 2014 and quickly became the dominant algorithm for tabular prediction tasks.

XGBoost's track record is striking. It has won more Kaggle competitions than any other single algorithm. In 2025, despite the deep learning revolution, XGBoost and its variants (LightGBM, CatBoost) still outperform neural networks on most tabular data benchmarks. For predicting customer churn, credit risk, fraud detection, recommendation ranking, and countless business applications built on structured databases, gradient-boosted trees remain the tool of choice.

This creates an interesting dynamic in the AI landscape. While attention focuses on frontier foundation models and their trillion-parameter architectures, a huge fraction of production AI systems—the ones actually driving the 6% EBIT impact that leading organizations report—run on XGBoost or similar ensemble methods. The lesson: the right tool depends on the data. Unstructured data (text, images, audio) favors deep learning. Structured data often favors gradient-boosted trees. Both are essential components of the AI toolkit.

Gradient-Boosted Trees Versus Tabular Foundation Models in 2026

The claim that trees beat neural networks on tabular data dates from benchmarks such as Grinsztajn et al. (2022), which found tree-based models still state of the art on medium-sized datasets of around 10,000 samples. It now needs qualifying. TabArena, a continuously maintained benchmark of 51 curated datasets launched in June 2025, concluded that gradient-boosted trees "are still strong contenders" but that deep learning methods had caught up under larger time budgets with ensembling, and that foundation models excel on smaller datasets. In the full-benchmark leaderboard table of the paper's November 2025 version, tuned and ensembled LightGBM ranked third and CatBoost fourth behind two neural networks, with tuned and ensembled XGBoost eighth; with default settings CatBoost was the strongest of the three. That version was limited to IID datasets of 500 to 250,000 training samples.

Tabular foundation models have since moved up the same leaderboard, though the evidence comes largely from their developers. Prior Labs reported that TabPFN-2.5 (November 2025) substantially outperformed tuned tree-based models on TabArena and had an 87% win rate against default XGBoost on classification datasets up to 100,000 samples. Its TabPFN-3.5 report (September 2026) places two variants first and second of 89 methods. These are vendor-reported results, and the TabArena maintainers and the TabPFN authors overlap. The same report says tuned and ensembled neural networks retain the lead on grouped, temporal and large datasets, and the open TabPFN-3.5 weights are licensed for research and evaluation, not production use.

The practical position as of October 2026: XGBoost is no longer the default accuracy leader on small and mid-sized benchmark tables, but it remains cheap to train and serve, unrestricted in licence, well understood in production, and competitive when tuned. No independent, like-for-like comparison on very large or strongly time-dependent business data was found. The trade-offs are set out in TabPFN vs XGBoost.