Foundation Models

Foundation models are large-scale AI models trained on broad, diverse datasets that serve as the base layer for a wide range of downstream applications. The term, coined by Stanford's Center for Research on Foundation Models, captures a key insight: a single powerful model can be adapted to countless specific tasks, rather than building separate models for each application.

The frontier foundation models of 2026—Anthropic's Claude, OpenAI's GPT series, Google's Gemini, Meta's Llama, and DeepSeek—are large language models at their core, but increasingly multimodal: they process and generate text, images, audio, video, and code. They serve as the reasoning engine for AI agents, the generation engine for creative AI tools, and the intelligence layer for enterprise applications.

The economics of foundation models have undergone radical deflation. Training costs remain enormous—hundreds of millions of dollars for frontier models—but inference costs have plummeted 92% in three years. This creates an unusual market dynamic: a small number of labs can afford to build frontier models, but the cost of using them is approaching commodity pricing, especially as open-source alternatives like DeepSeek and Llama compete on quality.

Foundation models are "foundational" in the same way that operating systems are: they create a platform upon which an ecosystem of applications, tools, and agents is built. The Model Context Protocol, agent frameworks, and the agentic web are all application layers that depend on foundation models as their substrate. The quality, cost, and accessibility of these models directly determines the pace of innovation across the entire stack.

Beyond Text: Tabular Foundation Models

The Stanford report that introduced the term (Bommasani et al., August 2021) defined foundation models as models "trained on broad data at scale" that are "adaptable to a wide range of downstream tasks." Nothing in that definition requires text, and the idea has since been applied to the rows-and-columns data that most organizations hold. Tabular foundation models are the clearest case. TabPFN (Hollmann et al., 2022) is a transformer trained offline on synthetic datasets; at prediction time it reads a new table's labelled rows as context and predicts the rest in a forward pass, with no per-dataset training or hyperparameter tuning. The first version handled about 1,000 training rows. Prior Labs' TabPFN-3.5 technical report (September 2026) says the model accepts up to 1 million rows and 20,000 features, and reports first place on the TabArena benchmark of 51 datasets. That ranking is vendor-reported, and the report itself notes that tuned and ensembled neural networks still lead on grouped, temporal and large datasets.

Behavioural and Event-Sequence Foundation Models

Behavioural foundation models take a third kind of input: ordered sequences of user events such as transactions, clicks and sessions. They are usually trained the way language models are, by predicting the next event, and the resulting representations are reused for tasks such as fraud detection, recommendation and churn or propensity scoring. Public evidence is thinner than for text or tables. A June 2026 paper (Gabrielsson) ran roughly 600 training runs spanning 10^15 to 10^19 FLOPs on real interaction data and found that a small event embedder, about 2% of parameters, was compute-optimal at every budget tested, and that the best training recipe changes with the evaluation metric. A July 2026 paper from a large Eastern European bank (Rusakov et al.) describes a production model pretrained on multimodal event sequences whose representations are combined with existing engineered features; the authors say it outperforms task-specific models, but the abstract gives no figures.

What Carries Over, and What Does Not

The shared pattern is pretraining once on broad data and reusing the result across tasks. The economics differ. Text models gain from the open web; tabular models lean on synthetic priors; behavioural models are generally trained on one organization's private logs, so they are foundational within a company and not across an industry. Adaptation differs as well: language models are prompted or fine-tuned, tabular models predict in context, and event-sequence models mostly supply embeddings to a smaller downstream model. In the non-text cases the incumbents are well-tuned task-specific models such as XGBoost, and the evidence that a foundation model beats them is specific to the benchmark, the data size and who ran the comparison.