Foundation Model
Definition
A foundation model is a model that is trained at large scale on vast and diverse data (mainly through self-supervision) and, through processes such as fine-tuning, can be broadly adapted to a variety of downstream tasks.
Explanation
The term foundation model was first proposed in August 2021 by the Center for Research on Foundation Models (CRFM) under Stanford’s Human-Centered Artificial Intelligence (HAI) institute. They defined a foundation model as “any model that is trained on broad data (generally using self-supervision at scale) that can be adapted to a wide range of downstream tasks.”
The researchers held that existing terms failed to fully capture this concept. A large language model is too narrow, since its subject is not limited to language; a self-supervised model is too specific to the training method; and a pretrained model gives the impression that what really matters happens before pretraining.
Representative examples of foundation models include models such as ChatGPT, Gemini, and Claude. Trained without regard to field (mathematics, physics, the humanities, etc.) or data format (images, audio, text, etc.), they broadly handle the various tasks that users request.
Foundation model is often used interchangeably with universal model, but the emphasis differs. Whereas a universal model focuses on handling multiple problems within a single domain, such as atomic systems, a foundation model emphasizes that it is trained at large scale on vast data spanning modalities and fields.