Foundation Models Shift Paradigms for Engineering and Energy

Understanding Foundation Models

From an engineering perspective, foundation models (FMs) can be understood as highly adaptable, pretrained artificial intelligence (AI) systems that have learned a rich, generalizable understanding of complex data patterns. Unlike traditional, task-specific engineering models that require extensive, labeled data sets for each new problem (e.g., a model for predicting a specific material’s fatigue life), FMs are trained on massive, diverse, and often unlabeled datasets (e.g., all available engineering documentation, sensor data streams, simulation outputs, and scientific literature). This pretraining allows them to develop a deep “intuition” for underlying relationships and structures within engineering data.

For the energy sector specifically, this means an FM can be trained on vast repositories of geological surveys, drilling reports, operational sensor data from pipelines and power plants, historical maintenance logs, and even regulatory documents. Once pretrained, these models can then be rapidly adapted or fine-tuned with relatively smaller, task-specific data sets to solve a wide array of engineering challenges—such as predicting reservoir performance, optimizing drilling operations, enhancing grid stability, and accelerating the discovery of new energy materials. Their ability to generalize across different data types and tasks makes them a powerful new tool in the engineer’s toolkit, enabling more efficient analysis, prediction, and decision-making in complex energy systems.

How Are They Different From Other Existing Methods Such as Deep Learning?

Useful FMs for Science and Engineering

The landscape of FMs is rapidly expanding, with many models specifically designed or adapted for scientific and engineering applications.

General and Cross-Domain Science FMs. These models are engineered to operate across multiple scientific disciplines, often integrating diverse data types to foster broader understanding and facilitate interdisciplinary discovery.

• AuroraGPT—This model leverages Department of Energy supercomputing resources to develop FMs for broad scientific domains such as biology, materials science, and chemistry. It aims to integrate AI into research by augmenting web-scale data with science-specific data sets for improved understanding and discovery.
   
• NatureLM—This is a sequence-based science FM handling complex data from molecules, proteins, DNA, RNA, materials, and associated text. Trained on a massive corpus, it aims for unified applications such as generating scientific entities and enabling cross-domain design based on text instructions.
  
• NVIDIA’s Polymathic AI—This focuses on building versatile FMs for scientific machine learning. It emphasizes leveraging heterogeneous numerical data sets and bridging different scientific subdisciplines, addressing the challenge of creating models that move beyond typical language or image data to truly understand and use varied scientific measurements and simulations.

Life Sciences and Biology FM. Tailored for biological sequences, structures, and functions, these models are pivotal in driving breakthroughs in medicine, biotechnology, and our fundamental understanding of life.

• AlphaFold Series—This model predicts high-accuracy 3D protein structures. Trained on vast public repositories of protein sequences and structures, these models represent a major breakthrough in understanding protein folding, crucial for drug discovery and understanding fundamental biological processes.
  
• The ESM Series (from Meta)—This model focuses on genomics and protein generation. Based on the Transformer architecture, it offers computational efficiency for large-scale screening of protein sequences compared to some alternatives. ESM3, for instance, has demonstrated an ability to generate functional proteins significantly different from known natural ones.
  
• DNABERT Series—Used in genomics, these models apply the BERT (Bidirectional Encoder Representations from Transformers) architecture, originally developed for natural language processing, to analyze DNA sequences. They help identify functional regions, predict regulatory elements, and understand the "language" of the genome.

Materials Science and Chemistry FMs. These models accelerate innovation in new material development by understanding and predicting properties, structures, and interactions of molecules and materials.

• GraphCL/MoICLR—These models are designed for molecular property prediction. They represent molecules as graphs and use contrastive learning techniques to learn robust representations that can predict various chemical properties or activities, proving highly useful in drug discovery and materials design.
  
•  M3GNet and Other MLIPS (Machine Learning Interatomic Potentials)—Used for molecular dynamics simulations, these models learn to quickly and accurately predict the forces between atoms, enabling simulations of material behavior over significantly longer timescales or larger scales than traditional quantum mechanical methods.
  
• CrystalLLM (based on LLaMA-2)—This focuses on crystal structures. Leveraging a large language model base, it is adapted and trained to predict the complex atomic arrangements in crystalline materials based on input information like chemical composition. Accurate prediction is key to discovering new materials with desired properties.
  
• MatterGen—This diffusion model is specifically designed for generating novel materials structures based on desired properties or constraints, thereby accelerating the process of searching for new functional materials.
  
• MoLFormer—Another model used for various molecular tasks, this uses a Transformer architecture and is applied to tasks such as molecular property prediction and drug discovery by learning complex relationships within molecular structures and vast chemical spaces.

Physics and Simulation FMs. These models incorporate or simulate physical laws and are often used to generate synthetic data, crucial for training other AI systems or for understanding complex physical phenomena.

• NVIDIA Cosmos WFMs—These generate physics-aware synthetic data. These models produce realistic simulations of physical environments and interactions (such as fluid dynamics or object behavior) that inherently adhere to physical laws. This is critical for training AI in areas such as robotics and autonomous systems where real-world data collection is difficult or dangerous.

Other Domain-Specific FMs. These models are tailored for specific scientific or engineering fields beyond the major areas listed previously, addressing unique challenges within those domains.

• IBM Granite for Science—This povides geospatial AI capabilities. It is designed to process and analyze large-scale spatial and temporal data, supporting scientific investigations in fields like Earth science, urban planning, and climate research by extracting insights from satellite imagery and geographic information systems.

Initiatives for Autonomous Discovery. These are high-level programs aimed at creating AI agents capable of leading scientific inquiry autonomously, accelerating the pace of scientific breakthroughs.

• DARPA’s FoundSci—This program aims to develop AI "autonomous scientists." The overarching goal is to create AI capable of skeptical reasoning, generating novel hypotheses, designing experiments, and interpreting results autonomously. This initiative represents a significant effort to accelerate the entire scientific discovery loop with AI.

Engineering and the Engineer’s Role in the Age of FMs

The advent of FMs is not merely an incremental improvement but a fundamental shift that will profoundly impact the day-to-day work, required skill sets, and overall culture within engineering disciplines, particularly in the energy sector. This section explores these key implications.

Reshaping the Research Workflow and Process. FMs are poised to redefine how engineering research and development are conducted, streamlining processes and enabling new avenues of inquiry.

• Automation of Routine Tasks—FMs can automate time-consuming, routine tasks such as data processing, literature review, and initial data analysis. This frees up engineers’ valuable time to focus on higher-value activities such as problem formulation, hypothesis generation, and complex design challenges.
• Leverage in Specific Tasks—These models provide significant leverage in specific tasks, including synthesizing complex information from disparate sources, analyzing vast data sets (e.g., sensor data from a refinery or seismic data from a new exploration site), and identifying subtle patterns that might be missed by human analysis.
• Bridging Disciplinary Boundaries—FMs inherently encourage interdisciplinary approaches by learning from and integrating concepts across diverse data sets. This capability fosters collaboration and encourages engineers to integrate concepts and methods from related fields, leading to more holistic solutions in complex systems such as integrated energy grids or advanced material development.

The Evolving Role of the Engineer: Human/AI Collaboration. The relationship between engineers and AI is shifting from tool-use to a more collaborative partnership, elevating the human role while demanding new forms of interaction.

• Focus on Higher-Level Work—With FMs handling much of the data crunching and pattern recognition, engineers can increasingly focus on higher-level cognitive work, such as formulating novel questions, defining research strategies, interpreting complex results, and making critical decisions that require nuanced judgment. This elevates the human role in the discovery and innovation process.
• Emergence of Human/AI Cognitive Partnership—A future is emerging where AI acts as an intelligent assistant and a source of new perspectives, helping engineers explore solutions they might not have considered. This cognitive partnership can lead to more innovative and robust engineering solutions.
• Human-in-the-Loop Approach—Despite their capabilities, FMs currently have limitations. A human-in-the-loop approach remains essential, ensuring critical human oversight for validating accuracy, mitigating biases, and applying ethical judgment, especially in safety-critical applications within the energy sector.
• Tools and Collaborators, Not Replacements—FMs should be viewed as powerful tools and collaborators that augment human intelligence, rather than as replacements for human engineers. Future development and integration should prioritize seamless, transparent workflows that enhance, not diminish, the engineer’s role.
• Fostering a Collaborative Culture—It is crucial to foster an engineering culture that views FMs as partners. This involves encouraging collaboration with AI tools while maintaining critical thinking and skepticism about AI outputs, understanding their probabilistic nature and potential for error.

Methodological and Trustworthiness Challenges. The integration of FMs introduces new considerations for integrity, validation, and evaluation within engineering.

• Validating Novel FM-Identified Patterns—A significant challenge lies in validating novel patterns, insights, or designs identified by FMs. New methodologies are needed to verify discoveries made by AI beyond traditional empirical testing, ensuring their scientific soundness and practical applicability.
• New Norms, Standards, and Best Practices—The scientific and engineering communities must develop new norms, standards, and best practices for conducting research and developing workflows and products with FMs. This includes guidelines for responsible AI usage, data provenance, and transparency in model application.
• Domain-Specific Benchmarks—Evaluating scientific FMs requires the development of domain-specific benchmarks that assess scientific plausibility, novelty of insights, and utility for discovery, rather than relying solely on general AI metrics such as accuracy or F1 score.
• Evolving Publication and Funding Requirements—Journals and funding agencies will need to adapt, developing new norms for reporting FM usage, archiving models and data sets, and establishing methods for validating AI-assisted findings to maintain research rigor and reproducibility.

Skills, Training, and Professional Development. The rapid evolution of FMs necessitates a continuous updating of skills and knowledge for current and future engineers.

• Understanding Model Limitations and Biases—Researchers and engineers must develop critical skills to assess the reliability and potential flaws of FM outputs, including understanding their inherent limitations and biases that might arise from training data.
• Effective Communication With FMs—Acquiring skills in effectively communicating tasks to FMs, often through prompt engineering, is crucial. This involves crafting well-designed prompts to guide AI behavior and elicit desired outputs for specific engineering problems.
• Adapting Pretrained Models—Engineers will need skills in adapting pretrained models through fine-tuning or other customization techniques, allowing them to tailor general FMs to specific scientific or engineering problems and proprietary data sets.
• Continuous Learning—The field of FMs is evolving at an extraordinary pace, requiring ongoing engagement with new developments, research, and best practices for engineering faculty and professionals.
• Interdisciplinary Collaborations—Industry professionals should actively engage in interdisciplinary collaborations to explore novel applications of FMs across different fields and share emerging best practices for their deployment in complex engineering scenarios.

Resources, Investment, and Institutional Support. Successful adoption and advancement of FMs in engineering research and practice require significant and strategic investment.

• Sustained and Strategic Investment—Key areas for investment include high-performance computing infrastructure, access to curated and high-quality scientific and engineering data sets, and robust training programs to upskill the workforce.
• Advocacy for Institutional Support—Engineers and researchers should advocate for institutional support, ensuring access to necessary computational resources and large, high-quality data sets where appropriate, as these are foundational for leveraging FMs effectively.

Broader Ecosystem and Policy Implications.

• Resource Concentration—While using pretrained FMs can democratize access to advanced AI capabilities for some, the development and training of these massive models concentrate significant computational and data resources in the hands of a few large entities. This raises concerns about the potential for increased inequality in guiding the development of FMs.
• Promoting Open Science and Accessibility—Policies promoting open science, shared infrastructure, and accessible models are vital. This ensures that the benefits of FMs broadly reach the entire scientific and engineering community, helping to mitigate the risk of exacerbating existing disparities in research and innovation in this field.

State of the Art and Weaknesses in FMs

While FMs offer immense promise for engineering, it is crucial to understand both their current cutting-edge capabilities and the inherent challenges and limitations that still need to be addressed.

State of the Art. The field of FMs is rapidly advancing, pushing boundaries in several key areas relevant to science and engineering:

• Multimodal FMs—A significant trend is the development of multimodal FMs that can seamlessly integrate and process diverse scientific data types, including text, images, biological sequences, and numerical sensor data. This allows for a more holistic understanding of complex systems and phenomena.
• Physics-Guided and Knowledge-Infused FMs—Researchers are actively developing models that incorporate established scientific laws, physical principles, and domain-specific knowledge directly into their architecture or training process. This improves the realism, accuracy, and trustworthiness of predictions, especially in critical engineering applications.
• Generalist Scientific FMs—Efforts are aimed at creating more generalist scientific FMs with broader applicability across multiple scientific and engineering disciplines. The goal is to enhance knowledge transfer between domains, allowing insights learned in one area to benefit another.
• Engines for AI Agents and Automated Scientific Discovery—Formation models are increasingly being developed as core components for AI agents and automated scientific discovery systems. These models power “self-driving labs” capable of autonomously generating hypotheses, designing experiments, executing robotic tasks, and interpreting results in a closed-loop fashion.
• Computational Efficiency—Ongoing work focuses on creating more computationally efficient FMs. This aims to reduce the immense resource burden associated with training and deploying these large models, making them more accessible and sustainable.
• Bias Detection and Mitigation—Robust methods for detecting and mitigating biases are being developed to address the challenge of biases that FMs can inadvertently learn and perpetuate from their vast training data sets.
• Privacy Preservation—Advanced techniques for privacy preservation are emerging to help protect sensitive data used in training and application, reducing the risk of information leakage and ensuring responsible data handling.

Weaknesses. Despite their impressive capabilities, FMs currently face several key limitations and challenges.

• Resource and Access Barriers
  
  • Computational Expense—The computational expense remains a significant weakness. Training and even fine-tuning and running inference with the very largest FMs are still substantially costly, requiring specialized hardware and considerable energy.
  • Equitable Access Concerns—This high cost leads to concerns about equitable access to cutting-edge AI. The financial and infrastructural requirements limit which researchers, institutions, or nations can effectively develop, use, or even critically scrutinize these powerful tools.
  • Limited Diversity in Development—High computational costs can inadvertently limit the diversity of developers and critical scrutiny, potentially slowing down progress on addressing crucial issues such as bias, interpretability, and ethical implications.
• Data Issues and Privacy Risks
  
  • Inherent Biases, Errors, and Sensitive Details—The vast training data sets used for FMs are compiled from diverse sources and often contain inherent biases, errors, and sensitive details. These data sets are not perfectly curated and can reflect societal prejudices or factual inaccuracies.
  • Reproduction of Undesirable Elements—FMs can inadvertently absorb and reproduce these undesirable elements from their training data. This manifests as biased outputs, the perpetuation of inaccuracies, or the reinforcement of stereotypes, which is particularly problematic in sensitive engineering applications.
  • Data Privacy Risks—FMs carry significant data privacy risks. They can inadvertently memorize and potentially leak sensitive private information from their training data, even if that data was not intended to be publicly exposed.
  • Reidentification Threats—Their outputs could potentially be used to reidentify individuals or proprietary information. Techniques such as membership inference attacks pose a threat to data privacy even without explicit memorization by the model.
• Trust, Interpretability, and Reliability
  
  • “Black Box” Nature—A major weakness of many large FMs is their “black box” nature, or lack of interpretability. It is often difficult to understand why an FM makes a specific prediction or reaches a particular conclusion, hindering trust and adoption in critical engineering scenarios.
  • Spurious Correlations—The lack of transparency complicates identifying spurious correlations. Models might rely on unreliable, noncausal patterns learned from complex data, leading to incorrect or misleading outputs.
  • Diagnosing and Mitigating Biases—The “black box” nature also makes it challenging to pinpoint the source of bias in the model’s decision-making process, making it difficult to diagnose and effectively mitigate inherent biases learned from data.
• Accountability Gaps
  
  • Ethical and Practical Challenge—Accountability is a key ethical and practical challenge. Determining who is responsible for harmful, incorrect, or biased scientific or engineering output generated by an FM is not straightforward, raising complex legal and ethical questions.

Future of Energy and Engineering Powered by FMs

FMs are not merely tools; they are catalysts fundamentally reshaping engineering, particularly within the energy sector, by accelerating discovery and enabling unprecedented innovation. Their ability to process vast, diverse data sets and identify complex patterns unlocks new efficiencies and capabilities across the entire lifecycle.

At the core of this transformation is the quest for deeper understanding. Current FMs excel at pattern recognition, but the next generation is moving towards causality and logic. Imagine FMs that don’t just predict equipment failure but explain why it occurs or suggest interventions in a complex energy grid with a clear understanding of their true effect. By combining data-driven insights with explicit physical laws and engineering knowledge, infusing logic into FMs promises more trustworthy and explainable AI for safety-critical operations and complex design processes in oil, gas, and renewables.

The quantum frontier is opening doors to solving previously intractable challenges. Quantum FMs hold the potential to revolutionize areas such as material science, enabling the precise simulation of molecules for designing next-generation battery components or advanced catalysts for carbon capture. They could even accelerate the discovery of novel superconductors for ultraefficient energy transmission, tackling problems that are fundamentally beyond the reach of classical supercomputers.

Perhaps the most exhilarating prospect is the rise of the self-improving scientist through meta-learning and autonomous discovery systems. Picture “self-driving” labs powered by FMs, automating the entire research cycle from hypothesis generation to experimental execution. These continuous, closed-loop learning systems can navigate vast, high-dimensional experimental spaces, rapidly identifying optimal conditions for material synthesis or uncovering nonintuitive phenomena. This could accelerate breakthroughs in energy storage, conversion, and exploration by orders of magnitude, minimizing human error and operating continuously.

In the realm of physics and complex energy systems, FMs are decoding critical challenges. They analyze vast diagnostic data for fusion energy research, optimizing reactor designs with digital twins that integrate various simulation codes. For environmental and climate science, FMs leverage diverse data for accurate climate predictions, synthetic data generation, and robust model ensembling, vital for understanding and managing energy’s environmental impact. Crucially, physics-guided FMs integrate fundamental physical laws, ensuring predictions are not only accurate but also physically realistic and trustworthy, essential for safety-critical energy applications.

The overarching shift in engineering is toward AI-first workflows. FMs are becoming indispensable at every stage of the product lifecycle: from intelligent design and rapid simulation, where engineers can explore thousands of iterations in seconds, to smart manufacturing with AI-enabled robotics and predictive maintenance that anticipates failures with unprecedented accuracy. This creates a powerful, virtuous cycle: More simulation data enhances FM accuracy; better FMs accelerate design and suggest superior candidates for simulation, which leads to faster innovation, more optimized products, and an ever-improving AI toolkit. This flywheel” effect represents a fundamental paradigm shift, promising exponential improvements in engineering efficiency and the capacity for innovation.

Ultimately, FMs are transforming engineers from data processors into strategic architects, collaborating with increasingly intelligent systems. This human/AI partnership is crucial for tackling the grand challenges of the energy transition, driving efficiency, enhancing safety, and charting new territories toward a more sustainable and innovative future.