Large Language Models Fundamentals
Welcome to our research hub on the fundamentals of Large Language Models (LLMs). This page is continuously updated with curated articles that explore the latest breakthroughs and foundational concepts in LLM research.
Closed Source LLMs
This report introduces the Gemini family of multimodal models—Ultra, Pro, and Nano—capable of handling tasks across image, audio, video, and text. The Gemini Ultra model sets new benchmarks, excelling in 30 out of 32 evaluations, including human-expert performance on MMLU. The report also outlines responsible deployment strategies via platforms like Google AI Studio and Cloud […]
This report details the development of GPT-4, a multimodal model handling image and text inputs with text outputs. Although not surpassing humans in many tasks, GPT-4 achieves human-level performance in professional benchmarks, such as scoring in the top 10% on a simulated bar exam. Key innovations include post-training alignment for factual accuracy and scalable infrastructure […]
Open Source LLMs
The LLaMA collection consists of language models ranging from 7B to 65B parameters, trained on publicly available datasets. LLaMA-13B surpasses GPT-3 (175B) on most benchmarks, while LLaMA-65B competes with top models like Chinchilla-70B and PaLM-540B. All models are made available for research purposes.
Tests of LLMs on Canonical Educational Tasks
Some examples of the expressive and generative capabilities of LLM’s applied to tasks that are the basis for student evaluations in higher education.
This paper evaluates large language models (LLMs) like GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra in solving undergraduate-level control engineering problems using a custom dataset, ControlBench. Expert evaluations reveal Claude 3 Opus as the leading LLM in this domain, highlighting strengths and limitations in reasoning and accuracy. The study aims to advance the application […]
Emerging Capabilities of LLMs
This study evaluates LLMs in generating novel research ideas, comparing LLM outputs directly with ideas from over 100 expert NLP researchers. Findings reveal that LLM-generated ideas are rated as more novel (p < 0.05) but slightly lower in feasibility. The study highlights challenges in LLM self-evaluation and idea diversity, proposing a comprehensive design for future […]
This study introduces AlphaGeometry, a neuro-symbolic system for solving complex Euclidean geometry theorems at the olympiad level without human-provided proofs. Using a neural model trained on synthesized theorem-proving data, AlphaGeometry guides symbolic deduction to solve 25 of 30 recent olympiad problems, surpassing previous methods. Notably, it generates human-readable proofs, solves historical IMO problems, and generalizes […]
This study assesses GPT-4’s effectiveness in providing scientific feedback on research papers. Comparing GPT-4 feedback with human peer reviewers across over 4,800 papers, researchers found comparable overlap between GPT-4 and human reviews, especially for lower-quality submissions. In a user study of 308 researchers, 57.4% rated GPT-4 feedback as helpful, and 82.4% found it more beneficial […]
This study evaluates large language models (LLMs) for generating new product ideas targeting college students. Comparing human-generated ideas with those produced by GPT-4 (zero-shot and few-shot prompts), results show AI-generated ideas have higher purchase intent but lower novelty and diversity. Few-shot prompting slightly outperforms zero-shot, and AI-generated ideas are seven times more likely to rank […]