# 斯坦福 CS329A — Self-Improving AI Agents

> 官网 https://cs329a.stanford.edu ｜ 2025 Autumn（2025-09-22 ~ 12-05）
> 讲师：Aakanksha Chowdhery、Azalia Mirhoseini ｜ 官方录播：Stanford Online YouTube

## 课程简介

讲授能通过与自身和环境交互而持续自我改进的 AI Agent 的最新技术与应用。先覆盖 LLM 自我改进技术（宪法式 AI、使用验证器、扩展测试时计算、搜索+LLM、RL 训练时扩展），再讲为 LLM 增强工具/代码/记忆、多模态交互编排，最后是多步推理规划、Agentic 工作流与鲁棒评测框架。

## 课表与必读论文（共 34 篇）

> ✅ 已下载全部 **34** 篇 → `~/Agent/CS329A/papers/`（119MB）

| # | 日期 | 主题 | 必读论文 |
|---|------|------|---------|
| 1 | Mon Sep 22 | Course Overview | （Guest/项目周，无指定论文） |
| 2 | Fri Sep 26 | Test-time Compute Scaling | Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (Brown et al. 2024) — ⏳<br>Archon: An Architecture Search Framework for Inference-Time Techniques (Saad-Falcon et al. 2024) — ⏳<br>Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Snell et al. 2024) — ⏳<br>How Do Large Language Monkeys Get Their Power (Laws)? — ⏳ |
| 3 | Mon Sep 29 | Robust Verification | Shrinking the Generation-Verification Gap with Weak Verifiers — ⏳<br>Training Verifiers to Solve Math Word Problems (Cobbe et al. 2021) — ⏳<br>Let's Verify step by step (Lightman et al. 2023) — ⏳<br>Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (Wang et al. 2023) — ⏳ |
| 4 | Fri Oct 3 | Learning from feedback with tools/code | ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al. 2022) — ⏳<br>RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning — ⏳<br>Constitutional AI: Harmlessness from AI Feedback — ⏳ |
| 5 | Mon Oct 6 | Multi-step Reasoning/Planning | SWiRL: Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use — ⏳<br>Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (Zhou et al. 2023) — ⏳<br>SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models — ⏳<br>ADaPT: As-Needed Decomposition and Planning with Language Models (Prasad et al. 2024) — ⏳<br>Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search — ⏳ |
| 6 | Fri Oct 10 | Train Time Scaling/Scaling RL | STaR: Bootstrapping Reasoning With Reasoning (Zelikman et al. 2022) — ⏳<br>DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — ⏳<br>DAPO: An Open-Source LLM Reinforcement Learning System at Scale — ⏳ |
| 7 | Mon Oct 13 | Open-Ended Evolution of Self-Improving Agents | Automated design of agentic systems — ⏳<br>The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (Lu et al. 2024) — ⏳<br>AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms — ⏳ |
| 8 | Fri Oct 17 | Self improvement with Search & Deep Research Agents | Competition-Level Code Generation with AlphaCode — ⏳<br>AlphaCode 2 Technical Report — ⏳<br>Search-o1: Agentic Search-Enhanced Large Reasoning Models — ⏳ |
| 9 | Mon Oct 20 | Guest Lecture Melvin Johnson (Google DeepMind) | （Guest/项目周，无指定论文） |
| 10 | Fri Oct 24 | Mid term presentations | （Guest/项目周，无指定论文） |
| 11 | Mon Oct 27 | Mid term presentations | （Guest/项目周，无指定论文） |
| 12 | Fri Oct 31 | Mid term presentations | （Guest/项目周，无指定论文） |
| 13 | Mon Nov 3 | Agentic Frameworks for Software Engineering | CodeMonkeys: Scaling Test-Time Compute for Software Engineering — ⏳<br>KernelBench: Can LLMs Write Efficient GPU Kernels? — ⏳<br>Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces — ⏳ |
| 14 | Fri Nov 7 | Augmenting Agents with Memory Guest Lecturer: Junchen Jiang (LMCache, UChicago) | Cartridges: Lightweight and general-purpose long context representations via self-study — ⏳<br>MemGPT: Towards LLMs as Operating Systems (Packer et al, 2023) — ⏳<br>CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — ⏳ |
| 15 | Mon Nov 10 | Guest Lecture Denny Zhou, Google DeepMind | （Guest/项目周，无指定论文） |
| 16 | Fri Nov 14 | Guest Lecture Thang Luong, Google DeepMind | （Guest/项目周，无指定论文） |
| 17 | Mon Nov 17 | Agentic Evaluations & Long-Horizon Tasks | Measuring AI Ability to Complete Long Tasks — ⏳<br>GDPVal: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — ⏳<br>DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis — ⏳ |
| 18 | Fri Nov 21 | Guest Lecture Misha Laskin (Reflection AI) | （Guest/项目周，无指定论文） |
| 19 | Mon Dec 1 | Guest Lecture Danny Driess (Physical Intelligence) | （Guest/项目周，无指定论文） |
| 20 | Fri Dec 5 | Future Research Areas | （Guest/项目周，无指定论文） |

## 完整论文链接清单

- ✅  **2407.21787_large_language_monkeys_scaling_inference.pdf**  [https://arxiv.org/pdf/2407.21787](https://arxiv.org/pdf/2407.21787)
- ✅  **2409.15254_archon_an_architecture_search_framework_.pdf**  [https://arxiv.org/pdf/2409.15254](https://arxiv.org/pdf/2409.15254)
- ✅  **2408.03314_scaling_llm_test_time_compute_optimally_.pdf**  [https://arxiv.org/pdf/2408.03314](https://arxiv.org/pdf/2408.03314)
- ✅  **2502.17578_how_do_large_language_monkeys_get_their_.pdf**  [https://arxiv.org/pdf/2502.17578](https://arxiv.org/pdf/2502.17578)
- ✅  **2506.18203_shrinking_the_generation_verification_ga.pdf**  [https://arxiv.org/pdf/2506.18203](https://arxiv.org/pdf/2506.18203)
- ✅  **2110.14168_training_verifiers_to_solve_math_word_pr.pdf**  [https://arxiv.org/pdf/2110.14168](https://arxiv.org/pdf/2110.14168)
- ✅  **2305.20050_let_s_verify_step_by_step_lightman_et_al.pdf**  [https://arxiv.org/pdf/2305.20050](https://arxiv.org/pdf/2305.20050)
- ✅  **2312.08935_math_shepherd_verify_and_reinforce_llms_.pdf**  [https://arxiv.org/pdf/2312.08935](https://arxiv.org/pdf/2312.08935)
- ✅  **2210.03629_react_synergizing_reasoning_and_acting_i.pdf**  [https://arxiv.org/pdf/2210.03629](https://arxiv.org/pdf/2210.03629)
- ✅  **2410.02089_rlef_grounding_code_llms_in_execution_fe.pdf**  [https://arxiv.org/pdf/2410.02089](https://arxiv.org/pdf/2410.02089)
- ✅  **2212.08073_constitutional_ai_harmlessness_from_ai_f.pdf**  [https://arxiv.org/pdf/2212.08073](https://arxiv.org/pdf/2212.08073)
- ✅  **2504.04736_swirl_synthetic_data_generation_multi_st.pdf**  [https://arxiv.org/pdf/2504.04736](https://arxiv.org/pdf/2504.04736)
- ✅  **2310.04406_language_agent_tree_search_unifies_reaso.pdf**  [https://arxiv.org/pdf/2310.04406](https://arxiv.org/pdf/2310.04406)
- ✅  **2506.05745_sprint_enabling_interleaved_planning_and.pdf**  [https://arxiv.org/pdf/2506.05745](https://arxiv.org/pdf/2506.05745)
- ✅  **2311.05772_adapt_as_needed_decomposition_and_planni.pdf**  [https://arxiv.org/pdf/2311.05772](https://arxiv.org/pdf/2311.05772)
- ✅  **2503.04412_wider_or_deeper_scaling_llm_inference_ti.pdf**  [https://arxiv.org/pdf/2503.04412](https://arxiv.org/pdf/2503.04412)
- ✅  **2203.14465_star_bootstrapping_reasoning_with_reason.pdf**  [https://arxiv.org/pdf/2203.14465](https://arxiv.org/pdf/2203.14465)
- ✅  **2402.03300_deepseekmath_pushing_the_limits_of_mathe.pdf**  [https://arxiv.org/pdf/2402.03300](https://arxiv.org/pdf/2402.03300)
- ✅  **2503.14476_dapo_an_open_source_llm_reinforcement_le.pdf**  [https://arxiv.org/pdf/2503.14476](https://arxiv.org/pdf/2503.14476)
- ✅  **2505.22954_automated_design_of_agentic_systems.pdf**  [https://arxiv.org/pdf/2505.22954](https://arxiv.org/pdf/2505.22954)
- ✅  **2408.06292_the_ai_scientist_towards_fully_automated.pdf**  [https://arxiv.org/pdf/2408.06292](https://arxiv.org/pdf/2408.06292)
- ✅  **AlphaEvolve_2025.pdf**  [https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf)
- ✅  **2203.07814_competition_level_code_generation_with_a.pdf**  [https://arxiv.org/pdf/2203.07814](https://arxiv.org/pdf/2203.07814)
- ✅  **AlphaCode2_TechReport_2023.pdf**  [https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf](https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf)
- ✅  **2501.05366_search_o1_agentic_search_enhanced_large_.pdf**  [https://arxiv.org/pdf/2501.05366](https://arxiv.org/pdf/2501.05366)
- ✅  **2501.14723_codemonkeys_scaling_test_time_compute_fo.pdf**  [https://arxiv.org/pdf/2501.14723](https://arxiv.org/pdf/2501.14723)
- ✅  **2502.10517_kernelbench_can_llms_write_efficient_gpu.pdf**  [https://arxiv.org/pdf/2502.10517](https://arxiv.org/pdf/2502.10517)
- ✅  **2410.15625_improving_parallel_program_performance_w.pdf**  [https://arxiv.org/pdf/2410.15625](https://arxiv.org/pdf/2410.15625)
- ✅  **2506.06266_cartridges_lightweight_and_general_purpo.pdf**  [https://arxiv.org/pdf/2506.06266](https://arxiv.org/pdf/2506.06266)
- ✅  **2310.08560_memgpt_towards_llms_as_operating_systems.pdf**  [https://arxiv.org/pdf/2310.08560](https://arxiv.org/pdf/2310.08560)
- ✅  **2405.16444_cacheblend_fast_large_language_model_ser.pdf**  [https://arxiv.org/pdf/2405.16444](https://arxiv.org/pdf/2405.16444)
- ✅  **2503.14499_measuring_ai_ability_to_complete_long_ta.pdf**  [https://arxiv.org/pdf/2503.14499](https://arxiv.org/pdf/2503.14499)
- ✅  **2510.04374_gdpval_evaluating_ai_model_performance_o.pdf**  [https://arxiv.org/pdf/2510.04374](https://arxiv.org/pdf/2510.04374)
- ✅  **2508.20033_deepscholar_bench_a_live_benchmark_and_a.pdf**  [https://arxiv.org/pdf/2508.20033](https://arxiv.org/pdf/2508.20033)