> ## Content Index
> Fetch the complete content index at: https://www.ai-360.online/llms.txt
> Use this file to discover other available public pages before exploring further.

# OpenAI Introduces MLE-bench
- URL: https://www.ai-360.online/openai-introduces-mle-bench/
- Published: 2024-10-11T06:33:05.000Z
- Updated: 2024-10-11T06:33:05.000Z
- Description: OpenAI's MLE-bench tests AI on ML tasks. Best AI setup achieved bronze-level in 16.9% of competitions. Study explores resource-scaling and contamination effects.
- Author: AI-360
- Tags: OpenAI, News

OpenAI has unveiled MLE-bench in a blog post and accompanying paper. A new benchmark designed to measure how well AI agents perform at machine learning engineering tasks.

MLE-bench curates ''75 Machine Learning engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments.'' The benchmark establishes human baselines for each competition using Kaggle's publicly available leaderboards.

To evaluate AI performance, the researchers used open-source agent scaffolds to test several frontier language models on the benchmark. The best-performing setup — OpenAI's o1-preview with AIDE scaffolding — achieved at least the level of a Kaggle bronze medal in 16.9% of competitions.

In addition to the main results, the study investigates various forms of resource-scaling for AI agents and the impact of contamination from pre-training. These provide insights into factors affecting AI performance in machine learning engineering tasks.

To facilitate future research in understanding the ML engineering capabilities of AI agents, OpenAI has open-sourced the benchmark code.

---

[To read more articles like this, and be notified of new content published each week, Sign Up to our FREE newsletter](#/portal/signup/free)

---