```html

The Challenge of AI Benchmarking in Software Engineering

Artificial intelligence (AI) is revolutionizing many areas, including software engineering. The development and application of AI models for tasks such as code generation, bug fixing, and testing is growing rapidly. To make progress in this field measurable and comparable, benchmarks are essential. They provide a standardized environment for evaluating AI models and enable the reproducibility of results. However, the rapid increase in AI models in software engineering also brings challenges for benchmarking.

Fragmented Landscape and Lack of Standards

A central challenge is the fragmented landscape of existing benchmarks. Knowledge about benchmarks is often scattered across different task areas, making it difficult to select suitable benchmarks for specific use cases. Furthermore, there is a lack of a uniform standard for the development and documentation of benchmarks. This leads to inconsistencies and makes it difficult to compare results from different studies. In addition, there are limitations of existing benchmarks, which can be due to their size, representativeness, or actuality.

BenchScout: A Semantic Search Tool

To facilitate the search for relevant benchmarks, BenchScout was developed. This semantic search tool uses automated clustering of context information from related studies to find benchmarks based on the needs of the users. A user study with 22 participants confirmed the usability, effectiveness, and intuitiveness of BenchScout with average ratings of 4.5, 4.0, and 4.1 out of 5 points, respectively.

BenchFrame: A Protocol for Improving Benchmarks

To improve the quality of benchmarks, BenchFrame was developed, a method for standardizing and extending benchmarks. As a case study, BenchFrame was applied to the HumanEval benchmark to address its limitations. The result is HumanEvalNext, an improved version of the benchmark with corrected errors, optimized language conversion, extended test coverage, and increased difficulty.

HumanEvalNext: A More Challenging Benchmark

The evaluation of ten state-of-the-art code language models on HumanEval, HumanEvalPlus, and HumanEvalNext showed that the models performed significantly worse on HumanEvalNext. Compared to HumanEval and HumanEvalPlus, the Pass@1 rate decreased by 31.22% and 19.94%, respectively. This highlights the increased difficulty of HumanEvalNext and underscores the importance of continuous improvement of benchmarks.

Outlook: The Future of AI Benchmarking

The development and application of AI in software engineering continues to advance. Therefore, it is essential to continue the development and standardization of benchmarks to ensure the comparability and reproducibility of research results. Initiatives like BenchScout and BenchFrame make an important contribution to improving benchmarking practices and help to promote the development of robust and reliable AI models for software engineering. The continuous adaptation and extension of benchmarks to the state of the art is crucial to objectively assess and promote progress in this dynamic field.

Bibliography: - https://arxiv.org/abs/2503.05860 - https://www.arxiv.org/pdf/2503.05860 - http://paperreading.club/page?id=290243 - https://huggingface.co/papers?q=BIRD%20benchmark - https://www.researchgate.net/publication/383455349_The_Integration_and_Impact_of_Artificial_Intelligence_in_Software_Engineering - https://www.mdpi.com/2076-3417/15/3/1344 - https://github.com/dair-ai/ML-Papers-of-the-Week - https://www.researchgate.net/publication/381964876_Benchmarking_Generative_AI_A_Call_for_Establishing_a_Comprehensive_Framework_and_a_Generative_AIQ_Test - https://www.sciencedirect.com/science/article/pii/S0268401221000761 - https://dl.acm.org/doi/10.1145/3487043 ```