Technology Benchmarking Practices

We are not talking enough about AI Agent benchmarking...comparing cost and accuracy. Current benchmarks prioritise accuracy, neglecting other critical metrics, leading to overly complex and costly state-of-the-art (SOTA) AI Agents and misconceptions about the sources of accuracy improvements. Benchmarks fail to distinguish between the needs of model developers and downstream developers, complicating the selection of agents best suited for specific applications. Many benchmarks lack sufficient holdout sets, or have none, resulting in fragile AI Agents that overfit to benchmarks by taking shortcuts. Evaluation practices lack standardisation, causing widespread issues with reproducibility. This research advocates for optimising both cost and accuracy, demonstrating through design and implementation that this approach significantly reduces costs while preserving accuracy. A structured framework is proposed to prevent overfitting, enhancing agent robustness. The outlined steps aim to foster the development of AI agents that are practical for real-world applications, moving beyond mere benchmark accuracy.

Systematic benchmarking of quantum computers: status and recommendations by EQCBC (European Quantum Computing Benchmarking Coordination Committee: https://lnkd.in/ebap-Dwt): https://lnkd.in/e24aVNXh Abstract Architectures for quantum computing can only be scaled up when they are accompanied by suitable benchmarking techniques. The document provides a comprehensive overview of the state and recommendations for systematic benchmarking of quantum computers. Benchmarking is crucial for assessing the performance of quantum computers, including the hardware, software, as well as algorithms and applications. The document highlights key aspects such as component-level, system-level, software-level, HPC-level, and application-level benchmarks. Component-level benchmarks focus on the performance of individual qubits and gates, while system-level benchmarks evaluate the entire quantum processor. Software-level benchmarks consider the compiler’s efficiency and error mitigation techniques. HPC-level and cloud benchmarks address integration with classical systems and cloud platforms, respectively. Application-level benchmarks measure performance in real-world use cases. The document also discusses the importance of standardization to ensure reproducibility and comparability of benchmarks, and highlights ongoing efforts in the quantum computing community towards establishing these benchmarks. Recommendations for future steps emphasize the need for developing standardized evaluation routines and integrating benchmarks with broader quantum technology activities.

🚀 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐢𝐧𝐠 𝐌𝐋-𝐁𝐚𝐬𝐞𝐝 𝐏𝐃𝐄 𝐒𝐨𝐥𝐯𝐞𝐫𝐬: 𝐁𝐞𝐬𝐭 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬 Last week, I shared insights from the study 𝘞𝘦𝘢𝘬 𝘉𝘢𝘴𝘦𝘭𝘪𝘯𝘦𝘴 𝘢𝘯𝘥 𝘙𝘦𝘱𝘰𝘳𝘵𝘪𝘯𝘨 𝘉𝘪𝘢𝘴𝘦𝘴 𝘓𝘦𝘢𝘥 𝘵𝘰 𝘖𝘷𝘦𝘳𝘰𝘱𝘵𝘪𝘮𝘪𝘴𝘮 𝘪𝘯 𝘔𝘢𝘤𝘩𝘪𝘯𝘦 𝘓𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘧𝘰𝘳 𝘍𝘭𝘶𝘪𝘥-𝘙𝘦𝘭𝘢𝘵𝘦𝘥 𝘗𝘢𝘳𝘵𝘪𝘢𝘭 𝘋𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘵𝘪𝘢𝘭 𝘌𝘲𝘶𝘢𝘵𝘪𝘰𝘯𝘴 by Nick McGreivy and Ammar Hakim (link in comments). The authors highlighted a crucial issue: many #ML -based #solvers aren't benchmarked against appropriate baselines, leading to misleading conclusions. ⚠️ 𝐒𝐨, 𝐰𝐡𝐚𝐭’𝐬 𝐭𝐡𝐞 𝐫𝐢𝐠𝐡𝐭 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡? 🤔 The key lies in comparing the #Cost vs. #Accuracy of #algorithms, reflecting the inherent trade-off between efficiency and precision in numerical methods. While quick, low-accuracy approximations are common, highly accurate results typically require more computational time. ⏱️ 📊 𝐀𝐜𝐜𝐮𝐫𝐚𝐜𝐲-𝐒𝐩𝐞𝐞𝐝 𝐏𝐚𝐫𝐞𝐭𝐨 𝐂𝐮𝐫𝐯𝐞𝐬: 𝐀 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐢𝐧𝐠 𝐒𝐭𝐚𝐧𝐝𝐚𝐫𝐝 From my experience, the most effective way to benchmark is by using Pareto curves of accuracy versus computational time (see the figure below). These curves offer a clear, visual comparison, showing how different methods perform under the same hardware conditions. They also mirror real-world engineering decisions, where finding a balance between speed and accuracy is critical. ⚖️ An example of this can be seen in Aditya Phopale’s master thesis, where the performance of a #NeuralNetwork-based solver was compared against the state-of-the-art general purpose #Fenics solver. 🔍 𝐂𝐡𝐨𝐨𝐬𝐢𝐧𝐠 𝐭𝐡𝐞 𝐑𝐢𝐠𝐡𝐭 𝐁𝐚𝐬𝐞𝐥𝐢𝐧𝐞 𝐒𝐨𝐥𝐯𝐞𝐫 Nick McGreivy and Ammar Hakim also emphasize the importance of selecting an appropriate baseline. While Fenics might not be the top-notch choice when it comes to computational efficiency for a specific problem (e.g., vs spectral solvers), it is still highly relevant from an #engineering perspective. Both the investigated solver and Fenics share a similar philosophy: they are general-purpose, Python-based solvers that are based on equation formulations. 🧩 Additionally, unlike #FiniteElement solvers like Fenics, the investigated Neural Network solvers don’t require complex discretization. Thus, Fenics serves as a suitable baseline for practical engineering applications, despite its "limitations" from a more theoretical context. 💡 𝐖𝐡𝐚𝐭 𝐀𝐫𝐞 𝐘𝐨𝐮𝐫 𝐁𝐞𝐬𝐭 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬? I’m curious to hear from others: what best practices do you follow when benchmarking ML-based PDE solvers? Let’s discuss! 👇

The European Commission's Joint Research Centre (JRC) published a useful paper titled 𝗔𝗜 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀: 𝗜𝗻𝘁𝗲𝗿𝗱𝗶𝘀𝗰𝗶𝗽𝗹𝗶𝗻𝗮𝗿𝘆 𝗜𝘀𝘀𝘂𝗲𝘀 𝗮𝗻𝗱 𝗣𝗼𝗹𝗶𝗰𝘆 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀. In my view, the findings of the paper suggest several important implications. Firstly, we need to improve evaluation methods and scientific practices. Secondly, we can educate policymakers, researchers, and the public to be critical of industry results. Thirdly, we should require third-party evaluators and fund the development of the AI evaluation ecosystem. Fourthly, we should ensure and check that evaluations measure what is truly important for a particular case or use. Finally, we should not let perfect be the enemy of good; improvements are necessary, but evaluating capabilities and other aspects of risks and benefits is crucial, and simply relying on vibes and anecdotes is not enough. Here's the abstract of the paper: "Artificial Intelligence (AI) benchmarks have emerged as essential for evaluating AI performance, capabilities, and risks. However, as their influence grows, concerns arise about their limitations and side effects when assessing sensitive topics such as high-impact capabilities, safety and systemic risks. In this work we summarise the results of an interdisciplinary meta-review of approximately 110 studies over the last decade (Eriksson et al., 2025), which identify key shortcomings in AI benchmarking practices, including issues in the design and application (e.g., biases, inadequate documentation, data contamination, and failures to distinguish signal from noise) and broader sociotechnical issues (e.g., over-focus on text-based and one-time evaluation logic, neglecting multimodality and interactions). We also highlight systemic flaws, such as misaligned incentives, construct validity issues, unknown unknowns, and the gaming of benchmark results. We underscore how benchmark practices are shaped by cultural, commercial and competitive dynamics that often prioritise performance at the expense of broader societal concerns. As a result, AI benchmarking may be ill-suited to provide the assurances required by policymakers. To address these challenges, it is crucial to consider key policy aspects that can help mitigate the shortcomings of current AI benchmarking practices." Read the full paper below (and a link to the paper and to a Euractiv piece by Maximilian Henning are in the comments):