Screening tasks that aim to identify a small subset of top alternatives from a large pool are common in business decision-making processes. These tasks often require substantial human effort to evaluate each alternative's performance, making them time-consuming and costly. Motivated by recent advances in large language models (LLMs), particularly their ability to generate outputs that align well with human evaluations, we consider an LLM-as-human-evaluator approach for conducting screening virtually, thereby reducing the cost burden. To achieve scalability and cost-effectiveness in virtual screening, we identify that the stochastic nature of LLM outputs and their cost structure necessitate efficient budget allocation across all alternatives. To address this, we propose using a top-m greedy evaluation mechanism, a simple yet effective approach that keeps evaluating the current top-m alternatives, and design the explore-first top-m greedy (EFG-m) algorithm. We prove that EFG-m is both sample-optimal and consistent in large-scale virtual screening. Surprisingly, we also uncover a bonus ranking effect, where the algorithm naturally induces an indifference-based ranking within the selected subset. To further enhance practicality, we design a suite of algorithm variants to improve screening performance and computational efficiency. Numerical experiments validate our results and demonstrate the effectiveness of our algorithms. Lastly, we conduct a case study on LLM-based virtual screening. The study shows that while LLMs alone may not provide meaningful screening and ranking results when directly queried, integrating them with our sample-optimal algorithms unlocks their potential for cost-effective, large-scale virtual screening.

Zaile Li is a Postdoctoral Researcher at the Healthcare Management Initiative of INSEAD. He holds a Ph.D. in Management Science and Engineering from the School of Management, Fudan University. His research focuses on data-driven selection decisions, simulation optimization, sequential learning, and generative AI, with applications in healthcare management. Zaile’s work has received several honors, including finalist in the INFORMS George Nicholson Student Paper Competition and First Prize in the 14th POMS-HK Best Student Paper Competition.