EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Learning When to Reason: Gating LLM Inference for Cost-Efficient Serverless Function Scheduling at Scale

Megan Sullivan, Boyang He, Patrick Evans

Academic Journal of Applied Sciences · 09.06.2026 · 2 (1) · 39-45

DOI: 10.54097/gwmv0761 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Academic Journal of Applied Sciences, online erschienen am 09.06.2026. Verfasst von Megan Sullivan und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Serverless computing platforms face a fundamental tension when integrating large language model based scheduling agents: invoking full chain-of-thought reasoning for every function placement decision is computationally prohibitive, yet relying exclusively on lightweight heuristics cannot capture complex workload semantics. This paper presents GateLLM, a learned gating framework that selectively routes incoming serverless scheduling requests to either a lightweight fast-path handler or a deliberative large language model reasoning pipeline based on a compact feature vector extracted from request metadata. The gating classifier is a multilayer perceptron trained through a combination of offline oracle labeling and online reinforcement feedback derived from observed scheduling outcomes, enabling continuous adaptation to workload distribution shifts. Evaluated on a large-scale production function trace, GateLLM reduces total large language model inference cost by 61.3% relative to a full-reasoning baseline while incurring only a 2.1% degradation in average job completion time and a 0.25 percentage point increase in service level objective violation rate. Analysis demonstrates that over 68% of real-world scheduling events are structurally simple and resolvable without large language model involvement, establishing inference gating as an essential primitive for operationally viable intelligent schedulers in production cloud environments.

Zitieren

APA

Sullivan, M., He, B., & Evans, P. (2026). Learning When to Reason: Gating LLM Inference for Cost-Efficient Serverless Function Scheduling at Scale. Academic Journal of Applied Sciences, 2(1), 39-45. https://doi.org/10.54097/gwmv0761

Vancouver

Sullivan M, He B, Evans P. Learning When to Reason: Gating LLM Inference for Cost-Efficient Serverless Function Scheduling at Scale. Academic Journal of Applied Sciences. 2026;2(1):39-45. doi:10.54097/gwmv0761

BibTeX
@article{Sullivan2026,
  author = {Sullivan, Megan and He, Boyang and Evans, Patrick},
  title = {Learning When to Reason: Gating LLM Inference for Cost-Efficient Serverless Function Scheduling at Scale},
  journal = {Academic Journal of Applied Sciences},
  year = {2026},
  volume = {2},
  number = {1},
  pages = {39--45},
  doi = {10.54097/gwmv0761}
}
Kostenlos starten →