Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model
Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 · 01.04.2024 · 4 (1) · 588-598
DOI: 10.60087/jaigs.v4i1.513 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, online erschienen am 01.04.2024. Verfasst von Kailash Mannepalli. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Banks, credit unions and insurers choosing between a managed large language model (LLM) endpoint and self-hosted open-weight models usually start with cost per million tokens. That ordering is wrong for regulated workloads: whether prompt data may leave the institution's boundary is a pass/fail constraint, and cost comparisons that assume full GPU utilization overstate the case for self-hosting. We propose a decision rubric with a mandatory data-boundary gate, followed by weighted criteria for volume, capability, fine-tuning need and latency, and a cost model expressed in sustained utilization and fixed operating overhead. We measure goodput, the highest request rate that meets a latency objective, for Llama-3.1-8B-Instruct in bf16 on one NVIDIA H100-80GB and for Llama-3.1-70B-Instruct in FP8 and bf16 on two and four H100s, using vLLM under open-loop Poisson load. Before fixed overhead, the 8B configuration breaks even with a per-token API at 40% sustained utilization, the 70B FP8 configuration at 57%, and the 70B bf16 configuration at 87%. With a fixed operating overhead of $15,000 per month, the 8B configuration needs 14 replicas at 60% utilization to break even, and the 70B bf16 configuration does not break even below 87% utilization at any replica count. We recommend that regulated institutions apply the data-boundary gate first, price self-hosting at measured goodput and forecast utilization rather than peak throughput, and treat quantization as a cost lever that can move a configuration across the break-even line.
Zitieren
APA
Mannepalli, K. (2024). Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model. Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, 4(1), 588-598. https://doi.org/10.60087/jaigs.v4i1.513
Vancouver
Mannepalli K. Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model. Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023. 2024;4(1):588-598. doi:10.60087/jaigs.v4i1.513
BibTeX
@article{Mannepalli2024,
author = {Mannepalli, Kailash},
title = {Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model},
journal = {Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023},
year = {2024},
volume = {4},
number = {1},
pages = {588--598},
doi = {10.60087/jaigs.v4i1.513}
}
EUVIMED