EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model

Kailash Mannepalli

Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023 · 01.04.2024 · 4 (1) · 588-598

DOI: 10.60087/jaigs.v4i1.513 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, online erschienen am 01.04.2024. Verfasst von Kailash Mannepalli. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Banks, credit unions and insurers choosing between a managed large language model (LLM) endpoint and self-hosted open-weight models usually start with cost per million tokens. That ordering is wrong for regulated workloads: whether prompt data may leave the institution's boundary is a pass/fail constraint, and cost comparisons that assume full GPU utilization overstate the case for self-hosting. We propose a decision rubric with a mandatory data-boundary gate, followed by weighted criteria for volume, capability, fine-tuning need and latency, and a cost model expressed in sustained utilization and fixed operating overhead. We measure goodput, the highest request rate that meets a latency objective, for Llama-3.1-8B-Instruct in bf16 on one NVIDIA H100-80GB and for Llama-3.1-70B-Instruct in FP8 and bf16 on two and four H100s, using vLLM under open-loop Poisson load. Before fixed overhead, the 8B configuration breaks even with a per-token API at 40% sustained utilization, the 70B FP8 configuration at 57%, and the 70B bf16 configuration at 87%. With a fixed operating overhead of $15,000 per month, the 8B configuration needs 14 replicas at 60% utilization to break even, and the 70B bf16 configuration does not break even below 87% utilization at any replica count. We recommend that regulated institutions apply the data-boundary gate first, price self-hosting at measured goodput and forecast utilization rather than peak throughput, and treat quantization as a cost lever that can move a configuration across the break-even line.

Zitieren

APA

Mannepalli, K. (2024). Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model. Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023, 4(1), 588-598. https://doi.org/10.60087/jaigs.v4i1.513

Vancouver

Mannepalli K. Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model. Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023. 2024;4(1):588-598. doi:10.60087/jaigs.v4i1.513

BibTeX
@article{Mannepalli2024,
  author = {Mannepalli, Kailash},
  title = {Managed Versus Self-Hosted Large Language Model Serving in Regulated Industries: A Data-Boundary-Gated Decision Rubric with a Utilization-Sensitive Cost Model},
  journal = {Journal of Artificial Intelligence General science (JAIGS) ISSN:3006-4023},
  year = {2024},
  volume = {4},
  number = {1},
  pages = {588--598},
  doi = {10.60087/jaigs.v4i1.513}
}
Kostenlos starten →