EUVIMEDEUVIMED entdecken
Crossref

Skewness-Aware Stream Partitioning: A Key Splitting Suppression Method for Distributed Stream Processing

Dawei Sun, Weilong Lv, Shang Gao, Keqin Li, Rajkumar Buyya

IEEE Transactions on Services Computing · 2026-07 · 19 (4) · 3059-3071

DOI: 10.1109/tsc.2026.3707515 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Services Computing, 2026-07. Verfasst von Dawei Sun und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 40 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Partitioning algorithms are crucial in distributed stream computing, as they directly affect load balancing across downstream task instances, cluster performance utilization, and data stream processing efficiency. However, existing algorithms suffer from key limitations: they fail to effectively balance load imbalance and key-splitting overhead as excessive splitting increases aggregation costs and memory usage, while insufficient splitting fails to reduce skew. In addition, they cannot timely identify newly emerging high-frequency keys, because historical frequency records dominate detection, which worsens load imbalance over time. To address these limitations, we propose Sa-Stream, a skewness-aware partitioning framework that rapidly captures changes in stream skewness and dynamically adjusts partitioning decisions, thereby balancing load imbalance and key-splitting overhead under evolving stream distributions. (1) A key frequency decay mechanism progressively reduces the influence of historical frequency statistics to improve the prediction of evolving key frequencies. (2) An adaptive threshold modulation method dynamically adjusts the hot-key threshold based on the current key frequency distribution, balancing load imbalance and aggregation costs. (3) A multi-tiered key segregation mechanism classifies keys into cold, warm, and hot categories: cold keys use deterministic hash routing to minimize aggregation costs, hot keys use degree-adjusted key splitting to reduce load imbalance, and warm keys use one-time key splitting to bridge the two extremes. Experiments conducted on Apache Storm demonstrate that Sa-Stream outperforms existing stream partitioning approaches, increasing throughput by up to 16% and reducing latency by up to 50%. Additionally, key splitting is reduced by up to 18%, and load imbalance is decreased by up to 98%.

Förderung

  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities

Zitieren

APA

Sun, D., Lv, W., Gao, S., Li, K., & Buyya, R. (2026). Skewness-Aware Stream Partitioning: A Key Splitting Suppression Method for Distributed Stream Processing. IEEE Transactions on Services Computing, 19(4), 3059-3071. https://doi.org/10.1109/tsc.2026.3707515

Vancouver

Sun D, Lv W, Gao S, Li K, Buyya R. Skewness-Aware Stream Partitioning: A Key Splitting Suppression Method for Distributed Stream Processing. IEEE Transactions on Services Computing. 2026;19(4):3059-3071. doi:10.1109/tsc.2026.3707515

BibTeX
@article{Sun2026,
  author = {Sun, Dawei and Lv, Weilong and Gao, Shang and Li, Keqin and Buyya, Rajkumar},
  title = {Skewness-Aware Stream Partitioning: A Key Splitting Suppression Method for Distributed Stream Processing},
  journal = {IEEE Transactions on Services Computing},
  year = {2026},
  volume = {19},
  number = {4},
  pages = {3059--3071},
  doi = {10.1109/tsc.2026.3707515}
}
Kostenlos starten →