EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

PPFedIT : Towards Privacy-preserving Federated Instruction Tuning with Few-shot Local Examples

Zhuo Zhang, Jingyuan Zhang, Jintao Huang, Hui Wang, Yue Yu, Hongzhi Zhang, Xun Zhou, Lizhen Qu, Zenglin Xu

ACM Transactions on Intelligent Systems and Technology · 31.10.2026 · 17 (5) · 1-23

DOI: 10.1145/3806196 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 23.07.2026. Verfasst von Zhuo Zhang und 8 weiteren Autor:innen. Die Arbeit nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Guangdong Basic and Applied Basic Research Foundation, Key Technologies R&D Program of Guangdong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Instruction tuning has been identified as a crucial technique for optimizing large language models (LLMs) to generate human-aligned responses. Nonetheless, gathering diversified and superior-quality instruction data for such tuning presents notable obstacles, especially in privacy-sensitive domains. Federated instruction tuning ( FedIT ) has emerged as a promising solution by consolidating collaborative training across multiple data owners, resulting in a privacy-enhancing learning model. Existing FedIT studies assume that clients have sufficient training data, however, in reality, many clients only have few-shot samples, leading to either overfitting in federated LLM or degraded performance. At the same time, this federated few-shot environment also increases the risk of training data extraction attacks, as the LLM may well memorize the limited training data. To address these issues, this article proposes a novel federated algorithm, PPFedIT , designed to enhance privacy protection and model performance of federated few-shot learning. PPFedIT comprises three vital steps on the client side: (1) synthetic data generation, which utilizes the strong generation capacity of LLMs to generate synthetic data, aiming to diversify and enrich local data; (2) parameter isolation training, which respectively updates the parameters of a shared global LLM on the synthetic data and the parameters of local LLMs on the local data, consequently mitigating the noise impact of the synthetic data; (3) local aggregation then sharing mechanism, which mixes the parameters of the global LLM and those of a local LLM first, before uploading them to a server for aggregation. This effectively mitigates data extraction attacks. Extensive experiments on three open source datasets demonstrate PPFedIT significantly enhances model performance (averaging 8.4%) and reduces the risk of data extraction attacks (approximately 20%) in practical and challenging federated few-shot scenarios.

Förderung

  • National Natural Science Foundation of China
  • Guangdong Basic and Applied Basic Research Foundation
  • Key Technologies R&D Program of Guangdong Province
  • Shenzhen Science and Technology Programs

Zitieren

APA

Zhang, Z., Zhang, J., Huang, J., Wang, H., Yu, Y., Zhang, H., Zhou, X., Qu, L., & Xu, Z. (2026). PPFedIT : Towards Privacy-preserving Federated Instruction Tuning with Few-shot Local Examples. ACM Transactions on Intelligent Systems and Technology, 17(5), 1-23. https://doi.org/10.1145/3806196

Vancouver

Zhang Z, Zhang J, Huang J, Wang H, Yu Y, Zhang H, et al. PPFedIT : Towards Privacy-preserving Federated Instruction Tuning with Few-shot Local Examples. ACM Transactions on Intelligent Systems and Technology. 2026;17(5):1-23. doi:10.1145/3806196

BibTeX
@article{Zhang2026,
  author = {Zhang, Zhuo and Zhang, Jingyuan and Huang, Jintao and Wang, Hui and Yu, Yue and Zhang, Hongzhi and Zhou, Xun and Qu, Lizhen and Xu, Zenglin},
  title = {PPFedIT : Towards Privacy-preserving Federated Instruction Tuning with Few-shot Local Examples},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2026},
  volume = {17},
  number = {5},
  pages = {1--23},
  doi = {10.1145/3806196}
}
Kostenlos starten →