EUVIMEDScopri EUVIMED
StudienprotokollOpen access · CC BY 4.0

Protocol Reverse Engineering via Deep Transfer Learning

Yanyang Zhao, Zhengxiong Luo, Wenlong Zhang, Feifan Wu, Yuanliang Chen, Fuchen Ma, Qi Xu, Heyuan Shi, Yu Jiang

Proceedings of the ACM on Software Engineering · 2026-06-30 · 3 (FSE) · 636-656

DOI: 10.1145/3797124 ↗

At a glance

Studienprotokoll in Proceedings of the ACM on Software Engineering, published online on 2026-06-30. Written by Yanyang Zhao and 8 co-authors. 54 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Natural Science Foundation of China, National Key Research and Development Project.

AbstractDati bibliograficiCiteRelated studies

Abstract (original language)

Protocol reverse engineering infers the specification of proprietary or poorly documented protocols and serves as the foundation for security analysis such as fuzz testing. While many existing techniques achieve this by mining statistical features from network traces, they face increasing challenges due to incomplete field pattern information available in the traces. Although protocol development has accumulated rich prior knowledge about protocol design, this knowledge remains largely untapped in protocol reverse engineering. This paper introduces TransRE, a protocol reverse engineering tool that leverages prior syntax knowledge from standardized protocols through deep transfer learning to better understand proprietary protocols. TransRE first selects optimal source domains by analyzing inter-domain differences between the existing knowledge base and the target protocol. It then employs a neural network to extract representation features and applies domain adaptation techniques to optimize the syntax transfer model, enabling accurate inference of protocol formats. Our evaluation on 12 widely used protocols shows that TransRE identifies fields with a perfection score of 0.43, which is 1.48×-3.07× the performance achieved by five state-of-the-art methods. Furthermore, to demonstrate practical applicability, we enhanced an existing protocol fuzzer with TransRE for testing proprietary protocols in real-world network cameras and discovered four bugs.

Funding

  • National Natural Science Foundation of China
  • National Key Research and Development Project

Cite

APA

Zhao, Y., Luo, Z., Zhang, W., Wu, F., Chen, Y., Ma, F., Xu, Q., Shi, H., & Jiang, Y. (2026). Protocol Reverse Engineering via Deep Transfer Learning. Proceedings of the ACM on Software Engineering, 3(FSE), 636-656. https://doi.org/10.1145/3797124

Vancouver

Zhao Y, Luo Z, Zhang W, Wu F, Chen Y, Ma F, et al. Protocol Reverse Engineering via Deep Transfer Learning. Proceedings of the ACM on Software Engineering. 2026;3(FSE):636-656. doi:10.1145/3797124

BibTeX
@article{Zhao2026,
  author = {Zhao, Yanyang and Luo, Zhengxiong and Zhang, Wenlong and Wu, Feifan and Chen, Yuanliang and Ma, Fuchen and Xu, Qi and Shi, Heyuan and Jiang, Yu},
  title = {Protocol Reverse Engineering via Deep Transfer Learning},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2026},
  volume = {3},
  number = {FSE},
  pages = {636--656},
  doi = {10.1145/3797124}
}
Start for free →