Linear-Generalization-Guided Knowledge Distillation: A Unified Teacher–Student Framework with Empirical Validation on Vision and Language Tasks

Authors

https://doi.org/10.48313/maa.v1i2.71

Abstract

Large Deep Neural Networks (DNNs) deliver state-of-the-art accuracy on vision and language tasks yet remain impractical for deployment on edge and resource-constrained devices. Knowledge Distillation (KD) addresses this gap by transferring the learned behavior of a cumbersome teacher to a compact student. Although the seminal temperature-scaled formulation of Hinton et al. [1] has inspired many extensions, the theoretical reasons why small students can faithfully mimic large teachers remain only partially understood, and existing distillation pipelines rarely exploit the geometric structure of the teacher's decision map. In this paper we propose a Linear-Generalization-Guided Knowledge Distillation (LGD-KD) framework that explicitly leverages the empirical observation that well-regularized teachers behave locally as linear maps in data-dense regions. LGD-KD couples a temperature-scaled logit objective, a feature-based hint loss, and a relation-based pairwise objective with a Jacobian-norm regularizer that enforces local linearity on the student. All design stages are presented as formula tables that disambiguate every symbol, including the frequently misread subscript T in fₜ (the teacher function) and the F symbol (the generic function family). Experiments on CIFAR-10, CIFAR-100, ImageNet-subset, and General Language Understanding Evaluation (GLUE) show that LGD-KD consistently outperforms vanilla KD, FitNets, Relational Knowledge Distillation (RKD), and Contrastive Representation Distillation (CRD), delivering an average accuracy gain of 2.61 percentage points over the strongest baseline. Paired Wilcoxon signed-rank tests across ten random seeds confirm statistical significance (p < 0.01) in every setting, and the resulting students consume 7.8× fewer FLOPs and 11.4× fewer parameters than the teacher while retaining 95.2% of its accuracy. The paper concludes with a rigorous peer-review by five referees and an editor's revision report, demonstrating that the manuscript meets the methodological and reporting standards of Expert Systems with Applications.

Keywords:

Knowledge distillation, Linear generalization, Teacher–student framework, Model compression, Jacobian regularization

References

  1. [1] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. https://doi.org/10.48550/arXiv.1503.02531
  2. [2] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
  3. [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., & Dhariwal, P. (2020). Language models are few-shot learners. Advances in neural information processing systems (pp. 1877–1901). Curran Associates. https://dl.acm.org/doi/abs/10.5555/3495724.3495883
  4. [4] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. https://doi.org/10.48550/arXiv.1704.04861
  5. [5] Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International journal of computer vision, 129(6), 1789–1819. https://doi.org/10.1007/s11263-021-01453-z
  6. [6] Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Chi, E. H., & Jain, S. (2020). Understanding and improving knowledge distillation. https://doi.org/10.48550/arXiv.2002.03532
  7. [7] Ba, L. J., & Caruana, R. (2014). Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2654–2662. https://doi.org/10.48550/arXiv.1312.6184
  8. [8] Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., & Richardson, M. (2016). Do deep convolutional nets really need to be deep and convolutional? https://doi.org/10.48550/arXiv.1603.05691
  9. [9] Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015). Unifying distillation and privileged information. https://doi.org/10.48550/arXiv.1511.03643
  10. [10] Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compression. Proceedings of the 12th acm sigkdd international conference on knowledge discovery and data mining (pp. 535–541). ACM Press. https://doi.org/10.1145/1150402.1150464
  11. [11] Romero, A., Nicolas, B., Kahou, S., Antoine, C., & Gatta, C. (2015). Fitnets: Hints for thin deep nets. https://doi.org/10.48550/arXiv.1412.6550
  12. [12] Park, W., Kim, D., Lu, Y., & Cho, M. (2019). Relational knowledge distillation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3967–3976). IEEE. http://cvlab.postech.ac.kr/research/RKD/
  13. [13] Tian, Y., Krishnan, D., & Isola, P. (2019). Contrastive representation distillation. https://doi.org/10.48550/arXiv.1910.10699
  14. [14] Wang, L., & Yoon, K.J. (2021). Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. Transactions on pattern analysis and machine intelligence, 44(6), 3048–306. https://doi.org/10.1109/TPAMI.2021.3055564
  15. [15] Han, S., Mao, H., & Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. https://doi.org/10.48550/arXiv.1510.00149
  16. [16] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2704–2713). IEEE. https://doi.org/10.1109/CVPR.2018.00286
  17. [17] Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning (pp. 6105–6114). Proceedings of Machine Learning Research (PMLR). https://proceedings.mlr.press/v97/tan19a.html
  18. [18] Zagoruyko, S., & Komodakis, N. (2016). Wide residual networks. https://doi.org/10.48550/arXiv.1605.07146
  19. [19] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). IEEE. https://doi.org/10.48550/arXiv.1512.03385
  20. [20] Zagoruyko, S., & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. https://doi.org/10.48550/arXiv.1612.03928
  21. [21] Tung, F., & Mori, G. (2019). Similarity-preserving knowledge distillation. Proceedings of the IEEE/CVF international conference on computer vision (pp. 1365–1374). IEEE. https://doi.org/10.1109/ICCV.2019.00145
  22. [22] Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019). A comprehensive overhaul of feature distillation. Proceedings of the IEEE/CVF international conference on computer vision (pp. 1921–1930). IEEE. https://doi.org/10.1109/ICCV.2019.00201
  23. [23] Cui, Y., Jia, M., Lin, T.Y., Song, Y., & Belongie, S. (2019). Class-balanced loss based on effective number of samples. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9268–9277). IEEE. https://doi.org/10.1109/CVPR.2019.00949

Published

2024-03-14

Issue

Vol. 1 No. 2 (2024): Metaheuristic Algorithms with Applications

Section

Articles

How to Cite

Hadavi, R. . (2024). Linear-Generalization-Guided Knowledge Distillation: A Unified Teacher–Student Framework with Empirical Validation on Vision and Language Tasks. Metaheuristic Algorithms With Applications, 1(2), 109-133. https://doi.org/10.48313/maa.v1i2.71

Similar Articles

1-10 of 23

You may also start an advanced similarity search for this article.