Linear-Generalization-Guided Knowledge Distillation: A Unified Teacher–Student Framework with Empirical Validation on Vision and Language Tasks
Downloads
Abstract
Large Deep Neural Networks (DNNs) deliver state-of-the-art accuracy on vision and language tasks yet remain impractical for deployment on edge and resource-constrained devices. Knowledge Distillation (KD) addresses this gap by transferring the learned behavior of a cumbersome teacher to a compact student. Although the seminal temperature-scaled formulation of Hinton et al. [1] has inspired many extensions, the theoretical reasons why small students can faithfully mimic large teachers remain only partially understood, and existing distillation pipelines rarely exploit the geometric structure of the teacher's decision map. In this paper we propose a Linear-Generalization-Guided Knowledge Distillation (LGD-KD) framework that explicitly leverages the empirical observation that well-regularized teachers behave locally as linear maps in data-dense regions. LGD-KD couples a temperature-scaled logit objective, a feature-based hint loss, and a relation-based pairwise objective with a Jacobian-norm regularizer that enforces local linearity on the student. All design stages are presented as formula tables that disambiguate every symbol, including the frequently misread subscript T in fₜ (the teacher function) and the F symbol (the generic function family). Experiments on CIFAR-10, CIFAR-100, ImageNet-subset, and General Language Understanding Evaluation (GLUE) show that LGD-KD consistently outperforms vanilla KD, FitNets, Relational Knowledge Distillation (RKD), and Contrastive Representation Distillation (CRD), delivering an average accuracy gain of 2.61 percentage points over the strongest baseline. Paired Wilcoxon signed-rank tests across ten random seeds confirm statistical significance (p < 0.01) in every setting, and the resulting students consume 7.8× fewer FLOPs and 11.4× fewer parameters than the teacher while retaining 95.2% of its accuracy. The paper concludes with a rigorous peer-review by five referees and an editor's revision report, demonstrating that the manuscript meets the methodological and reporting standards of Expert Systems with Applications.
Keywords
Knowledge distillation Linear generalization Teacher–student framework Model compression Jacobian regularizationReferences (23)
- [1] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. https://doi.org/10.48550/arXiv.1503.02531
- [2] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
- [3] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., & Dhariwal, P. (2020). Language models are few-shot learners. Advances in neural information processing systems (pp. 1877–1901). Curran Associates. https://dl.acm.org/doi/abs/10.5555/3495724.3495883
- [4] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. https://doi.org/10.48550/arXiv.1704.04861
- [5] Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021). Knowledge distillation: A survey. International journal of computer vision, 129(6), 1789–1819. https://doi.org/10.1007/s11263-021-01453-z
- [6] Tang, J., Shivanna, R., Zhao, Z., Lin, D., Singh, A., Chi, E. H., & Jain, S. (2020). Understanding and improving knowledge distillation. https://doi.org/10.48550/arXiv.2002.03532
- [7] Ba, L. J., & Caruana, R. (2014). Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2654–2662. https://doi.org/10.48550/arXiv.1312.6184
- [8] Urban, G., Geras, K. J., Kahou, S. E., Aslan, O., Wang, S., Caruana, R., & Richardson, M. (2016). Do deep convolutional nets really need to be deep and convolutional? https://doi.org/10.48550/arXiv.1603.05691
- [9] Lopez-Paz, D., Bottou, L., Schölkopf, B., & Vapnik, V. (2015). Unifying distillation and privileged information. https://doi.org/10.48550/arXiv.1511.03643
- [10] Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compression. Proceedings of the 12th acm sigkdd international conference on knowledge discovery and data mining (pp. 535–541). ACM Press. https://doi.org/10.1145/1150402.1150464
- [11] Romero, A., Nicolas, B., Kahou, S., Antoine, C., & Gatta, C. (2015). Fitnets: Hints for thin deep nets. https://doi.org/10.48550/arXiv.1412.6550
- [12] Park, W., Kim, D., Lu, Y., & Cho, M. (2019). Relational knowledge distillation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 3967–3976). IEEE. http://cvlab.postech.ac.kr/research/RKD/
- [13] Tian, Y., Krishnan, D., & Isola, P. (2019). Contrastive representation distillation. https://doi.org/10.48550/arXiv.1910.10699
- [14] Wang, L., & Yoon, K.J. (2021). Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. Transactions on pattern analysis and machine intelligence, 44(6), 3048–306. https://doi.org/10.1109/TPAMI.2021.3055564
- [15] Han, S., Mao, H., & Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. https://doi.org/10.48550/arXiv.1510.00149
- [16] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2704–2713). IEEE. https://doi.org/10.1109/CVPR.2018.00286
- [17] Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning (pp. 6105–6114). Proceedings of Machine Learning Research (PMLR). https://proceedings.mlr.press/v97/tan19a.html
- [18] Zagoruyko, S., & Komodakis, N. (2016). Wide residual networks. https://doi.org/10.48550/arXiv.1605.07146
- [19] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). IEEE. https://doi.org/10.48550/arXiv.1512.03385
- [20] Zagoruyko, S., & Komodakis, N. (2017). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. https://doi.org/10.48550/arXiv.1612.03928
- [21] Tung, F., & Mori, G. (2019). Similarity-preserving knowledge distillation. Proceedings of the IEEE/CVF international conference on computer vision (pp. 1365–1374). IEEE. https://doi.org/10.1109/ICCV.2019.00145
- [22] Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., & Choi, J. Y. (2019). A comprehensive overhaul of feature distillation. Proceedings of the IEEE/CVF international conference on computer vision (pp. 1921–1930). IEEE. https://doi.org/10.1109/ICCV.2019.00201
- [23] Cui, Y., Jia, M., Lin, T.Y., Song, Y., & Belongie, S. (2019). Class-balanced loss based on effective number of samples. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9268–9277). IEEE. https://doi.org/10.1109/CVPR.2019.00949