World Hyper-Heuristic: A Reinforcement Learning Approach for Dynamic Exploration–Exploitation in Cloud Resource Management

Authors

  • Adel Cheshmeh * Department of Computer Engineering, Faculty of Computer Engineering, Sharif University of Technology, Tehran, Iran.

https://doi.org/10.48313/maa.v1i4.109

Abstract

Cloud computing resource management presents a formidable optimization challenge characterized by dynamic and unpredictable workloads, heterogeneous resource pools, competing multi-objective trade-offs among energy consumption, Service Level Agreement (SLA) compliance, resource utilization, makespan, and operational cost, as well as the NP-hard combinatorial nature of core sub-problems including Virtual Machine (VM) placement, task scheduling, and dynamic resource scaling. Existing approaches suffer from well-documented limitations: static heuristics such as First Fit Decreasing (FFD) and Round-Robin (RR) lack adaptability to workload fluctuations; single metaheuristics including Particle Swarm Optimization (PSO) and Genetic Algorithms (GA) exhibit premature convergence and maintain a fixed exploration–exploitation balance; and pure Reinforcement Learning (RL) methods, while adaptive, are sample-inefficient and transfer poorly across workload types. In this paper, we propose World Hyper-Heuristic (WHH) for Cloud Resource Management (World-CRM), an extended adaptation of the WHH originally introduced by Daliri et al. [1] in expert systems with applications. World-CRM adapts the World's novel RL-based dynamic exploration–exploitation switching mechanism to cloud environments through five key innovations: 1) a cloud-aware 12-dimensional RL state representation encoding CPU utilization variance, memory fragmentation, SLA violation rate, and energy metrics, 2) a composite multi-objective fitness function with adaptive weight adjustment, 3) a cloud-specific Low-Level Heuristic (LLH) pool comprising 12 operators spanning VM placement, task scheduling, and dynamic scaling, 4) an online sliding-window adaptation mechanism employing Kullback–Leibler divergence for workload shift detection and real-time policy updates, and 5) a multi-tier coordination architecture operating across infrastructure, platform, and application layers. World-CRM is evaluated on CloudSim Plus simulations with three real-world workload traces Google Cluster Trace (CCT) 2019, Azure VM Traces 2019, and bitbrains distributed data center across small (50 hosts), medium (200 hosts), and large (800 hosts) data center configurations. Results across 30 independent runs per configuration demonstrate that World-CRM achieves 23–31% energy reduction, 42–58% fewer SLA violations, 15–22% better resource utilization, and 18–27% lower makespan compared to the best-performing baselines including NSGA-II-VM, PSO-Cloud, DQN-VMPlace, Modified Best Fit Decreasing (MBFD), and the original World algorithm. The online adaptation module maintains performance under workload shifts with decision latency below 300 milliseconds and enables 3–4× faster recovery compared to RL and evolutionary baselines. All improvements are statistically significant under the Wilcoxon rank-sum test with Bonferroni correction at p < 0.05.

Keywords:

Hyper-heuristic, Cloud resource management, Reinforcement learning, Virtual machine placement, Task scheduling, Exploration–exploitation

References

  1. [1] Daliri, A., Alimoradi, M., Zabihimayvan, M., & Sadeghi, R. (2024). World hyper-heuristic: A novel reinforcement learning approach for dynamic exploration and exploitation. Expert systems with applications, 244, 122931. https://doi.org/10.1016/j.eswa.2023.122931
  2. [2] Buyya, R., Yeo, C. S., Venugopal, S., Broberg, J., & Brandic, I. (2009). Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future generation computer systems, 25(6), 599–616. https://doi.org/10.1016/j.future.2008.12.001
  3. [3] Mell, P., & Grance, T. (2011). The NIST definition of cloud computing. https://doi.org/10.6028/NIST.SP.800-145
  4. [4] Gartner. (2024). Forecast: Public cloud services, Worldwide. https://www.gartner.com/en/documents/5541595
  5. [5] Masanet, E., Shehabi, A., Lei, N., Smith, S., & Koomey, J. (2020). Recalibrating global data center energy-use estimates. Science, 367(6481), 984–986. https://doi.org/10.1126/science.aba3758
  6. [6] Beloglazov, A., & Buyya, R. (2012). Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers. Concurrency and computation: Practice and experience, 24(13), 1397–1420. https://doi.org/10.1002/cpe.1867
  7. [7] Rodriguez, M. A., & Buyya, R. (2014). Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE transactions on cloud computing, 2(2), 222–235. https://doi.org/10.1109/TCC.2014.2314655
  8. [8] Lorido-Botran, T., Miguel-Alonso, J., & Lozano, J. A. (2014). A review of auto-scaling techniques for elastic applications in cloud environments. Journal of grid computing, 12(4), 559–592. https://doi.org/10.1007/s10723-014-9314-7
  9. [9] Farahnakian, F., Ashraf, A., Pahikkala, T., Liljeberg, P., Plosila, J., Porres, I., & Tenhunen, H. (2014). Using ant colony system to consolidate VMs for green cloud computing. IEEE transactions on services computing, 8(2), 187–198. https://doi.org/10.1109/TSC.2014.2382555
  10. [10] Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., & Kozuch, M. A. (2012). Heterogeneity and dynamicity of clouds at scale: Google trace analysis. Proceedings of the third ACM symposium on cloud computing (pp. 1-13). Association for Computing Machinery. https://doi.org/10.1145/2391229.2391236
  11. [11] Cortez, E., Bonde, A., Muzio, A., Russinovich, M., Fontoura, M., & Bianchini, R. (2017). Resource central: Understanding and predicting workloads for improved resource management in large cloud platforms. Proceedings of the 26th symposium on operating systems principles (pp. 153-167). Association for Computing Machinery. https://doi.org/10.1145/3132747.3132772
  12. [12] Beloglazov, A., Abawajy, J., & Buyya, R. (2012). Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing. Future generation computer systems, 28(5), 755–768. https://doi.org/10.1016/j.future.2011.04.017
  13. [13] Zhan, Z.H., Liu, X.F., Gong, Y.J., Zhang, J., Chung, H. S. H., & Li, Y. (2015). Cloud computing resource scheduling and a survey of its evolutionary approaches. ACM computing surveys (CSUR), 47(4), 1–33. https://doi.org/10.1145/2788397
  14. [14] Mejahed, S., & Elshrkawey, M. (2022). A multi-objective algorithm for virtual machine placement in cloud environments using a hybrid of particle swarm optimization and flower pollination optimization. PeerJ computer science, 8, e834. https://doi.org/10.7717/peerj-cs.834
  15. [15] Chen, W.-N., & Zhang, J. (2008). An ant colony optimization approach to a grid workflow scheduling problem with various QoS requirements. IEEE transactions on systems, man, and cybernetics, part c (applications and reviews), 39(1), 29–43. https://doi.org/10.1109/TSMCC.2008.2001722
  16. [16] Storn, R., & Price, K. (1997). Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization, 11(4), 341–359. https://doi.org/10.1023/A:1008202821328
  17. [17] Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, 6(2), 182–197. https://doi.org/10.1109/4235.996017
  18. [18] Zhang, Q., & Li, H. (2007). MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE transactions on evolutionary computation, 11(6), 712–731. https://doi.org/10.1109/TEVC.2007.892759
  19. [19] Tuli, S., Ilager, S., Ramamohanarao, K., & Buyya, R. (2020). Dynamic scheduling for stochastic edge-cloud computing environments using a3c learning and residual recurrent neural networks. IEEE transactions on mobile computing, 21(3), 940–954. https://doi.org/10.1109/TMC.2020.3017079
  20. [20] Arabnejad, H., Pahl, C., Jamshidi, P., & Estrada, G. (2017). A comparison of reinforcement learning techniques for fuzzy cloud auto-scaling. 2017 17th IEEE/ACM international symposium on cluster, cloud and grid computing (CCGRID) (pp. 64-73). IEEE. https://doi.org/10.1109/CCGRID.2017.15
  21. [21] Mao, H., Schwarzkopf, M., Venkatakrishnan, S. B., Meng, Z., & Alizadeh, M. (2019). Learning scheduling algorithms for data processing clusters. Proceedings of the acm special interest group on data communication (pp. 270–288). Association for Computing Machinery. https://doi.org/10.1145/3341302.3342080
  22. [22] Burke, E. K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., & Qu, R. (2013). Hyper-heuristics: A survey of the state of the art. Journal of the operational research society, 64(12), 1695–1724. https://doi.org/10.1057/jors.2013.71
  23. [23] Zhang, J., Yu, H., Fan, G., Li, Z., Xu, J., & Li, J. (2024). Handling hierarchy in cloud data centers: A Hyper-Heuristic approach for resource contention and energy-aware Virtual Machine management. Expert systems with applications, 249, 123528. https://doi.org/10.1016/j.eswa.2024.123528
  24. [24] Tan, B., Ma, H., & Mei, Y. (2018). A genetic programming Hyper-Heuristic approach for online resource allocation in container-based clouds. Parallel problem solving from nature – PPSN XV (pp. 146–152). Springer. https://doi.org/10.1007/978-3-319-99253-2_13
  25. [25] Li, K., Zheng, H., & Wu, J. (2013). Migration-based virtual machine placement in cloud systems. 2013 IEEE 2nd international conference on cloud networking (CloudNet) (pp. 83-90). IEEE. https://doi.org/10.1109/CloudNet.2013.6710561
  26. [26] Ferdaus, M. H., Murshed, M., Calheiros, R. N., & Buyya, R. (2014). Virtual machine consolidation in cloud data centers using ACO metaheuristic. European conference on parallel processing (pp. 306-317). Springer. https://doi.org/10.1007/978-3-319-09873-9_26
  27. [27] Zhou, X., Zhang, G., Sun, J., Zhou, J., Wei, T., & Hu, S. (2019). Minimizing cost and makespan for workflow scheduling in cloud using fuzzy dominance sort based HEFT. Future generation computer systems, 93, 278–289. https://doi.org/10.1016/j.future.2018.10.046
  28. [28] Pham, D. T., & Castellani, M. (2015). A comparative study of the Bees algorithm as a tool for function optimisation. Cogent engineering, 2(1), 1091540. https://doi.org/10.1080/23311916.2015.1091540
  29. [29] Mangalampalli, S., Karri, G. R., & Kumar, M. (2023). Multi objective task scheduling algorithm in cloud computing using grey wolf optimization. Cluster computing, 26(6), 3803–3822. https://doi.org/10.1007/s10586-022-03786-x
  30. [30] Mirjalili, S., Mirjalili, S. M., & Lewis, A. (2014). Grey wolf optimizer. Advances in engineering software, 69, 46–61. https://doi.org/10.1016/j.advengsoft.2013.12.007
  31. [31] Mirjalili, S., & Lewis, A. (2016). The whale optimization algorithm. Advances in engineering software, 95, 51–67. https://doi.org/10.1016/j.advengsoft.2016.01.008
  32. [32] Mirjalili, S. (2016). SCA: A sine cosine algorithm for solving optimization problems. Knowledge-based systems, 96, 120–133. https://doi.org/10.1016/j.knosys.2015.12.022
  33. [33] Heidari, A. A., Mirjalili, S., Faris, H., Aljarah, I., Mafarja, M., & Chen, H. (2019). Harris Hawks optimization: Algorithm and applications. Future generation computer systems, 97, 849–872. https://doi.org/10.1016/j.future.2019.02.028
  34. [34] Wolpert, D. H., & Macready, W. G. (1997). No Free Lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1), 67–82. https://doi.org/10.1109/4235.585893
  35. [35] Cheng, M., Li, J., & Nazarian, S. (2018). DRL-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers. 2018 23rd Asia and South Pacific design automation conference (ASP-DAC) (pp. 129-134). IEEE. https://doi.org/10.1109/ASPDAC.2018.8297294
  36. [36] Peng, Y., Bao, Y., Chen, Y., Wu, C., Meng, C., & Lin, W. (2021). DL2: A deep learning-driven scheduler for deep learning clusters. IEEE transactions on parallel and distributed systems, 32(8), 1947–1960. https://doi.org/10.1109/TPDS.2021.3052895
  37. [37] Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021). Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis. Machine learning, 110(9), 2419–2468. https://doi.org/10.1007/s10994-021-05961-4
  38. [38] Cowling, P., Kendall, G., & Soubeiga, E. (2000). A hyperheuristic approach to scheduling a sales summit. International conference on the practice and theory of automated timetabling (pp. 176-190). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/3-540-44629-X_11
  39. [39] López-Camacho, E., Terashima-Marin, H., Ross, P., & Ochoa, G. (2014). A unified Hyper-Heuristic framework for solving bin packing problems. Expert systems with applications, 41(15), 6876–6889. https://doi.org/10.1016/j.eswa.2014.04.043
  40. [40] Pillay, N., & Banzhaf, W. (2009). A study of heuristic combinations for Hyper-Heuristic systems for the uncapacitated examination timetabling problem. European journal of operational research, 197(2), 482–491. https://doi.org/10.1016/j.ejor.2008.07.023
  41. [41] Sabar, N. R., Ayob, M., Kendall, G., & Qu, R. (2014). A dynamic multiarmed bandit-gene expression programming Hyper-Heuristic for combinatorial optimization problems. IEEE transactions on cybernetics, 45(2), 217–228. https://doi.org/10.1109/TCYB.2014.2323936
  42. [42] Drake, J. H., Kheiri, A., Özcan, E., & Burke, E. K. (2020). Recent advances in selection Hyper-Heuristics. European journal of operational research, 285(2), 405–428. https://doi.org/10.1016/j.ejor.2019.07.073
  43. [43] Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT Press. https://mitpress.mit.edu/9780262039246/reinforcement-learning/
  44. [44] Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine learning, 8(3), 279–292. https://doi.org/10.1007/BF00992698
  45. [45] Silva Filho, M. C., Oliveira, R. L., Monteiro, C. C., Inácio, P. R., & Freire, M. M. (2017). CloudSim Plus: A cloud computing simulation framework pursuing software engineering principles for improved modularity, extensibility and correctness. 2017 IFIP/IEEE symposium on integrated network and service management (IM) (pp. 400-406). IEEE. https://doi.org/10.23919/INM.2017.7987304
  46. [46] Calheiros, R. N., Ranjan, R., Beloglazov, A., De Rose, C. A. F., & Buyya, R. (2011). CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and experience, 41(1), 23–50. https://doi.org/10.1002/spe.995
  47. [47] Tirmazi, M., Barker, A., Deng, N., Haque, M. E., Qin, Z. G., Hand, S., ... & Wilkes, J. (2020). Borg: The next generation. Proceedings of the fifteenth European conference on computer systems (pp. 1-14). Association for Computing Machinery. https://doi.org/10.1145/3342195.3387517
  48. [48] Shen, S., Van Beek, V., & Iosup, A. (2015). Statistical characterization of business-critical workloads hosted in cloud datacenters. 2015 15th IEEE/ACM international symposium on cluster, cloud and grid computing (pp. 465-474). IEEE. https://doi.org/10.1109/CCGrid.2015.60

Published

2024-12-24

Issue

Vol. 1 No. 4 (2024): Metaheuristic Algorithms with Applications (MAA)

Section

Articles

How to Cite

Cheshmeh, A. . (2024). World Hyper-Heuristic: A Reinforcement Learning Approach for Dynamic Exploration–Exploitation in Cloud Resource Management. Metaheuristic Algorithms With Applications, 1(4), 440-464. https://doi.org/10.48313/maa.v1i4.109

Similar Articles

41-50 of 62

You may also start an advanced similarity search for this article.