The HPO problem is formulated as a black-box global optimization task. The objective is to find the hyperparameter vector that minimizes the validation loss:

Θ * = arg min θ ∈ Θ ℒ ( � � val, (1) subject to: w *( θ ) = arg min w ℒ (�� train; w, θ), (2)

where θ represents the hyperparameter vector, Θ is the constrained search space, ℒ is the loss function, train and val are training and validation datasets respectively, and w*(θ) denotes the optimal model weights obtained by training with hyperparameters θ. The objective function is expensive to evaluate (each evaluation requires full model training), noisy (due to stochastic training), and generally non-convex [4].

I. GS: computational complexity O(mn) is prohibitive. For 5 hyperparameters with 10 values each, GS requires 100,000 evaluations. It also misses inter-grid optima.

II. RS: Bergstra and Bengio [6] showed RS is more efficient when some hyperparameters are more important than others, but it still lacks directed search.

III. BO: effective for low-dimensional spaces (<20 dimensions) but scales poorly. Surrogate model updates become expensive. Struggles with categorical and conditional hyperparameters [7].

GA is inspired by natural selection and genetics [9]. The key components of GA are as follows:

I. Encoding: hyperparameters encoded as chromosomes (binary or real-valued).

II. Selection: tournament or roulette-wheel selection based on fitness (validation performance).

III. Crossover: single-point, multi-point, or uniform crossover to combine parent solutions.

IV. Mutation: random perturbation of individual genes to maintain diversity.

V. Population: typically 20–100 individuals, evolved over 50–200 generations.

GA excels at global exploration through its population-based search and genetic operators but can be slow to converge to precise optima due to its discrete crossover operations [10].

Particle Swarm Optimization (PSO) simulates the social behavior of bird flocking or fish schooling [11]. Each particle i updates its velocity and position according to:

V i(t+1) =w·v i(t) +c 1·r 1·(pbest i−x i(t)) +c 2·r 2·(gbest−xi(t)), (3)

x i(t+1) =x i(t) +v i(t+1), (4)

where w is the inertia weight (typically 0.4–0.9), c1 and c2 are cognitive and social coefficients (typically 1.5– 2.0), r1 and r2 are random numbers in [0, 1], pbesti is the personal best position, and gbest is the global best. PSO converges faster than GA due to continuous position updates but is prone to premature convergence, especially in high-dimensional spaces with many local optima [12].

Ant Colony Optimization (ACO) is inspired by the foraging behavior of ants, which deposit pheromones on paths to food sources [13]. The probability of ant k selecting hyperparameter value j for parameter i is:

P ij k= (τ ij α·η ij β) ⁄ ∑(τ il α·η il β). (5) Pheromone update follows the rule:

τ ij(t+1) = (1−ρ)·τ ij(t) +Δτij. (6) where ρ is the evaporation rate (0.1–0.5), and Δτij is the pheromone deposit proportional to solution quality. ACO is particularly suited for discrete/combinatorial hyperparameters (e.g., number of layers, activation function type) but requires discretization for continuous parameters [14].

Differential Evolution (DE) is a population-based optimization algorithm that uses difference vectors for perturbation [15]. For each target vector xi, a donor vector is created:

V i=x r1+F·(x r2−x r3), (7)

where r1, r2, r3 are distinct random indices, and F is the scaling factor (typically 0.5–1.0). Crossover with rate CR (0.1–0.9) produces trial vectors. DE is effective for continuous optimization and exhibits good convergence properties but limited exploration in early stages [16].

No single metaheuristic universally outperforms others across all problem landscapes, a principle formalized by the No Free Lunch theorem [17]. Hybridization combines complementary strengths of individual algorithms:

I. GA: strong global exploration, weak local exploitation.

II. PSO: fast convergence, risk of premature convergence.

III. ACO: excellent for discrete spaces, limited for continuous.

IV. DE: good for continuous optimization, limited exploration.

In sequential hybrids, one algorithm runs first to provide initial solutions that seed the second algorithm [18]:

I. GA → PSO: GA explores the global landscape (20–50 generations), then PSO refines the best solutions with focused exploitation (30–50 iterations).

II. PSO → GA: PSO provides fast initial convergence; GA applies crossover/mutation to escape local optima. Advantages: clear separation of exploration and exploitation phases.

Disadvantages: transition timing is critical; early switching loses global coverage, late switching wastes computation.

Multiple algorithms run simultaneously on the same population or subpopulations, periodically sharing best solutions:

I. Island Model: population divided into islands, each optimized by a different MA. Migration of top solutions occurs every k generations.

II. Cooperative Co-evolution: different MAs optimize different subsets of hyperparameters simultaneously. Advantages: leverages multiple search strategies; natural parallelism for distributed computing.

Disadvantages: communication overhead; requires careful migration/synchronization policies.

Operators from different MAs are combined within a single algorithmic framework [19]:

I. Memetic Algorithms: GA + local search (gradient descent or hill climbing) applied to each individual after genetic operations.

II. Hybrid Operators: PSO velocity update combined with GA crossover; ACO pheromone guides DE mutation.

Advantages: tight integration allows dynamic adaptation; can switch strategies per individual.

Disadvantages: complex implementation; more hyperparameters for the optimizer itself.

A CNN architecture was tuned for the Canadian Institute for Advanced Research-10 (CIFAR-10) image classification benchmark (60,000 32×32 color images, 10 classes) [13]. The hyperparameters tuned were:

I. Learning rate: [0.0001, 0.1] (continuous, log-scale).

II. Batch size: {16, 32, 64, 128, 256} (discrete).

III. Number of convolutional filters: {16, 32, 64, 128} per layer (discrete).

IV. Kernel size: {3, 5, 7} (discrete).

V. Dropout rate: [0.0, 0.5] (continuous).

VI. Fitness function: validation accuracy after 50 training epochs. Each evaluation takes approximately 10–15 minutes on a single GPU.

I. GS: exhaustive evaluation over a discretized grid.

II. RS: 200 random configurations.

III. GA-only: population 30, 50 generations.

IV. PSO-only: swarm 30, 50 iterations.

V. GA-PSO Hybrid (Sequential): GA exploration (20 generations) → PSO refinement (30 iterations), population 30.

The GA-PSO hybrid achieved the highest accuracy (92.3%) by leveraging GA's broad exploration in early generations to identify promising regions, followed by PSO's rapid convergence for fine-tuning within those regions. Notably, convergence occurred around iteration 50 (total), compared to iteration 42 for GA-only and 35 for PSO-only, but at a significantly higher accuracy level.

The results confirm the complementary nature of GA and PSO: GA's crossover and mutation operators effectively explore diverse hyperparameter configurations in the first 20 generations, while PSO's velocity-based updates efficiently exploit the identified promising regions. The hybrid achieves 2.6% higher accuracy than GA-only and 2.2% higher than PSO-only, demonstrating that the sequential combination produces synergistic benefits exceeding either component alone.