The concept of portfolio optimization was formalized by Harry Markowitz [7], introducing the Mean-Variance framework that considers risk and return simultaneously [2]. A central theme is selecting an optimal investment portfolio relative to its risk-return profile. Markowitz demonstrated that portfolio risk depends not only on the variance of individual assets but also on the correlation between their returns. Therefore, two high-risk stocks, when combined, can create a lower-risk portfolio provided their correlation is low. The primary objective is to find specific asset weights that minimize portfolio variance while maximizing expected return.

RF, introduced by Breiman [8], is recognized as one of the most robust ML algorithms for both classification and regression tasks. By using multiple decision trees and ensemble learning techniques, this algorithm can model complex, nonlinear patterns with high precision. Its stability, resistance to noise, and ability to handle outliers make it an exceptional performer in analyzing complex financial data in volatile markets.

XGBoost, introduced by Chen and Guestrin [9], has attracted significant attention in the financial sector for its ability to process and predict data within an optimal timeframe. Its sequential gradient boosting hierarchy and optimized decision trees capture complex, nonlinear interactions and cross-correlations among financial variables more effectively than classical models. For stock return forecasting, volatility prediction, trend identification, and risk analysis, XGBoost has demonstrated superior performance.

DRL has emerged as a powerful framework for sequential decision-making in financial markets [6]. Unlike supervised learning approaches that predict returns independently, DRL agents learn optimal policies by interacting with the market environment, receiving rewards based on portfolio performance. PPO provides stable policy updates with guaranteed monotonic improvement, making it particularly suitable for the non-stationary dynamics of financial markets [10].

The statistical database for this study comprises companies listed on the TSE between the years 2013 and 2022 (1392 to 1401 SH). Following data collection and preliminary calculations, 20 high-liquidity companies were selected based on liquidity, trading volume, transaction volume, and market impact. Daily adjusted returns were calculated for all selected companies.

The feature engineering process involves extracting relevant financial indicators from raw price and volume data:

I. Daily returns and log returns II. Moving averages (5-day, 10-day, 20-day, 50-day) III. Relative Strength Index (RSI) IV. Moving Average Convergence Divergence (MACD) V. Bollinger Bands (upper and lower) VI. Volume-Weighted Average Price (VWAP) VII. Historical volatility measures

The stock selection process is conducted in several stages. First, the XGBoost model is trained to predict future returns of all stocks within the research sample. Subsequently, the predicted return for each stock is calculated. The XGBoost model's output ranks stocks, and only those in the upper percentiles of the predicted return distribution are considered selected. Furthermore, XGBoost allows the extraction of Feature Importance, which identifies which technical indicators played the most significant role in predicting stock returns.

The XGBoost objective function consists of two primary components: the loss function (prediction error) and the regularization term controlling model complexity:

ℒ(φ) = ∑i ∗ l(ŷi, yi) + ∑kΩ( fk), (1)

where l is the loss function measuring the difference between the predicted value ŷi and the actual value yi, and Ω is the regularization term preventing overfitting by penalizing model complexity.

The RF algorithm complements XGBoost by capturing nonlinear features for price prediction. Each decision tree is constructed by classifying bootstrap samples of the input data. The ensemble output is determined by averaging each tree's votes, providing robust predictions that are resistant to noise and outliers.

The optimization algorithm is based on DRL using PPO. The DRL framework is defined as follows:

I. State space: the state at time t includes predicted returns from ML models, current portfolio weights, historical volatility measures, and market indicators.

II. Action space: the action is a vector of portfolio weights w = (w1, w2, … , wN) where ∑wi = 1 and wi ≥ 0 for all i.

III. Reward function: the reward is defined as the risk-adjusted portfolio return, incorporating the selected risk metric (Variance, Mean Absolute Deviation (MAD), Semi-Variance, or Conditional Value at Risk (CVaR)). The PPO agent learns to dynamically adjust portfolio weights by continuously interacting with historical market data, optimizing the trade-off between return maximization and risk minimization. The PPO objective function is given by:

LCLIP(θ) = t[ min( rt(θ)Ât, clip( rt(θ), 1 − ε, 1 + ε )Ât)], (2)

where rt(θ) is the probability ratio between the new and old policy, Ât is the estimated advantage function, and ε is the clipping hyperparameter.

Four distinct risk metrics are employed for comprehensive risk assessment:

Variance: The variance of returns is computed as

σi2 = ∑( ri − r̄ ) ∗ (2/ N), (3)

representing the degree of dispersion of actual returns around the expected return.

MAD:

MAD = ∑| ri− r̄ | / N , (4)

which is less sensitive to outliers than variance, allowing simpler calculations in large-scale optimization models.

Semi-Variance:

SV = { min( ri − r̄, 0 ) ∗ 2}, (5)

focusing only on downside risk by considering only returns that fall below the mean.

CVaR: addresses the limitations of standard VaR by accounting for the severity of losses in the tail of the distribution, defined as the expected loss given that the loss exceeds the VaR threshold:

CvaRα = [ −R | R ≤ −VaRα]. (6)

The dataset was divided into training (2013–2019) and testing (2020–2022) periods. Both XGBoost and RF models were trained on the training set with cross-validation to prevent overfitting. The DRL agent was trained using the PPO algorithm with the following hyperparameters: learning rate = 0.0003, batch size = 64, epochs = 500, discount factor γ = 0.99, clip ratio ε = 0.2.

The combined ML pipeline (XGBoost + RF) achieved strong prediction accuracy. The algorithm's error rate was close to zero, indicating accurate learning and prediction. Feature importance analysis revealed that volume-related features, momentum indicators, and historical volatility were the most discriminative predictors.

The portfolio optimization results demonstrate that the expected rate of return across all four risk models is 0.001, with risk indices of 0.01 for Variance, Semi-Variance, and MAD, and 0.03 for CVaR. The CVaR model shows a higher risk index because it accounts for tail risks more conservatively, providing a more comprehensive view of extreme loss scenarios.

The proposed hybrid model outperforms all benchmark strategies across every metric. The combination of ML-based stock selection with DRL-based dynamic allocation achieves the highest annual return (22.3%), the lowest volatility (13.2%), and the maximum drawdown (−11.8%). The Sharpe ratio of 1.69 indicates a significantly superior risk-adjusted return profile compared to the traditional Mean-Variance approach (0.94) and individual ML models (1.16 and 1.24 for RF and XGBoost, respectively).