Abstract: Accurate deformation prediction is essential for the long-term safety monitoring and structural health evaluation of ultra-high arch dams. Traditional prediction methods often fail to comprehensively capture the coupled periodic fluctuation, long-term trend evolution, and strong nonlinear random noise contained in dam deformation time series, resulting in limited prediction accuracy and generalizability. To enhance the prediction accuracy of the deformation prediction for ultra-high arch dams, a combined model is developed by integrating long short-term memory (LSTM), variational mode decomposition (VMD), extreme gradient boosting (XGBoost), self-attention (SA), and the rapidly exploring random tree optimization algorithm (RRTO), referred to as the SAXGBoost-LSTM prediction model. First, the measured time series of the deformation time series is decomposed into a series of intrinsic modes and a residual sequence using VMD. Unlike conventional empirical decomposition methods, VMD can adaptively separate signal components with independent frequency characteristics and effectively suppress mode mixing phenomena, providing a stable decomposition foundation for subsequent multi-component modeling. The intrinsic modes are reconstructed into periodic and trend terms based on sample entropy. Sample entropy can quantitatively reflect the complexity and randomness of each mode component, which provides a reliable basis for distinguishing periodic load-induced deformation and long-term structural trend deformation. The RRTO-SAXGBoost model is utilized for prediction based on the characteristics of the periodic and trend terms. The self-attention mechanism is introduced to optimize the feature extraction capability of XGBoost, which can adaptively assign weights to different time-series features and highlight key deformation variation information. Moreover, the RRTO algorithm is adopted to optimize the hyperparameters of the model, avoiding the prediction instability caused by manual parameter selection. Due to the strong nonlinear characteristics of the residual data, the residual sequence is first denoised and the denoised data is then subjected to secondary decomposition to reduce the prediction difficulty. The wavelet denoising operation can effectively eliminate high-frequency measurement noise in residual sequences and retain effective deformation information. The secondary decomposition further reduces the nonlinear complexity and randomness of residual subsequences, making the difficult-to-predict residual components more regular and predictable. The decomposed residual subsequences are predicted using the LSTM. LSTM possesses unique advantages in learning long-term dependencies and nonlinear variation rules of time-series data, which is suitable for fine prediction of complex residual subsequences. Finally, the final prediction values of the dam deformation predictions are obtained by superimposing the prediction results of the periodic, trend, and residual sequences. The research results demonstrate that wavelet denoising, combined with secondary decomposition, can extract useful information from the residual sequence. The combined prediction model can capture the nonlinear relationship between dam deformation and environmental variables. Compared with single prediction models and traditional decomposition-combined models, the proposed hybrid framework achieves better fitting performance and generalization ability. It provides a new approach to predicting the structural performance of ultra-high arch dams.