Comparison of LSTM and Transformer Models in Predicting NVIDIA Stock Closing Prices and the Application of Rule-based Trading Strategies

Abstract

In today’s modern financial landscape, where accuracy and speed of prediction are increasingly critical, machine learning techniques play a vital role in stock price forecasting.
This study evaluates the effectiveness of two deep learning models—Long Short-Term Memory (LSTM) and Transformer—in predicting NVIDIA (NVDA) stock prices using historical data from June 7, 2021 to June 7, 2025, with an 80% training and 20% testing data split. The results show that the LSTM model achieved a Root Mean Squared Error (RMSE) of 2.7703 on the training data and 7.3796 on the testing data, while the Transformer model produced an RMSE of 5.3573 (training) and 10.8563 (testing). A hybrid model demonstrated improved prediction accuracy with an RMSE of 3.5643 (training) and 8.6727 (testing), although it still did not outperform LSTM. The model also indicated a moderately declining trend in stock prices over the projected 30-day period. Gaussian noise augmentation was applied during training to improve model generalization. This study also explores investment strategy development by analyzing rule-based trading signals, generating buy (long) and sell (short) signals based on predicted price movements. Additionally, risks such as market volatility and potential overfitting were evaluated, alongside the influence of non-technical factors such as market sentiment. The primary focus of the research is to compare the performance of the LSTM and Transformer models in forecasting NVIDIA’s closing stock prices and applying a simple rule-based trading strategy. For future work, the use of methods such as Prophet, ARIMA, and hybrid ensemble approaches is recommended to enhance prediction accuracy, improve market adaptability, and deliver a more robust stock forecasting system leveraging advanced machine learning techniques for more optimal investment decisions.