The new alchemists. Using deep learning to predict stock prices gains maturity

Alchemists of yore sought to short-cut their way to richness by finding ways to convert metals into gold. Today, it appears that the job of finding a quick pathway to prosperity has been taken up by data scientists. Alchemists, of course, failed to transmute cheap metals into gold, but data scientists may be sniffing success.

Today, it is possible to take data sets of quarterly results of a company, daily high, low and close prices and traded volumes, as well as economic data, news and sentiment analysis, and use the data to train a machine learning model to make predictions. The model essentially sees patterns that we would miss.

Data scientists first clean up the data (filling in any missing values and removing outliers) and make it usable by a model. Then they select a deep learning technique (like Long Short-Term Memory) or Convolutional Neural Networks, for milking the data for insights. LSTM is particularly appropriate for time series data, such as historical stock prices, because it can remember previous information and use it to make predictions. Then the model is trained. The training process involves making the model adjust its parameters to reduce the gap between its own predictions and actual stock prices. Then comes validation of the model, using a separate set of data to verify the model’s accuracy.

In broad terms, AI and deep learning predict stock prices by analysing historical data, identifying patterns and making future price forecasts. This involves collecting and pre-processing data, selecting and training appropriate models, making predictions, and continuously evaluating and refining the models to improve their accuracy and reliability.

Hitting the marks

In the last few years, a lot of work has been done in this area, with each model getting better than the earlier ones. A recent work in this area by three data scientists, Jaydip Sen, Hetvi Waghela and Sneha Rakshit, of the Department of Data Science and Artificial Intelligence at the Praxis Business School, Kolkata, has thrown up a model that boasts of 95.8 per cent accuracy in predicting the next day’s stock prices.

Sen, Waghela and Sneha used Long Short-term Memory (LSTM), which is a special kind of artificial neural network used in deep learning, designed to remember information for long periods of time, for “accurate stock price prediction”. In a recent paper (which is yet to be peer-reviewed), they note that “despite the efficient market hypothesis suggesting that such predictions are impossible, there are propositions in the literature demonstrating that advanced algorithms and predictive models can effectively forecast future stock prices.” They add that in recent times, the use of machine learning and deep learning systems has become popular for market price prediction.

Sen, Waghela and Rakshit believe that an LSTM model could predict stock prices better than other techniques such as convolutional neural networks that has been used by others earlier. “This model automatically retrieves historical stock prices using a Python function, utilising stock ticker names from the NSE within a specified interval determined by a start and end date.”

They took historical prices of 180 stocks across 18 sectors, between January 1, 2005 and April 23, 2024. They trained the data on LSTM model to make predictions. They used three metrics — Huber Loss, Mean Absolute Error and Accuracy Score — which give an idea of how wrong a prediction could be, to assess the performance of their model. For a model to be accurate, it must have low values for Huber loss and MAE, and a high value for the accuracy score.

Golden insight

Sen Waghela and Rakshit observed that their LSTM model yielded the minimum value of Huber loss for the auto sector, the minimum value of MAE for the banking sector, and the maximum value of accuracy score for the PSU banks sector. “Hence, the model is found to be the most accurate for these three sectors,” they note. On the other hand, Huber loss and MAE are the maximum for the media sector and accuracy score is the minimum for the energy sector. Therefore, the performance of the model has been the worst for media and energy sectors on the three metrics.

However, overall, the model’s performance has been highly accurate, since the lowest value of the accuracy score is 0.958315. “In other words, in the worst case, the model correctly predicted the direction of movement of the price (upward or downward) of the stock the next day in 95.83 per cent of cases. Thus, the model can be reliably used in stock trading decisions,” they say.

The use of deep learning techniques for making stock predictions will call for radical changes in regulations, so that the market does not get skewed in favour of those who know how to use techniques at the expense of those who don’t.