Can Stock Price be predicted using Artificial Intelligence and Sentiment Analysis?

The stock market price prediction is the act of trying to determine the future value of company stock or other financial instrument traded on an exchange. The successful prediction of a stock’s future price could yield a significant profit. We can use a Machine Learning (ML) approach that will be trained from available stocks data, gain intelligence and then uses the acquired knowledge for accurate prediction. Neural network models having the features and customize parameters makes it possible to implement a wide number of features along with the cross-validation sets.

A hypothesis is based on the premise of behavioural economics, that the emotions and moods of individuals affect their decision-making process, thus, leading to a direct correlation between public sentiment and market sentiment. We can also perform sentiment analysis and use these moods and previous days’ stock values to predict future stock movements. This will help researchers in this field to know the current trend as well as help to inform their future research efforts.

Though there are many stock prediction models already available in the market they all use a single static dataset of stock prices for training and hence predict stock prices. This article is about building such a stock prediction mechanism which could be used for real-life stock prediction problems and take in consideration the requirements of its user without compromising on the accuracy. It should be able to predict stock prices for any data input by the user rather than just measuring the predictability of stocks of an organization using machine learning algorithms on a static dataset. Thus it is essential to dynamically gather data which could be used for training of stock prediction model according to the dates specified by the user and then make predictions about the stock prices using machine learning regression techniques. To make out for the only-mathematical predictions made by machine learning regression techniques is not enough we also need to include sentiment analysis of news articles on the organization for which stock prices are to be predicted.

The stock market prediction process is filled with uncertainty and can be influenced by multiple factors. Therefore, the stock market plays an important role in business and finance. The technical and fundamental analysis is done by sentimental analysis process. Social media data has a high impact due to its increased usage, and it can help predict the trend of the stock market. Technical analysis is done using by applying machine learning algorithms on historical data of stock prices.

The Three essential methods

There were always three methods to analyze and predict the stock market: financial, technical and sentiment. 

  1. The financial analysis evaluates past statements, reports and balances sheets and comparing it to prospects, the market and changes in government policy.
  2. Technical analysis relies on the idea all factors which can influence the price are included in the current price of the stock and therefore no fundamental information analysis is required. They believe that the prices move in trends and the same historic patterns.
  3. Sentiment analysis relies on taking advice from experts and going through newspapers to monitor the stocks they’d like to invest in., going through text and data points to understand any changes that would move the price.
Fig – Processes involved in Machine Learning. Image Source –https://towardsdatascience.com/data-preprocessing-3cd01eefd438

Objectives

  • Collecting Data: – Collecting historical data of a particular stock is required to carry out further analysis and prediction tasks. For that purpose, web scraping is used as a tool to collect information (historical data) of a stock.
  • Visualization of the stock data: – Stock Visualization is very important to get the complete and clear picture of how the stock of a particular company performing. For stock visualization various graphs like Time-Series graph, Scatter Plots etc. are used.
Fig – Time series plot for open , low and high prices.
  • Pre-processing: – Data pre-processing is an important phase in which missing data is filled, some columns are scaled, some columns are encoded and many more. All this is done to apply various machine learning models on the preprocessed data.
  • Prediction of Stock Price using linear regression: – Feeding open price to the Linear Regression Model will predict the High price of a stock for that day.
  • Prediction using Recurrent Neural Network (RNN): – Previous days’ data is utilized to predict the prices for the next day and so on. This model is more realistic as stock prices very much depend on the previous days’ prices. 
  • Sentimental Analysis: – Sentimental analysis using Twitter and other news sources and incorporating this extra feature in both the models. Applications of sentimental analysis are social media monitoring, brand monitoring, customer feedback, business intelligence build up and market research.
  • Developing Web application: – A complete web application that can efficiently demonstrate the data visualization, predictions made by the two models and their respective accuracy and many more.

Road-Map

  • Data Collection and Web Scrapping

Collecting data allows you to capture a record of past events so that data analysis can be used to find recurring patterns. From those patterns, you build predictive models using machine learning algorithms that look for trends and predict future changes. Predictive models are only as good as the data from which they are built, so good data collection practices are crucial to developing high-performing models. The data need to be error-free (garbage in, garbage out) and contain relevant information for the task at hand.

  • Data Visualization

Data visualization is a quick, easy way to convey concepts universally and it can be experimented with different scenarios by making slight adjustments. Data visualization also helps identify areas that need attention, e.g. outliers, which can later impact our machine learning model.

Fig – Scatter Plot Between Open and High Price.
Fig – Candlestick-OHLC Graph for better visualization.
  • Applying Machine Learning Models and Sentiment Analysis

The third phase is applying Linear regression model on our pre-processed data-set. The next model that can be used is the Recurrent Neural Network (RNN) in which Long Short Term Memory (LSTM) architecture is used. In the next phase, sentiments can be used as a tool to increase the accuracy of our prediction model to some extent. So for sentimental analysis, there are several pre-existing options available like twitter sentimental analysis, Reddit sentimental analysis and many more.

References

  1. Dinesh Bhuriya, Girish Kaushal, Ashish Sharma and Upendra Singh, “Stock
    Market Predication Using a Linear Regression.”, International Conference of
    Electronics, Communication and Aerospace Technology, ICECA 2017, 20-22 April
    2017, Coimbatore, India, IEEE, 2017, pp 510-513.

https://ieeexplore.ieee.org/document/8212716


2. Yahya Eru Cakra and Bayu Distiawan Trisedya, “Stock Price Prediction Using
Linear Regression Based on Sentiment Analysis.”, in International Conference on
Advanced Computer Science and Information Systems, ICACSIS 2015,10-11 October
2015, Depok, Indonesia, IEEE, 2015, pp 147-154.

https://ieeexplore.ieee.org/abstract/document/7415179

2 thoughts on “Can Stock Price be predicted using Artificial Intelligence and Sentiment Analysis?

  1. This is an amazing article. I have been thinking on these terms for quite a while and all the information you put out is so relevant and apt. Loved it thoroughly!

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s