Context

Stock market data can be interesting to analyze and as a further incentive, strong predictive models can have large financial payoff. The amount of financial data on the web is seemingly endless. A large and well structured dataset on a wide array of companies can be hard to come by. Here I provide a dataset with historical stock prices (last 5 years) for all companies currently found on the S&P 500 index.

The script I used to acquire all of these .csv files can be found in this GitHub repository In the future if you wish for a more up to date dataset, this can be used to acquire new versions of the .csv files.

Content

The data is presented in a couple of formats to suit different individual's needs or computational limitations. I have included files containing 5 years of stock data (in the all_stocks_5yr.csv and corresponding folder) and a smaller version of the dataset (all_stocks_1yr.csv) with only the past year's stock data for those wishing to use something more manageable in size.

The folder individual_stocks_5yr contains files of data for individual stocks, labelled by their stock ticker name. The all_stocks_5yr.csv and all_stocks_1yr.csv contain this same data, presented in merged .csv files. Depending on the intended use (graphing, modelling etc.) the user may prefer one of these given formats.

All the files have the following columns: Date - in format: yy-mm-dd Open - price of the stock at market open (this is NYSE data so all in USD) High - Highest price reached in the day Low Close - Lowest price reached in the day Volume - Number of shares traded Name - the stock's ticker name

Acknowledgements

I scraped this data from Google finance using the python library 'pandas_datareader'. Special thanks to Kaggle, Github and The Market.

Inspiration

This dataset lends itself to a some very interesting visualizations. One can look at simple things like how prices change over time, graph an compare multiple stocks at once, or generate and graph new metrics from the data provided. From these data informative stock stats such as volatility and moving averages can be easily calculated. The million dollar question is: can you develop a model that can beat the market and allow you to make statistically informed trades!

The dataset contains the stock prices of all the assets comprising the S&P 500 index.
Features such as Date, Open, Close, High , Low, Volume for all 503 companies are included in the csv file.
There are approximately 2500 columns and 3400 rows present. Data from 2010 to 07-2023 is present.
This dataset can be effectively used for Exploratory Data Analysis, Time Series Analysis, Predictive Modelling, comparing growth of different companies and visualization.

What is the data?

Used data from Yahoo Finance to get daily data for Opening & Closing Price, Highest & Lowest Prices, Volume of the S&P 500 index.

How was the dataset compiled?

Code: Github Used the yfinance library (github) to import data from yahoo finance directly. Some processing of data was done.

Quality of data

All but a few open prices were missing between 1962-01-01 and 1982-04-10. For these, it was assumed that open price is equal to closing price of previous trading day.

Volume figures until 1949-12-13 are not available.

Some earlier years have less than expected calendar dates | Year with less than expected trading days| Number of Trading Days Recorded | | ---| --- | |1927| 1 | |1928| 195 | | 1929 | 199 | | 1930 | 155 | | 1931 | 183 | | 1932 | 169 | | 1933 | 136 | | 1934 | 91 | | 1935 | 83 | | 1936 | 107 | | 1937 | 83 | | 1938 | 57 | | 1939 | 27 | | 1940 | 8 | | 1941 | 6 | | 1942 | 16 | | 1943 | 7 | | 1944 | 6 | | 1945 | 42 | | 1946 | 48 | | 1947 | 18 | | 1948 | 16 | | 1949 | 1 | | 1968 | 226 |

Added columns for:

 1. percentage Gain/Loss (calculated by taking the percentage difference between closing prices of 2 consecutive trading days)
 2. price variation percentage: (High-Low)/Closing

This dataset provides historical stock market performance data for specific companies. It enables users to analyze and understand the past trends and fluctuations in stock prices over time. This information can be utilized for various purposes such as investment analysis, financial research, and market trend forecasting.

This project delivers a comprehensive analysis of S&P 500 companies using SQL and Python to uncover actionable insights from financial data. The process began with data cleaning in Microsoft SQL Server, where critical variables—such as company sector, market capitalization, EBITDA, and employee count—were standardized and prepared for analysis.

This analysis presents a rigorous exploration of financial data, incorporating a diverse range of statistical features. By providing a robust foundation, it facilitates advanced research and innovative modeling techniques within the field of finance.

Is the S&P Bitcoin Index the Future of Crypto Investment?

Financial data:

• Historical daily stock prices (open, high, low, close, volume)

• Fundamental data (e.g., market capitalization, price to earnings P/E ratio, dividend yield, earnings per share EPS, price to earnings growth, debt-to-equity ratio, price-to-book ratio, current ratio, free cash flow, projected earnings growth, return on equity, dividend payout ratio, price to sales ratio, credit rating)

• Technical indicators (e.g., moving averages, RSI, MACD, average directional index, aroon oscillator, stochastic oscillator, on-balance volume, accumulation/distribution A/D line, parabolic SAR indicator, bollinger bands indicators, fibonacci, williams percent range, commodity channel index)

Machine learning features:

• Feature engineering based on financial data and technical indicators

• Sentiment analysis data from social media and news articles

• Macroeconomic data (e.g., GDP, unemployment rate, interest rates, consumer spending, building permits, consumer confidence, inflation, producer price index, money supply, home sales, retail sales, bond yields)

Potential Applications:

• Stock price prediction

• Portfolio optimization

• Algorithmic trading

• Market sentiment analysis

• Risk management

Use Cases:

• Researchers investigating the effectiveness of machine learning in stock market prediction

• Analysts developing quantitative trading Buy/Sell strategies

• Individuals interested in building their own stock market prediction models

• Students learning about machine learning and financial applications

Additional Notes:

• The dataset may include different levels of granularity (e.g., daily, hourly)

• Data cleaning and preprocessing are essential before model training

• Regular updates are recommended to maintain the accuracy and relevance of the data

Full historical data for the S&P 500 (ticker ^GSPC), sourced from Yahoo Finance (https://finance.yahoo.com/).

Including Open, High, Low and Close prices in USD + daily volumes.

Info about S&P 500: https://en.wikipedia.org/wiki/S%26P_500

This analysis presents a rigorous exploration of financial data, incorporating a diverse range of statistical features. By providing a robust foundation, it facilitates advanced research and innovative modeling techniques within the field of finance.

Is the S&P Bitcoin Index the Future of Digital Asset Investing?

Financial data:

• Historical daily stock prices (open, high, low, close, volume)

• Fundamental data (e.g., market capitalization, price to earnings P/E ratio, dividend yield, earnings per share EPS, price to earnings growth, debt-to-equity ratio, price-to-book ratio, current ratio, free cash flow, projected earnings growth, return on equity, dividend payout ratio, price to sales ratio, credit rating)

• Technical indicators (e.g., moving averages, RSI, MACD, average directional index, aroon oscillator, stochastic oscillator, on-balance volume, accumulation/distribution A/D line, parabolic SAR indicator, bollinger bands indicators, fibonacci, williams percent range, commodity channel index)

Machine learning features:

• Feature engineering based on financial data and technical indicators

• Sentiment analysis data from social media and news articles

• Macroeconomic data (e.g., GDP, unemployment rate, interest rates, consumer spending, building permits, consumer confidence, inflation, producer price index, money supply, home sales, retail sales, bond yields)

Potential Applications:

• Stock price prediction

• Portfolio optimization

• Algorithmic trading

• Market sentiment analysis

• Risk management

Use Cases:

• Researchers investigating the effectiveness of machine learning in stock market prediction

• Analysts developing quantitative trading Buy/Sell strategies

• Individuals interested in building their own stock market prediction models

• Students learning about machine learning and financial applications

Additional Notes:

• The dataset may include different levels of granularity (e.g., daily, hourly)

• Data cleaning and preprocessing are essential before model training

• Regular updates are recommended to maintain the accuracy and relevance of the data

This dataset explores the relationship between ESG (Environmental, Social, Governance) ratings and key financial metrics across companies in the S&P 500. It includes data points such as P/E Ratio, Revenue, Net Income, EPS, Market Cap, and 1-Year Stock Return, alongside ESG scores and sector classification. Ideal for machine learning, regression modeling, and exploratory data analysis, it helps answer questions like: Do high ESG scores correlate with higher valuation? Or are they a signal of overpriced equity?

This analysis presents a rigorous exploration of financial data, incorporating a diverse range of statistical features. By providing a robust foundation, it facilitates advanced research and innovative modeling techniques within the field of finance.

Will the S&P Bitcoin Index Spark a New Era of Investment?

Financial data:

• Historical daily stock prices (open, high, low, close, volume)

• Fundamental data (e.g., market capitalization, price to earnings P/E ratio, dividend yield, earnings per share EPS, price to earnings growth, debt-to-equity ratio, price-to-book ratio, current ratio, free cash flow, projected earnings growth, return on equity, dividend payout ratio, price to sales ratio, credit rating)

• Technical indicators (e.g., moving averages, RSI, MACD, average directional index, aroon oscillator, stochastic oscillator, on-balance volume, accumulation/distribution A/D line, parabolic SAR indicator, bollinger bands indicators, fibonacci, williams percent range, commodity channel index)

Machine learning features:

• Feature engineering based on financial data and technical indicators

• Sentiment analysis data from social media and news articles

• Macroeconomic data (e.g., GDP, unemployment rate, interest rates, consumer spending, building permits, consumer confidence, inflation, producer price index, money supply, home sales, retail sales, bond yields)

Potential Applications:

• Stock price prediction

• Portfolio optimization

• Algorithmic trading

• Market sentiment analysis

• Risk management

Use Cases:

• Researchers investigating the effectiveness of machine learning in stock market prediction

• Analysts developing quantitative trading Buy/Sell strategies

• Individuals interested in building their own stock market prediction models

• Students learning about machine learning and financial applications

Additional Notes:

• The dataset may include different levels of granularity (e.g., daily, hourly)

• Data cleaning and preprocessing are essential before model training

• Regular updates are recommended to maintain the accuracy and relevance of the data

This analysis presents a rigorous exploration of financial data, incorporating a diverse range of statistical features. By providing a robust foundation, it facilitates advanced research and innovative modeling techniques within the field of finance.

iShares S&P Mid-Cap 400 Value ETF: A Smart Investment?

Financial data:

• Historical daily stock prices (open, high, low, close, volume)

• Fundamental data (e.g., market capitalization, price to earnings P/E ratio, dividend yield, earnings per share EPS, price to earnings growth, debt-to-equity ratio, price-to-book ratio, current ratio, free cash flow, projected earnings growth, return on equity, dividend payout ratio, price to sales ratio, credit rating)

• Technical indicators (e.g., moving averages, RSI, MACD, average directional index, aroon oscillator, stochastic oscillator, on-balance volume, accumulation/distribution A/D line, parabolic SAR indicator, bollinger bands indicators, fibonacci, williams percent range, commodity channel index)

Machine learning features:

• Feature engineering based on financial data and technical indicators

• Sentiment analysis data from social media and news articles

• Macroeconomic data (e.g., GDP, unemployment rate, interest rates, consumer spending, building permits, consumer confidence, inflation, producer price index, money supply, home sales, retail sales, bond yields)

Potential Applications:

• Stock price prediction

• Portfolio optimization

• Algorithmic trading

• Market sentiment analysis

• Risk management

Use Cases:

• Researchers investigating the effectiveness of machine learning in stock market prediction

• Analysts developing quantitative trading Buy/Sell strategies

• Individuals interested in building their own stock market prediction models

• Students learning about machine learning and financial applications

Additional Notes:

• The dataset may include different levels of granularity (e.g., daily, hourly)

• Data cleaning and preprocessing are essential before model training

• Regular updates are recommended to maintain the accuracy and relevance of the data

VIX Daily Price Data

Overview

Contains historical data of the VIX Volatility Index from 2000 - 2025. The data is obtained from the yfinance api created by yahoo finance and contains the daily price data for the VIX.

The dataset contains the daily Open, Close, High, and Low of the VIX.

Columns Open: Starting price level of VIX for the day Close: Final price level of VIX for the day High: Highest price level of VIX for the day Low: Lowest price level of VIX for the day

The VIX is an index that measures near term volatility expectations for the S&P 500 gathered from SPX options data. VIX was created and maintained by CBOE.

Uses

This data can be used to train models on predicting the market's volatility forecasts. The VIX can also be compared to the realized historical volatility over a period of time.