This dataset compiles the top 2500 datasets from Kaggle, encompassing a diverse range of topics and contributors. It provides insights into dataset creation, usability, popularity, and more, offering valuable information for researchers, analysts, and data enthusiasts.

Research Analysis: Researchers can utilize this dataset to analyze trends in dataset creation, popularity, and usability scores across various categories.

Contributor Insights: Kaggle contributors can explore the dataset to gain insights into factors influencing the success and engagement of their datasets, aiding in optimizing future submissions.

Machine Learning Training: Data scientists and machine learning enthusiasts can use this dataset to train models for predicting dataset popularity or usability based on features such as creator, category, and file types.

Market Analysis: Analysts can leverage the dataset to conduct market analysis, identifying emerging trends and popular topics within the data science community on Kaggle.

Educational Purposes: Educators and students can use this dataset to teach and learn about data analysis, visualization, and interpretation within the context of real-world datasets and community-driven platforms like Kaggle.

Column Definitions:

Dataset Name: Name of the dataset. Created By: Creator(s) of the dataset. Last Updated in number of days: Time elapsed since last update. Usability Score: Score indicating the ease of use. Number of File: Quantity of files included. Type of file: Format of files (e.g., CSV, JSON). Size: Size of the dataset. Total Votes: Number of votes received. Category: Categorization of the dataset's subject matter.

I imported the two Olist Kaggle datasets into an SQLite database. I modified the original table names to make them shorter and easier to understand. Here's the Entity-Relationship Diagram of the resulting SQLite database:

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2473556%2F23a7d4d8cd99e36e32e57303eb804fff%2Fdb-schema.png?generation=1714391550829633&alt=media" alt="Database Schema">

Data sources:

https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce

https://www.kaggle.com/datasets/olistbr/marketing-funnel-olist

I used this database as a data source for my notebook:

SQL Challenge: E-commerce data analysis

From wiki

Kaggle, a subsidiary of Google LLC, is an online community of data scientists and machine learning practitioners. Kaggle allows users to find and publish data sets, explore and build models in a web-based data-science environment, work with other data scientists and machine learning engineers, and enter competitions to solve data science challenges.

Kaggle got its start in 2010 by offering machine learning competitions and now also offers a public data platform, a cloud-based workbench for data science, and Artificial Intelligence education. Its key personnel were Anthony Goldbloom and Jeremy Howard. Nicholas Gruen was founding chair succeeded by Max Levchin. Equity was raised in 2011 valuing the company at $25 million. On 8 March 2017, Google announced that they were acquiring Kaggle.[1][2]

Source: Kaggle

Context

I've always wanted to explore Kaggle's Meta Kaggle dataset but I am more comfortable on using TSQL when it comes to writing (very) complex queries. Also, I tend to write queries faster when using SQL MANAGEMENT STUDIO, like 100x faster. So, I ported Kaggle's Meta Kaggle dataset into MS SQL SERVER 2022 database format, created a backup file, then uploaded it here.

• MSSQL VERSION: SQL Server 2022
• Collation: SQL_Latin1_General_CP1_CI_AS
• Recovery model: simple

Requirements

• Download and install the SQL SERVER 2022 Developer edition here
• Download the backup file
• Restore the backup file into your local. If you havent done this before, it's easy and straightforward. Here is a guide.

(QUOTED FROM THE ORIGINAL DATASET)

Meta Kaggle

Explore Kaggle's public data on competitions, datasets, kernels (code/ notebooks) and more Meta Kaggle may not be the Rosetta Stone of data science, but they think there's a lot to learn (and plenty of fun to be had) from this collection of rich data about Kaggle’s community and activity.

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F2ad97bce7839d6e57674e7a82981ed23%2F2Egeb8R.png?generation=1688912953875842&alt=media" alt="">

Notes

• I repeat, I just ported the dataset. All credits to Kaggle for the amazing source dataset.
• Cover image from https://picryl.com/media/space-earth-bug-ce3ca6

This is the sample database from sqlservertutorial.net. This is a great dataset for learning SQL and practicing querying relational databases.

Database Diagram:

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4146319%2Fc5838eb006bab3938ad94de02f58c6c1%2FSQL-Server-Sample-Database.png?generation=1692609884383007&alt=media" alt="">

Terms of Use

The sample database is copyrighted and cannot be used for commercial purposes. For example, it cannot be used for the following but is not limited to the purposes: - Selling - Including in paid courses

Dataset

This dataset was created by nitishaadhikari

Contents

Cleaned Meta-Kaggle Dataset

The Original Dataset - Meta-Kaggle

Explore our public data on competitions, datasets, kernels (code / notebooks) and more Meta Kaggle may not be the Rosetta Stone of data science, but we do think there's a lot to learn (and plenty of fun to be had) from this collection of rich data about Kaggle’s community and activity.

Strategizing to become a Competitions Grandmaster? Wondering who, where, and what goes into a winning team? Choosing evaluation metrics for your next data science project? The kernels published using this data can help. We also hope they'll spark some lively Kaggler conversations and be a useful resource for the larger data science community.

https://i.imgur.com/2Egeb8R.png" alt="" title="a title">

This dataset is made available as CSV files through Kaggle Kernels. It contains tables on public activity from Competitions, Datasets, Kernels, Discussions, and more. The tables are updated daily.

Please note: This data is not a complete dump of our database. Rows, columns, and tables have been filtered out and transformed.

August 2023 update

In August 2023, we released Meta Kaggle for Code, a companion to Meta Kaggle containing public, Apache 2.0 licensed notebook data. View the dataset and instructions for how to join it with Meta Kaggle here

We also updated the license on Meta Kaggle from CC-BY-NC-SA to Apache 2.0.

The Problems with the Original Dataset

• The original dataset is 32 CSV files, with 268 colums and 7GB of compressed data. Having so many tables and columns makes it hard to understand the data.
• The data is not normalized, so when you join tables you get a lot of errors.
• Some values refer to non-existing values in other tables. For example, the UserId column in the ForumMessages table has values that do not exist in the Users table.
• There are missing values.
• There are duplicate values.
• There are values that are not valid. For example, Ids that are not positive integers.
• The date and time columns are not in the right format.
• Some columns only have the same value for all rows, so they are not useful.
• The boolean columns have string values True or False.
• Incorrect values for the Total columns. For example, the DatasetCount is not the total number of datasets with the Tag according to the DatasetTags table.
• Users upvote their own messages.

The Solution

• To handle so many tables and columns I use a relational database. I use MySQL, but you can use any relational database.
• The steps to create the database are:
• Creating the database tables with the right data types and constraints. I do that by running the db_abd_create_tables.sql script.
• Downloading the CSV files from Kaggle using the Kaggle API.
• Cleaning the data using pandas. I do that by running the clean_data.py script. The script does the following steps for each table:
  • Drops the columns that are not needed.
  • Converts each column to the right data type.
  • Replaces foreign keys that do not exist with NULL.
  • Replaces some of the missing values with default values.
  • Removes rows where there are missing values in the primary key/not null columns.
  • Removes duplicate rows.
• Loading the data into the database using the LOAD DATA INFILE command.
• Checks that the number of rows in the database tables is the same as the number of rows in the CSV files.
• Adds foreign key constraints to the database tables. I do that by running the add_foreign_keys.sql script.
• Update the Total columns in the database tables. I do that by running the update_totals.sql script.
• Backup the database.

Overview: This dataset comprises information scraped from wonderbk.com, a popular online bookstore. The dataset contains details of 103,063 books, with key attributes such as title, authors, description, category, publisher, starting price, and publish date.

Columns:

• Title: The title of the book.
• Authors: The authors of the book.
• Description: A brief description of the book.
• Category: The category or genre to which the book belongs.
• Publisher: The publishing house responsible for the book.
• Price Starting With ($): The initial price of the book.
• Publish Date (Month): The month in which the book was published.
• Publish Date (Year): The year of publication.

Context

Typically e-commerce datasets are proprietary and consequently hard to find among publicly available data. However, The UCI Machine Learning Repository has made this dataset containing actual transactions from 2010 and 2011. The dataset is maintained on their site, where it can be found by the title "Online Retail".

Content

"This is a transnational data set which contains all the transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail.The company mainly sells unique all-occasion gifts. Many customers of the company are wholesalers."

Acknowledgements

Per the UCI Machine Learning Repository, this data was made available by Dr Daqing Chen, Director: Public Analytics group. chend '@' lsbu.ac.uk, School of Engineering, London South Bank University, London SE1 0AA, UK.

Image from stocksnap.io.

Inspiration

Analyses for this dataset could include time series, clustering, classification and more.

Context

ECG data from mit-bih database from physionet

Content

Raw signals in .csv files and original annotations in .txt.

Acknowledgements

https://www.physionet.org/physiobank/database/mitdb/

Note: This is a work in progress, and not all the Kaggle forums are included in this dataset. The remaining forums will be added when I end solving some issues with the data generators related to these forums.

Summary

Welcome to the Kaggle Forum Discussions dataset!. This dataset contains curated data about recent discussions opened in the different forums on Kaggle. The data is obtained through web scraping techniques, using the selenium libraries, and converting text data into markdown style using the markdownify package.

This dataset contains information about the discussion main topic, topic title, comments, votes, medals and more, and is designed to serve as a complement to the data available on the Kaggle meta dataset, specifically for recent discussions. Keep reading to see the details.

Extraction Technique

As a dynamic website that relies heavily in JavaScript (JS), I extracted the data in this dataset through web scraping techniques using the selenium library.

The functions and classes used to scrape the data on Kaggle where stored on a utility script publicly available here. As JS-generated pages like Kaggle are unstable where trying to scrape them, the mentioned script implements capabilities for retrying connections and to await for elements to appear.

Each Forum was scrapped using a one notebook for each, then the mentioned notebooks were connected to a central notebook that generates this dataset. Also the discussions are scrapped in parallel so to enhance speed. This dataset represents all the data that can be gathered in a single notebook session, from the most recent to the most old.

If you need more control on the data you want to research, feel free to import all you need from the utility script mentioned before.

Structure

This dataset contains several folders, each named as the discussion forum they contain data about. For example, the 'competition-hosting' folder contains data about the Competition Hosting forum. Inside each folder, you'll find two files: one is a csv file and the other a json file.

The json file (in Python, represented as a dictionary) is indexed with the ID that Kaggle assigns to the mentioned discussion. Each ID is paired with its corresponding discussion, which is represented as a nested dictionary (the discussion dict), which contains the following fields: - title: The title of the main topic. - content: Content of the main topic. - tags: List containing the discussion's tags. - datetime: Date and time at which the discussion was published (in ISO 8601 format). - votes: Number of votes gotten by the discussion. - medal: Medal awarded by the main topic (if any). - user: User that published the main topic. - expertise: Publisher's expertise, measured by the Kaggle progression system. - n_comments: Total number of comments in the current discussion. - n_appreciation_comments: Total number of appreciation comments in the current discussion. - comments: Dictionary containing data about the comments in the discussion. Each comment is indexed by an ID assigned by Kaggle, containing the following fields: - content: Comment's content. - is_appreciation: Wether the comment is of appreciation. - is_deleted: Wether the comment was deleted. - n_replies: Number of replies to the comment. - datetime: Date and time at which the comment was published (in ISO 8601 format). - votes: Number of votes gotten by the current comment. - medal: Medal awarded by the comment (if any). - user: User that published the comment. - expertise: Publisher's expertise, measured by the Kaggle progression system. - n_deleted: Total number of deleted replies (including self). - replies: A dict following this same format.

By other side, the csv file serves as a summary of the json file, containing information about the comments limited to the hottest and most voted comments.

Note: Only the 'content' field is mandatory for each discussion. The availability of the other fields is subject to the stability of the scraping tasks, which may also affect the update frequency.

This is a synthetic data created for a college project. This data aims to predict whether students will continue to go to college or not. With machine learning explainability, school counselors can help students that will not go to college by finding the factor and helping them. Lets build something really helpful. Here is my recommendation notebook.

PS: Like I said before, this is synthetic data. If you have a resource to get real data, your contribution is welcome. Thank you.

Dataset: Online Shopping Dataset;

CustomerID

Description: Unique identifier for each customer. Data Type: Numeric;

Gender:

Description: Gender of the customer (e.g., Male, Female). Data Type: Categorical;

Location:

Description: Location or address information of the customer. Data Type: Text;

Tenure_Months:

Description: Number of months the customer has been associated with the platform. Data Type: Numeric;

Transaction_ID:

Description: Unique identifier for each transaction. Data Type: Numeric;

Transaction_Date:

Description: Date of the transaction. Data Type: Date;

Product_SKU:

Description: Stock Keeping Unit (SKU) identifier for the product. Data Type: Text;

Product_Description:

Description: Description of the product. Data Type: Text;

Product_Category:

Description: Category to which the product belongs. Data Type: Categorical;

Quantity:

Description: Quantity of the product purchased in the transaction. Data Type: Numeric;

Avg_Price:

Description: Average price of the product. Data Type: Numeric;

Delivery_Charges:

Description: Charges associated with the delivery of the product. Data Type: Numeric;

Coupon_Status:

Description: Status of the coupon associated with the transaction. Data Type: Categorical;

GST:

Description: Goods and Services Tax associated with the transaction. Data Type: Numeric;

Date:

Description: Date of the transaction (potentially redundant with Transaction_Date). Data Type: Date;

Offline_Spend:

Description: Amount spent offline by the customer. Data Type: Numeric;

Online_Spend:

Description: Amount spent online by the customer. Data Type: Numeric;

Month:

Description: Month of the transaction. Data Type: Categorical;

Coupon_Code:

Description: Code associated with a coupon, if applicable. Data Type: Text;

Discount_pct:

Description: Percentage of discount applied to the transaction. Data Type: Numeric;

The Databases_DBMS_2024 dataset provides information about leading databases with a worldwide footprint.

The dataset contains records of 417 databases and has information about the DBMS type, multi-model capability, vendor, and vendor country.

The dataset also contains data on DBMS score and rankings, from DB-engines.com.

Kagglers can utilise the dataset to explore the

• Composition of DBMS Types and Multi-model capability
• Distribution of DBMS vendors and Vendor countries, etc.
• Trends and patterns in DBMS rankings and scores

Context

Kaggle is one of the largest communities of data scientists and machine learning practitioners in the world, and its platform hosts thousands of datasets covering a wide range of topics and industries. With so many options to choose from, it can be difficult to know where to start or what datasets are worth exploring. That's where this dataset comes in. By scraping information about the top 10,000 datasets on Kaggle, we have created a single source of truth for the most popular and useful datasets on the platform. This dataset is not just a list of names and numbers, but a valuable tool for data enthusiasts and professionals alike, providing insights into the latest trends and techniques in data science and machine learning

Column description - Dataset_name - Name of the dataset - Author_name - Name of the author - Author_id - Kaggle id of the author - No_of_files - Number of files the author has uploaded - size - Size of all the files - Type_of_file - Type of the files such as csv, json etc. - Upvotes - Total upvotes of the dataset - Medals - Medal of the dataset - Usability - Usability of the dataset - Date - Date in which the dataset is uploaded - Day - Day in which the dataset is uploaded - Time - Time in which the dataset is uploaded - Dataset_link - Kaggle link of the dataset

Acknowledgements The data has been scraped from the official Kaggle Website and is available under the Creative Common License.

Enjoy & Keep Learning !!!

Context

Stack Overflow is the largest online community for programmers to learn, share their knowledge, and advance their careers.

Content

Updated on a quarterly basis, this BigQuery dataset includes an archive of Stack Overflow content, including posts, votes, tags, and badges. This dataset is updated to mirror the Stack Overflow content on the Internet Archive, and is also available through the Stack Exchange Data Explorer.

Fork this kernel to get started with this dataset.

Acknowledgements

Dataset Source: https://archive.org/download/stackexchange

https://bigquery.cloud.google.com/dataset/bigquery-public-data:stackoverflow

https://cloud.google.com/bigquery/public-data/stackoverflow

Banner Photo by Caspar Rubin from Unplash.

Inspiration

What is the percentage of questions that have been answered over the years?

What is the reputation and badge count of users across different tenures on StackOverflow?

What are 10 of the “easier” gold badges to earn?

Which day of the week has most questions answered within an hour?

Supply chain analytics is a valuable part of data-driven decision-making in various industries such as manufacturing, retail, healthcare, and logistics. It is the process of collecting, analyzing and interpreting data related to the movement of products and services from suppliers to customers.

Sunset

As of Nov 29, 2023, it has been more than a month since someone has downloaded this dataset, so I will be sunsetting this and wont be updating this anymore.

Context

I've ported the official Meta-Kaggle dataset to a SQLITE DB. No 3rd party SQL engine is needed.

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F6c0345968381127a5e4e03a5b237ab48%2Fsqllite.png?generation=1693744427261509&alt=media" alt="">

Recommendation

• Preferably, a lot of RAM and fast SSD

Table Names

They can be found on the official Kaggle Dataset

(QUOTED FROM THE ORIGINAL DATASET)

Meta Kaggle

Explore Kaggle's public data on competitions, datasets, kernels (code/ notebooks) and more Meta Kaggle may not be the Rosetta Stone of data science, but they think there's a lot to learn (and plenty of fun to be had) from this collection of rich data about Kaggle’s community and activity.

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F2ad97bce7839d6e57674e7a82981ed23%2F2Egeb8R.png?generation=1688912953875842&alt=media" alt="">

Notes

• I repeat, I just ported the dataset. All credits to Kaggle for the amazing source dataset

This dataset can be used to train an Open Book model for Kaggle's LLM Science Exam competition. This dataset was generated by searching and concatenating all publicly shared datasets on Sept 1 2023.

The context column was generated using Mgoksu's notebook here with NUM_TITLES=5 and NUM_SENTENCES=20

The source column indicates where the dataset originated. Below are the sources:

source = 1 & 2 * Radek's 6.5k dataset. Discussion here annd here, dataset here.

source = 3 & 4 * Radek's 15k + 5.9k. Discussion here and here, dataset here

source = 5 & 6 * Radek's 6k + 6k. Discussion here and here, dataset here

source = 7 * Leonid's 1k. Discussion here, dataset here

source = 8 * Gigkpeaeums 3k. Discussion here, dataset here

source = 9 * Anil 3.4k. Discussion here, dataset here

source = 10, 11, 12 * Mgoksu 13k. Discussion here, dataset here

Description

This comprehensive dataset provides a wealth of information about all countries worldwide, covering a wide range of indicators and attributes. It encompasses demographic statistics, economic indicators, environmental factors, healthcare metrics, education statistics, and much more. With every country represented, this dataset offers a complete global perspective on various aspects of nations, enabling in-depth analyses and cross-country comparisons.

Key Features

• Country: Name of the country.
• Density (P/Km2): Population density measured in persons per square kilometer.
• Abbreviation: Abbreviation or code representing the country.
• Agricultural Land (%): Percentage of land area used for agricultural purposes.
• Land Area (Km2): Total land area of the country in square kilometers.
• Armed Forces Size: Size of the armed forces in the country.
• Birth Rate: Number of births per 1,000 population per year.
• Calling Code: International calling code for the country.
• Capital/Major City: Name of the capital or major city.
• CO2 Emissions: Carbon dioxide emissions in tons.
• CPI: Consumer Price Index, a measure of inflation and purchasing power.
• CPI Change (%): Percentage change in the Consumer Price Index compared to the previous year.
• Currency_Code: Currency code used in the country.
• Fertility Rate: Average number of children born to a woman during her lifetime.
• Forested Area (%): Percentage of land area covered by forests.
• Gasoline_Price: Price of gasoline per liter in local currency.
• GDP: Gross Domestic Product, the total value of goods and services produced in the country.
• Gross Primary Education Enrollment (%): Gross enrollment ratio for primary education.
• Gross Tertiary Education Enrollment (%): Gross enrollment ratio for tertiary education.
• Infant Mortality: Number of deaths per 1,000 live births before reaching one year of age.
• Largest City: Name of the country's largest city.
• Life Expectancy: Average number of years a newborn is expected to live.
• Maternal Mortality Ratio: Number of maternal deaths per 100,000 live births.
• Minimum Wage: Minimum wage level in local currency.
• Official Language: Official language(s) spoken in the country.
• Out of Pocket Health Expenditure (%): Percentage of total health expenditure paid out-of-pocket by individuals.
• Physicians per Thousand: Number of physicians per thousand people.
• Population: Total population of the country.
• Population: Labor Force Participation (%): Percentage of the population that is part of the labor force.
• Tax Revenue (%): Tax revenue as a percentage of GDP.
• Total Tax Rate: Overall tax burden as a percentage of commercial profits.
• Unemployment Rate: Percentage of the labor force that is unemployed.
• Urban Population: Percentage of the population living in urban areas.
• Latitude: Latitude coordinate of the country's location.
• Longitude: Longitude coordinate of the country's location.

Potential Use Cases

• Analyze population density and land area to study spatial distribution patterns.
• Investigate the relationship between agricultural land and food security.
• Examine carbon dioxide emissions and their impact on climate change.
• Explore correlations between economic indicators such as GDP and various socio-economic factors.
• Investigate educational enrollment rates and their implications for human capital development.
• Analyze healthcare metrics such as infant mortality and life expectancy to assess overall well-being.
• Study labor market dynamics through indicators such as labor force participation and unemployment rates.
• Investigate the role of taxation and its impact on economic development.
• Explore urbanization trends and their social and environmental consequences.

Data Source: This dataset was compiled from multiple data sources

If this was helpful, a vote is appreciated ❤️ Thank you 🙂