Explore our public notebook content!

Meta Kaggle Code is an extension to our popular Meta Kaggle dataset. This extension contains all the raw source code from hundreds of thousands of public, Apache 2.0 licensed Python and R notebooks versions on Kaggle used to analyze Datasets, make submissions to Competitions, and more. This represents nearly a decade of data spanning a period of tremendous evolution in the ways ML work is done.

Why we’re releasing this dataset

By collecting all of this code created by Kaggle’s community in one dataset, we hope to make it easier for the world to research and share insights about trends in our industry. With the growing significance of AI-assisted development, we expect this data can also be used to fine-tune models for ML-specific code generation tasks.

Meta Kaggle for Code is also a continuation of our commitment to open data and research. This new dataset is a companion to Meta Kaggle which we originally released in 2016. On top of Meta Kaggle, our community has shared nearly 1,000 public code examples. Research papers written using Meta Kaggle have examined how data scientists collaboratively solve problems, analyzed overfitting in machine learning competitions, compared discussions between Kaggle and Stack Overflow communities, and more.

The best part is Meta Kaggle enriches Meta Kaggle for Code. By joining the datasets together, you can easily understand which competitions code was run against, the progression tier of the code’s author, how many votes a notebook had, what kinds of comments it received, and much, much more. We hope the new potential for uncovering deep insights into how ML code is written feels just as limitless to you as it does to us!

Sensitive data

While we have made an attempt to filter out notebooks containing potentially sensitive information published by Kaggle users, the dataset may still contain such information. Research, publications, applications, etc. relying on this data should only use or report on publicly available, non-sensitive information.

Joining with Meta Kaggle

The files contained here are a subset of the KernelVersions in Meta Kaggle. The file names match the ids in the KernelVersions csv file. Whereas Meta Kaggle contains data for all interactive and commit sessions, Meta Kaggle Code contains only data for commit sessions.

File organization

The files are organized into a two-level directory structure. Each top level folder contains up to 1 million files, e.g. - folder 123 contains all versions from 123,000,000 to 123,999,999. Each sub folder contains up to 1 thousand files, e.g. - 123/456 contains all versions from 123,456,000 to 123,456,999. In practice, each folder will have many fewer than 1 thousand files due to private and interactive sessions.

The ipynb files in this dataset hosted on Kaggle do not contain the output cells. If the outputs are required, the full set of ipynbs with the outputs embedded can be obtained from this public GCS bucket: kaggle-meta-kaggle-code-downloads. Note that this is a "requester pays" bucket. This means you will need a GCP account with billing enabled to download. Learn more here: https://cloud.google.com/storage/docs/requester-pays

Questions / Comments

We love feedback! Let us know in the Discussion tab.

Happy Kaggling!

KGTorrent is a dataset of Python Jupyter notebooks from the Kaggle platform.

The dataset is accompanied by a MySQL database containing metadata about the notebooks and the activity of Kaggle users on the platform. The information to build the MySQL database has been derived from Meta Kaggle, a publicly available dataset containing Kaggle metadata.

In this package, we share the complete KGTorrent dataset (consisting of the dataset itself plus its companion database), as well as the specific version of Meta Kaggle used to build the database.

More specifically, the package comprises the following three compressed archives:

1. KGT_dataset.tar.bz2, the dataset of Jupyter notebooks;

2. KGTorrent_dump_10-2020.sql.tar.bz2, the dump of the MySQL companion database;

3. MetaKaggle27Oct2020.tar.bz2, a copy of the Meta Kaggle version used to build the database.

Moreover, we include KGTorrent_logical_schema.pdf, the logical schema of the KGTorrent MySQL database.

GitHub Issues & Kaggle Notebooks

  Description

GitHub Issues & Kaggle Notebooks is a collection of two code datasets intended for language models training, they are sourced from GitHub issues and notebooks in Kaggle platform. These datasets are a modified part of the StarCoder2 model training corpus, precisely the bigcode/StarCoder2-Extras dataset. We reformat the samples to remove StarCoder2's special tokens and use natural text to delimit comments in issues and display… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceTB/issues-kaggle-notebooks.

Kaggle Notebooks LLM Filtered

Model: meta-llama/Meta-Llama-3.1-70B-Instruct Sample: 12,400 Source dataset: data-agents/kaggle-notebooks Prompt:

Below is an extract from a Jupyter notebook. Evaluate whether it has a high analysis value and could help a data scientist.

The notebooks are formatted with the following tokens:

START

Here comes markdown content

Here comes python code

Here comes code output

More… See the full description on the dataset page: https://huggingface.co/datasets/data-agents/kaggle-notebooks-edu-v0.

Overview

DistilKaggle is a curated dataset extracted from Kaggle Jupyter notebooks spanning from September 2015 to October 2023. This dataset is a distilled version derived from the download of over 300GB of Kaggle kernels, focusing on essential data for research purposes. The dataset exclusively comprises publicly available Python Jupyter notebooks from Kaggle. The essential information for retrieving the data needed to download the dataset is obtained from the MetaKaggle dataset provided by Kaggle.

Contents

The DistilKaggle dataset consists of three main CSV files:

code.csv: Contains over 12 million rows of code cells extracted from the Kaggle kernels. Each row is identified by the kernel's ID and cell index for reproducibility.

markdown.csv: Includes over 5 million rows of markdown cells extracted from Kaggle kernels. Similar to code.csv, each row is identified by the kernel's ID and cell index.

notebook_metrics.csv: This file provides notebook features described in the accompanying paper released with this dataset. It includes metrics for over 517,000 Python notebooks.

Directory Structure

The kernels directory is organized based on Kaggle's Performance Tiers (PTs), a ranking system in Kaggle that classifies users. The structure includes PT-specific directories, each containing user ids that belong to this PT, download logs, and the essential data needed for downloading the notebooks.

The utility directory contains two important files:

aggregate_data.py: A Python script for aggregating data from different PTs into the mentioned CSV files.

application.ipynb: A Jupyter notebook serving as a simple example application using the metrics dataframe. It demonstrates predicting the PT of the author based on notebook metrics.

DistilKaggle.tar.gz: It is just the compressed version of the whole dataset. If you downloaded all of the other files independently already, there is no need to download this file.

Usage

Researchers can leverage this distilled dataset for various analyses without dealing with the bulk of the original 300GB dataset. For access to the raw, unprocessed Kaggle kernels, researchers can request the dataset directly.

Note

The original dataset of Kaggle kernels is substantial, exceeding 300GB, making it impractical for direct upload to Zenodo. Researchers interested in the full dataset can contact the dataset maintainers for access.

Citation

If you use this dataset in your research, please cite the accompanying paper or provide appropriate acknowledgment as outlined in the documentation.

If you have any questions regarding the dataset, don't hesitate to contact me at mohammad.abolnejadian@gmail.com

Thank you for using DistilKaggle!

Dataset

This dataset was created by Rahab Uwamahoro

Contents

Latest polars sources. This can be included when you are on an offline notebook.

I'll try to keep this up-to-date, but if there is a newer version, please do tell me and I'll update it.

Current version: 1.31.0

To install in your notebook: jupyter !pip install -q --no-index --find-links /kaggle/input/offline-polars polars

License: MIT

bigcomputer/kaggle-notebooks-outputs-filtered-0 dataset hosted on Hugging Face and contributed by the HF Datasets community

Dataset

This dataset was created by JadenQ

Contents

It contains the following files:

Analysis of ‘Kaggle Notebooks Ranking’ provided by Analyst-2 (analyst-2.ai), based on source dataset retrieved from https://www.kaggle.com/vivovinco/kaggle-notebooks-ranking on 13 February 2022.

--- Dataset description provided by original source is as follows ---

Context

This dataset contains Kaggle ranking of notebooks.

Content

+3000 rows and 8 columns. Columns' description are listed below.

• Rank : Rank of the user
• Tier : Grandmaster, Master or Expert
• Username : Name of the user
• Join Date : Year of join
• Gold Medals : Number of gold medals
• Silver Medals : Number of silver medals
• Bronze Medals : Number of bronze medals
• Points : Total points

Acknowledgements

Data from Kaggle. Image from Wikiwand.

If you're reading this, please upvote.

--- Original source retains full ownership of the source dataset ---

Computational notebooks have become the primary coding environment for data scientists. Despite their popularity, research on the code quality of these notebooks is still in its infancy, and the code shared in these notebooks is often of poor quality. Considering the importance of maintenance and reusability, it is crucial to pay attention to the comprehension of the notebook code and identify the notebook metrics that play a significant role in their comprehension. The level of code comprehension is a qualitative variable closely associated with the user's opinion about the code. Previous studies have typically employed two approaches to measure it. One approach involves using limited questionnaire methods to review a small number of code pieces. Another approach relies solely on metadata, such as the number of likes and user votes for a project in the software repository. In our approach, we enhanced the measurement of the understandability level of notebook code by leveraging user comments within a software repository. As a case study, we started with 248,761 Kaggle Jupyter notebooks introduced in previous studies and their relevant metadata. To identify user comments associated with code comprehension within the notebooks, we utilized a fine-tuned DistillBERT transformer. We established a \emph{user comment based criterion} for measuring code understandability by considering the number of code understandability-related comments, the upvotes on those comments, the total views of the notebook, and the total upvotes received by the notebook. This criterion has proven to be more effective than alternative methods, making it the ground truth for evaluating the code comprehension of our notebook set. In addition, we collected a total of 34 metrics for 10,857 notebooks, categorized as script-based and notebook-based metrics. These metrics were utilized as features in our dataset. Using the Random Forest classifier, our predictive model achieved 85% accuracy in predicting code comprehension levels in computational notebooks, identifying developer expertise and markdown-based metrics as key factors.

This dataset contains a couple of great open source models!

• version 2 -- the best open source LLM at the time of writing (NousResearch/Nous-Hermes-Llama2-13b) that we can load on Kaggle! didn't manage to load anything larger than 13B
• version 14 -- loading models using a new library, curated-transformers that should allow for easier modifications of the underlying architectures.

This dataset also includes all the dependencies we need to load the model in 8bit, if that is what you would like to do (updated version of transfomers, accelerate, etc).

I show how to load and run Nous-Hermes-Llama2-13b in the following notebook:

👉 💡 Best Open Source LLM Starter Pack 🧪🚀

If you find this dataset helpful, please leave an upvote! 🙂 Thank you! 🙏

deberta-billy is trained locally by @hqfang primarily using @radek1's notebook.

deberta-lora-lindsey is trained locally by @lindseywei using the LoRA technique.

deberta-openbook-eric-088 comes from @yuekaixueirc's dataset.

deberta-openbook-eric-0897 comes from @yuekaixueirc's dataset.

deberta-openbook-eric-0916 comes from @yuekaixueirc's dataset.

54k_with_context_v1.csv was created by dropping duplicates @cdeotte's 60k training data all_12_with_context2.csv in this dataset.

54k.csv was created by dropping the context column from the 54k_with_context_v1.csv.

val_with_context_v1.csv was created by adding a context column to @itsuki9180's validation dataset.

Context

A significant amount of software is available in Kaggle's Python notebook. I had hoped to find a reference somewhere listing which Python packages were available and what each one did.

When I didn't find what I was looking for, I decided to build this dataset instead.

Content

This dataset was assembled in four steps:

1. Code inside a Kaggle notebook was used to gather the names of over 600 installed packages.
2. A package list was scraped from Anaconda and cross-referenced against the notebook package list.
3. The roughly 400 packages that remained were carefully queried from the Python Package Index using its JSON API.
4. The results were collated into a manifest.

Reference

• Anaconda's 64-bit Linux Python package list.
• HTML Scraping - The Hitchhiker's Guide to Python
• The PyPI JSON API.
• Rate Limiting for the PyPI API

Acknowledgements

• Thanks to @nagadomi for the original script.
• Thanks to the Kaggle team for creating a powerful notebook environment.
• Photo by j zamora.

Context and Content

Images used in several of my Kaggle notebooks, each source is mentioned explicitly.

This dataset accompanies my notebook 🎨Hidden Gems: SpecialSauces to create amazing EDA specially created as a Guide to create a successful notebook in any Analytics Competition.

Dataset

This dataset was created by Nick Potter

Contents

Analysis of ‘NYC Most Popular Baby Names Over the Years’ provided by Analyst-2 (analyst-2.ai), based on source dataset retrieved from https://www.kaggle.com/yamqwe/most-popular-baby-names-in-nyce on 13 February 2022.

--- Dataset description provided by original source is as follows ---

About this dataset

Popular Baby Name Data In NYC from 2011-2014

Rows: 13962; Columns: 6

The data include items, such as:

• BRTH_YR: birth year the baby
• GNDR: gender
• ETHCTY: mother's ethnicity
• NM: baby's name
• CNT: count of the name
• RNK: ranking of the name

Source: NYC Open Data

https://data.cityofnewyork.us/Health/Most-Popular-Baby-Names-by-Sex-and-Mother-s-Ethnic/25th-nujf

This dataset was created by Data Society and contains around 10000 samples along with Nm, Rnk, technical information and other features such as: - Gndr - Ethcty - and more.

How to use this dataset

• Analyze Brth Yr in relation to Cnt
• Study the influence of Nm on Rnk
• More datasets

Acknowledgements

If you use this dataset in your research, please credit Data Society

Start A New Notebook!

--- Original source retains full ownership of the source dataset ---

Dataset

This dataset was created by yu5uke

Contents

Analysis of ‘🏈 NFL Favorite Team’ provided by Analyst-2 (analyst-2.ai), based on source dataset retrieved from https://www.kaggle.com/yamqwe/nfl-favorite-teame on 28 January 2022.

--- Dataset description provided by original source is as follows ---

About this dataset

See Readme for more details.
This repository contains a selection of the data -- and the data-processing scripts -- behind the articles, graphics and interactives at FiveThirtyEight.

We hope you'll use it to check our work and to create stories and visualizations of your own. The data is available under the Creative Commons Attribution 4.0 International License and the code is available under the MIT License. If you do find it useful, please let us know](andrei.scheinkman@fivethirtyeight.com).

Source: https://github.com/fivethirtyeight/data

This dataset was created by FiveThirtyEight and contains around 0 samples along with Cch, Slp, technical information and other features such as: - Nyp - Beh - and more.

How to use this dataset

• Analyze Cch in relation to Slp
• Study the influence of Nyp on Beh
• More datasets

Acknowledgements

If you use this dataset in your research, please credit FiveThirtyEight

Start A New Notebook!

--- Original source retains full ownership of the source dataset ---