Data Analysis is the process that supports decision-making and informs arguments in empirical studies. Descriptive statistics, Exploratory Data Analysis (EDA), and Confirmatory Data Analysis (CDA) are the approaches that compose Data Analysis (Xia & Gong; 2014). An Exploratory Data Analysis (EDA) comprises a set of statistical and data mining procedures to describe data. We ran EDA to provide statistical facts and inform conclusions. The mined facts allow attaining arguments that would influence the Systematic Literature Review of DL4SE.

The Systematic Literature Review of DL4SE requires formal statistical modeling to refine the answers for the proposed research questions and formulate new hypotheses to be addressed in the future. Hence, we introduce DL4SE-DA, a set of statistical processes and data mining pipelines that uncover hidden relationships among Deep Learning reported literature in Software Engineering. Such hidden relationships are collected and analyzed to illustrate the state-of-the-art of DL techniques employed in the software engineering context.

Our DL4SE-DA is a simplified version of the classical Knowledge Discovery in Databases, or KDD (Fayyad, et al; 1996). The KDD process extracts knowledge from a DL4SE structured database. This structured database was the product of multiple iterations of data gathering and collection from the inspected literature. The KDD involves five stages:

Selection. This stage was led by the taxonomy process explained in section xx of the paper. After collecting all the papers and creating the taxonomies, we organize the data into 35 features or attributes that you find in the repository. In fact, we manually engineered features from the DL4SE papers. Some of the features are venue, year published, type of paper, metrics, data-scale, type of tuning, learning algorithm, SE data, and so on.

Preprocessing. The preprocessing applied was transforming the features into the correct type (nominal), removing outliers (papers that do not belong to the DL4SE), and re-inspecting the papers to extract missing information produced by the normalization process. For instance, we normalize the feature “metrics” into “MRR”, “ROC or AUC”, “BLEU Score”, “Accuracy”, “Precision”, “Recall”, “F1 Measure”, and “Other Metrics”. “Other Metrics” refers to unconventional metrics found during the extraction. Similarly, the same normalization was applied to other features like “SE Data” and “Reproducibility Types”. This separation into more detailed classes contributes to a better understanding and classification of the paper by the data mining tasks or methods.

Transformation. In this stage, we omitted to use any data transformation method except for the clustering analysis. We performed a Principal Component Analysis to reduce 35 features into 2 components for visualization purposes. Furthermore, PCA also allowed us to identify the number of clusters that exhibit the maximum reduction in variance. In other words, it helped us to identify the number of clusters to be used when tuning the explainable models.

Data Mining. In this stage, we used three distinct data mining tasks: Correlation Analysis, Association Rule Learning, and Clustering. We decided that the goal of the KDD process should be oriented to uncover hidden relationships on the extracted features (Correlations and Association Rules) and to categorize the DL4SE papers for a better segmentation of the state-of-the-art (Clustering). A clear explanation is provided in the subsection “Data Mining Tasks for the SLR od DL4SE”. 5.Interpretation/Evaluation. We used the Knowledge Discover to automatically find patterns in our papers that resemble “actionable knowledge”. This actionable knowledge was generated by conducting a reasoning process on the data mining outcomes. This reasoning process produces an argument support analysis (see this link).

We used RapidMiner as our software tool to conduct the data analysis. The procedures and pipelines were published in our repository.

Overview of the most meaningful Association Rules. Rectangles are both Premises and Conclusions. An arrow connecting a Premise with a Conclusion implies that given some premise, the conclusion is associated. E.g., Given that an author used Supervised Learning, we can conclude that their approach is irreproducible with a certain Support and Confidence.

Support = Number of occurrences this statement is true divided by the amount of statements Confidence = The support of the statement divided by the number of occurrences of the premise

Thorough knowledge of the structure of analyzed data allows to form detailed scientific hypotheses and research questions. The structure of data can be revealed with methods for exploratory data analysis. Due to multitude of available methods, selecting those which will work together well and facilitate data interpretation is not an easy task. In this work we present a well fitted set of tools for a complete exploratory analysis of a clinical dataset and perform a case study analysis on a set of 515 patients. The proposed procedure comprises several steps: 1) robust data normalization, 2) outlier detection with Mahalanobis (MD) and robust Mahalanobis distances (rMD), 3) hierarchical clustering with Ward’s algorithm, 4) Principal Component Analysis with biplot vectors. The analyzed set comprised elderly patients that participated in the PolSenior project. Each patient was characterized by over 40 biochemical and socio-geographical attributes. Introductory analysis showed that the case-study dataset comprises two clusters separated along the axis of sex hormone attributes. Further analysis was carried out separately for male and female patients. The most optimal partitioning in the male set resulted in five subgroups. Two of them were related to diseased patients: 1) diabetes and 2) hypogonadism patients. Analysis of the female set suggested that it was more homogeneous than the male dataset. No evidence of pathological patient subgroups was found. In the study we showed that outlier detection with MD and rMD allows not only to identify outliers, but can also assess the heterogeneity of a dataset. The case study proved that our procedure is well suited for identification and visualization of biologically meaningful patient subgroups.

Dataset

This dataset was created by Monis Ahmad

Contents

Airbnb® is an American company operating an online marketplace for lodging, primarily for vacation rentals. The purpose of this study is to perform an exploratory data analysis of the two datasets containing Airbnb® listings and across 10 major cities. We aim to use various data visualizations to gain valuable insight on the effects of pricing, covid, and more!

Despite exploratory data analysis (EDA) is a powerful approach for uncovering insights from unfamiliar datasets, existing EDA tools face challenges in assisting users to assess the progress of exploration and synthesize coherent insights from isolated findings. To address these challenges, we present FactExplorer, a novel fact-based EDA system that shifts the analysis focus from raw data to data facts. FactExplorer employs a hybrid logical-visual representation, providing users with a comprehensive overview of all potential facts at the outset of their exploration. Moreover, FactExplorer introduces fact-mining techniques, including topic-based drill-down and transition path search capabilities. These features facilitate in-depth analysis of facts and enhance the understanding of interconnections between specific facts. Finally, we present a usage scenario and conduct a user study to assess the effectiveness of FactExplorer. The results indicate that FactExplorer facilitates the understanding of isolated findings and enables users to steer a thorough and effective EDA.

Dataset

This dataset was created by Mohammad Osama

Contents

Diabetes Dataset — Exploratory Data Analysis (EDA)

This repository contains a diabetes-related tabular dataset and a complete Exploratory Data Analysis (EDA).The main objective of this project was to learn how to conduct a structured EDA, apply best practices, and extract meaningful insights from real-world health data.
The analysis includes correlations, distributions, group comparisons, class balance exploration, and statistical interpretations that illustrate how different… See the full description on the dataset page: https://huggingface.co/datasets/guyshilo12/diabetes_eda_analysis.

Context

The What to do in Paris site is a participative agenda, Parisian places such as the City Libraries and Museums, parks and gardens, entertainment centers, swimming pools, theaters, major venues such as the Gaîté Lyrique, the CENTQUATRE, the Carreau du Temple, concert halls, associations and even Parisians are invited to insert their events in the site.

The source of the data

Content

What's inside is more than just rows and columns. Make it easy for others to get started by describing how you acquired the data and what time period it represents, too.

Acknowledgements

We wouldn't be here without the help of others. If you owe any attributions or thanks, include them here along with any citations of past research.

Inspiration

Your data will be in front of the world's largest data science community. What questions do you want to see answered?

I did exploratory data analysis using this data.

Offers a window into the world of bank telemarketing, with the goal of understanding how customers respond to campaigns promoting term deposit subscriptions. It provides a rich collection of information, including:

Customer Demographics: A snapshot of who your customers are (age, job, marital status, etc.). Campaign History: Insights into how customers have reacted to past campaigns (contact method, duration). Call Metrics: Data on call duration and conversion rates, both on an individual call level and overall. Originally sourced from a public repository, this dataset offers valuable potential for analysis. It's perfect for exploring: Customer Behavior: What are the characteristics of customers who do (and don't) sign up for term deposits? Campaign Effectiveness: Which types of campaigns or communication strategies are most successful?

By conducting exploratory data analysis (including univariate, bivariate, and segmented approaches), you can uncover hidden patterns and optimize future marketing efforts. This data is your key to better understanding your customers and driving higher subscription rates.

Context

This data is publicly available on GitHub here. It can be utilized for EDA, Statistical Analysis, and Visualizations.

Content

The data set ifood_df.csv consists of 2206 customers of XYZ company with data on: - Customer profiles - Product preferences - Campaign successes/failures - Channel performance

Acknowledgement

I do not own this dataset. I am simply making it accessible on this platform via the public GitHub link.

The Smart Energy Research Lab Exploratory Data, 2019-2020 is an initial study part of the SERL project and is to be accessed by SERL researchers to conduct exploratory analysis ahead of provisioning SERL data to the wider academic research community.

The goals of the SERL portal are to provide:

• a consistent, trusted, and sustainable channel for researchers to access large-scale, high-resolution energy data, thereby providing a reliable empirical dataset for research;
• an effective mechanism for collecting energy data alongside other variables from national surveys (e.g. English Housing Survey) or individual research projects
• a confidential, ongoing repository of smart meter data enhanced with contextual dwelling, household and neighbourhood attributes for use in primary and secondary data analysis.

Further information about SERL can be found on "https://serl.ac.uk/" target="_blank"> serl.ac.uk.

ABSTRACT Context: exploratory factor analysis (EFA) is one of the statistical methods most widely used in administration; however, its current practice coexists with rules of thumb and heuristics given half a century ago. Objective: the purpose of this article is to present the best practices and recent recommendations for a typical EFA in administration through a practical solution accessible to researchers. Methods: in this sense, in addition to discussing current practices versus recommended practices, a tutorial with real data on Factor is illustrated. The Factor software is still little known in the administration area, but is freeware, easy-to-use (point and click), and powerful. The step-by-step tutorial illustrated in the article, in addition to the discussions raised and an additional example, is also available in the format of tutorial videos. Conclusion: through the proposed didactic methodology (article-tutorial + video-tutorial), we encourage researchers/methodologists who have mastered a particular technique to do the same. Specifically about EFA, we hope that the presentation of the Factor software, as a first solution, can transcend the current outdated rules of thumb and heuristics, by making best practices accessible to administration researchers.

The technological advancements of the modern era have enabled the collection of huge amounts of data in science and beyond. Extracting useful information from such massive datasets is an ongoing challenge as traditional data visualization tools typically do not scale well in high-dimensional settings. An existing visualization technique that is particularly well suited to visualizing large datasets is the heatmap. Although heatmaps are extremely popular in fields such as bioinformatics, they remain a severely underutilized visualization tool in modern data analysis. This article introduces superheat, a new R package that provides an extremely flexible and customizable platform for visualizing complex datasets. Superheat produces attractive and extendable heatmaps to which the user can add a response variable as a scatterplot, model results as boxplots, correlation information as barplots, and more. The goal of this article is two-fold: (1) to demonstrate the potential of the heatmap as a core visualization method for a range of data types, and (2) to highlight the customizability and ease of implementation of the superheat R package for creating beautiful and extendable heatmaps. The capabilities and fundamental applicability of the superheat package will be explored via three reproducible case studies, each based on publicly available data sources.

Assignment 1: EDA - US Company Bankruptcy Prediction

Student Name: Reef Zehavi Date: November 10, 2025

  📹 Project Presentation Video

[(https://www.loom.com/share/6920e493e8654ef3bb4f67a10eb9b03d)]

  1. Overview and Project Goal

The goal of this project is to perform Exploratory Data Analysis (EDA) on a fundamental dataset of American companies. The analysis focuses on understanding the financial characteristics that differentiate between companies that survived… See the full description on the dataset page: https://huggingface.co/datasets/reefzehavi/EDA-US-Bankruptcy-Prediction.

Dataset Overview: This dataset contains transactional data of customers from an e-commerce platform, to analyze and understand their purchasing behavior. The dataset includes customer ID, product purchased, purchase amount, purchase date, and product category.

Purpose of the Dataset: The primary objective of this dataset is to provide an opportunity to perform data exploration and preprocessing, allowing users to practice and enhance their data cleaning and analysis skills. The dataset has been intentionally modified to simulate a "messy" scenario, where some values have been removed, and inconsistencies have been introduced, which provides a real-world challenge for users to handle during data preparation.

Key Features: CustomerID: Unique identifier for each customer. ProductID: Unique identifier for each product purchased. PurchaseAmount: Amount spent by the customer on a particular transaction. PurchaseDate: Date when the transaction took place. ProductCategory: Category of the purchased product.

Analysis Opportunities:

Perform data cleaning and preprocessing to handle missing values, duplicates, and outliers. Conduct exploratory data analysis (EDA) to uncover trends and patterns in customer behavior. Apply machine learning models like clustering and association rule mining for segmenting customers and understanding purchasing patterns.

This dataset provides data about an exploratory research that analyzed the Privacy and Security Policies and the Instruction Manuals of 59 home automation equipment for Smart Home in order to verify which personal data was handled and how these documents were providing information about processes performed in personal data. The analysis was conducted with a quantitative approach followed by a qualitative analysis, using content analysis.

This Dataset consists of two categories of Data, namely - Primary Data obtained through survey responses and Secondary Data obtained through Websites and APIs.

This Dataset contains categories of evaluation, to name a few: 1) Employment Status 2) Job Satisfaction 3) Retention of Key Employees

Context

I was reading Every Nose Counts: Using Metrics in Animal Shelters when I got inspired to conduct an EDA on animal shelter data. I looked online for data and found this dataset which is curated by Austin Animal Center. The data can be found on https://data.austintexas.gov.

This data can be utilized for EDA practice. So go ahead and help animal shelters with your EDA powers by completing this task!

Content

The data set contains three CSVs: 1. Austin_Animal_Center_Intakes.csv 2. Austin_Animal_Center_Outcomes.csv 3. Austin_Animal_Center_Stray_Map.csv

More TBD!

Acknowledgement

Thank you Austin Animal Center for all the animal protection you provide to stray & owned animals. Also, thank you for making your data accessible to the public.

PLEASE UPVOTE THIS DATASET IF THIS HELP YOU... GLAD TO ANY FORKS HERE

BACKGROUND DQLab Telco is a telecommunications company with numerous locations all over the world. In order to ensure that customers are not left behind, DQLab Telco has consistently paid attention to the customer experience since its establishment in 2019.

Even though DQLab Telco is only a little over a year old, many of its customers have already changed their subscriptions to rival companies. By using machine learning, management hopes to lower the number of customers who leave.

After cleaning the data yesterday, it is now time for us to build the best model to forecast customer churn.

TASKS & STEPS Yesterday, we completed "Cleansing Data" as part of project part 1. You are now expected to develop the appropriate model as a data scientist.

You will perform "Machine Learning Modeling" in this assignment using data from the previous month, specifically June 2020.

The actions that must be taken are, 1. Analyze exploratory data first. 2. Carry out pre-processing of the data. 3. Using modeling from machine learning. 4. Picking the Ideal Model.