Dataset Card for Deita Complexity Scorer Training Data

GitHub | Paper Deita is an open-sourced project designed to facilitate Automatic Data Selection for instruction tuning in Large Language Models (LLMs). This dataset includes data for training Deita Complexity Scorer. Model Family: Other models and the dataset are found in the Deita Collection

  Performance

Model Align Data Size MT-Bench AlpacaEval(%) OpenLLM (Avg.)

Proprietary Models

GPT-4-Turbo… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/deita-complexity-scorer-data.

Banking data (RIBITS) - freshwater targets and bank specs.

This is the supplemental material for the paper entitled “Handling Data Complexity and Class-imbalance for Software Defect Prediction”.

Data files complexity of network data.

This is the replication package for "Complexity and Sophistication," accepted in 2023 by the Journal of Political Economy Microeconomics.

Features arising following single feature complexity analysis.

Runtimes and number of variables for different state distributions and for 2, 3 and 4 states for covariate model 1. Runtimes are on Intel Xeon E5-2697v2 @ 2.7 GHz.

The metadata set does not comprise any description or summary. The information has not been provided.

Each year, researchers at Harvard's Growth Lab release growth forecasts for the upcoming decade as well as annual rankings of countries by economic complexity. The Economic Complexity Index (ECI) ranking is a measure of the amount of capabilities and knowhow of a given country determined by the diversity, ubiquity, and complexity of the products it exports. Growth projections are calculated through a process largely based on determining whether a country's economic complexity is higher or lower than expected given its level of income. We expect countries whose economic complexity is greater than we would expect for its level of income to grow faster than those that are "too rich" for their current level of complexity. In this data, a country's growth projection value for a given year is for the decade beginning with that year. For example, a value in a 2017 row is the projection of annualized growth for 2017–2027.

Data and code to accompany Lunt et al. "Background choice is mediated by complexity in cephalopods" Complete download (zip, 2.4 MiB)

Perceptual load is a well-established determinant of attentional engagement in a task. So far, perceptual load has typically been manipulated by increasing either the number of task-relevant items or the perceptual processing demand (e.g. conjunction vs. feature tasks). The tasks used often involved rather simple visual displays (e.g. letters or single objects). How can perceptual load be operationalised for richer, real-world images? A promising proxy is the visual complexity of an image. However, current predictive models for visual complexity have limited applicability to diverse real-world images. Here we modelled visual complexity using a deep convolutional neural network trained to learn perceived ratings of visual complexity. We presented 53 observers with 4000 images from the PASCAL VOC dataset, obtaining 75,020 2AFC paired comparisons across observers. Image visual complexity scores were obtained using the TrueSkill algorithm. A CNN with weights pre-trained on an object recognition task predicted complexity ratings with r=0.83. In contrast, feature-based models as used in the literature, working on image statistics such as entropy, edge density and JPEG compression ratio, only achieved r = 0.70. Thus, our model offers a promising method to quantify the perceptual load of real-world scenes through visual complexity.

Thresholds for single feature complexity.

Different classes of quantifiers provably require different verification algorithms with different complexity profiles. The algorithm for proportional quantifiers, like most', is more complex than that for nonproportional quantifiers, likeall' and `three'. We tested the hypothesis that different complexity profiles affect ERP responses during sentence verification, but not during sentence comprehension. In experiment 1, participants had to determine the truth value of a sentence relative to a previously presented array of geometric objects. We observed a sentence-final negative effect of truth value, modulated by quantifier class. Proportional quantifiers elicited a sentence-internal positivity compared to nonproportional quantifiers, in line with their different verification profiles. In experiment 2, the same stimuli were shown, followed by comprehension questions instead of verification. ERP responses specific to proportional quantifiers disappeared in experiment 2, suggesting that they are only evoked in a verification task and thus reflect the verification procedure itself. The present dataset contains behavioural and EEG data from both experiments, as well as analysis scripts for both data types in R and Matlab/FieldTrip.

Revised data and code to accompany Lunt et al. "Intensity contrast drives background choice in cephalopods"

The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management

This dataset contains our raw experimental data (ie. agent trajectories) accompanying the paper "The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management" and Tobias Lindenbauer's Master's thesis. The data in this repository are compressed to .tar.gz archives. For detailed instructions on how to use these data… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/the-complexity-trap.

Research data, sources and documents for thesis on Exploring Complexity Metrics for Artifact-Centric Business Process Models This repository contains the supplemental material for the thesis "Exploring Complexity Metrics for Artifact-Centric Business Process Models" by Marin, Mike A., Ph.D., University of South Africa (South Africa), 2017.

This dataset compiles key indicators of global trade and economic complexity, curated from the Harvard Growth Lab's Atlas of Economic Complexity.

Data spans multiple classification systems (HS12, HS92, SITC, and Services), enabling a wide range of cross-national and historical trade analyses.

Data Sources

All data is directly downloaded from: The Atlas of Economic Complexity: Downloads - Rankings This project repackages publicly available data into a Kaggle-friendly format for exploration and analysis.

The Dataset 2: annotated corpora by level of complexity for FR, PT and SP is a collection of texts categorized by complexity level and annotated for complexity features, presented in Excel format (.xlsx). These corpora were compiled and annotated under the scope of the project iRead4Skills – Intelligent Reading Improvement System for Fundamental and Transversal Skills Development, funded by the European Commission (grant number: 1010094837). The project aims to enhance reading skills within the adult population by creating an intelligent system that assesses text complexity and recommends suitable reading materials to adults with low literacy skills, contributing to reducing skills gaps and facilitating access to information and culture (https://iread4skills.com).

This dataset is the result of specifically devised classification and annotation tasks, in which selected texts were organized and distributed to trainers in Adult Learning (AL) and Vocational Education Training (VET) Centres, as well as to adult students in AL and VET centres. This task was conducted via the Qualtrics platform.

The Dataset 2: annotated corpora by level of complexity for FR, PT and SP is derived from the iRead4Skills Dataset 1: corpora by level of complexity for FR, PT and SP ( https://doi.org/10.5281/zenodo.10055909), which comprises written texts of various genres and complexity levels. From this collection, a sample of texts was selected for classification and annotation. This classification and annotation task aimed to provide additional data and test sets for the complexity analysis systems for the three languages of the project: French, Portuguese, and Spanish. The sample texts in each of the language corpora were selected taking into account the diversity of topics/domains, genres, and the reading preferences of the target audience of the iRead4Skills project. This percentage amounted to the total of 462 texts per language, which were divided by level of complexity, resulting in the following distribution:

· 140 Very Easy texts

· 140 Easy texts

· 140 Plain texts

· 42 More Complex texts.

Trainers and students were asked to classify the texts according to the complexity levels of the project, here informally defined as:

· Very Easy (everyone can understand the text or most of the text).

· Easy (a person with less than the 9th year of schooling can understand the text or most of the text)

· Plain (a person with the 9th year of schooling can understand the text the first time he/she reads it)

· More complex (a person with the 9th year of schooling cannot understand the text the first time he/she reads it).

Annotators were also asked to mark the parts of the texts considered complex according to various type of features, at word-level and at sentence-level (e.g., word order, sentence composition, etc.), The full details regarding the students and the trainers’ tasks, data qualitative and quantitative description and inter-annotator agreement are described here: https://zenodo.org/records/14653180

The results are here presented in Excel format. For each language, and for each group (trainers and students), two pairs of files exist – the annotation and the classification files – resulting in four files per language and twelve files, in total.

In all files, the data is organized as a matrix, with each row representing an ‘answer’ from a particular participant, and the columns containing various details about that specific input, as shown below:

Column name

Data

Annotator's ID

The randomly generated ID code for each annotator, together with information on the dataset assigned to them.

Progress

Information on the completion of the task (for each text).

Duration (seconds)

Time used in the completion of the task (for each text).

File Name

N1 = Very Easy

N2 = Easy

N3 = Plain

N4=More Complex

File internal identification, providing its iRead4Skills classification.

Text

The content of the file, i.e. the text itself.

Annotated Level

Level assigned by the annotator (trainer).

Proficiency SubLevel

(Likert Scale - 1 to 5)

SubLevel assigned by the annotator (trainer) for FR data.

Corresponding CEFR Level

CEFR level closest to the iRead4Skills

Additional Info

Observations made by the trainers/students

Annotated Term

Word or set of words selected for annotation

Term Label

Annotation assigned to the Annotated Term (difficult word, word order, etc.)

Term Index

Position of the annotated term in the text

Annotator's Proficiency Level

Level of AL/VET of the student

Text adequate for user

Validation of the text by the students

The content of the column “File Name” is color-coded, where a green shade alludes to a text with a lower level of complexity and a red one alludes to one with a higher level of complexity.

The complete datasets are available under creative CC BY-NC-ND 4.0.

Graph representation learning—especially via graph neural networks (GNNs)—has demonstrated considerable promise in modeling intricate interaction systems, such as social networks and molecular structures. However, the deployment of GNN-based frameworks in industrial settings remains challenging due to the inherent complexity and noise in real-world graph data. This dissertation systematically addresses these challenges by advancing novel methodologies to improve the comprehensiveness and robustness of graph representation learning, with a dual focus on resolving data complexity and denoising across diverse graph-learning scenarios. In addressing graph data denoising, we design auxiliary self-supervised optimization objectives that disentangle noisy topological structures and misinformation while preserving the representational sufficiency of critical graph features. These tasks operate synergistically with primary learning objectives to enhance robustness against data corruption. The efficacy of these techniques is demonstrated through their application to real-world opioid prescription time series data for predicting potential opioid over-prescription. To mitigate data complexity, the study investigates two complementary approaches: (1) multimodal fusion, which employs attentive integration of graph data with features from other modalities, and (2) hierarchical substructure mining, which extracts semantic patterns at multiple granularities to enhance model generalization in demanding contexts. Finally, the dissertation explores the adaptability of graph data in a range of practical applications, including E-commerce demand forecasting and recommendations, to further enhance prediction and reasoning capabilities.

Statistical Analysis Software Market size was valued at USD 7,963.44 Million in 2023 and is projected to reach USD 13,023.63 Million by 2030, growing at a CAGR of 7.28% during the forecast period 2024-2030.

Global Statistical Analysis Software Market Drivers

The market drivers for the Statistical Analysis Software Market can be influenced by various factors. These may include:

Growing Data Complexity and Volume: The demand for sophisticated statistical analysis tools has been fueled by the exponential rise in data volume and complexity across a range of industries. Robust software solutions are necessary for organizations to evaluate and extract significant insights from huge datasets. Growing Adoption of Data-Driven Decision-Making: Businesses are adopting a data-driven approach to decision-making at a faster rate. Utilizing statistical analysis tools, companies can extract meaningful insights from data to improve operational effectiveness and strategic planning. Developments in Analytics and Machine Learning: As these fields continue to progress, statistical analysis software is now capable of more. These tools' increasing popularity can be attributed to features like sophisticated modeling and predictive analytics. A greater emphasis is being placed on business intelligence: Analytics and business intelligence are now essential components of corporate strategy. In order to provide business intelligence tools for studying trends, patterns, and performance measures, statistical analysis software is essential. Increasing Need in Life Sciences and Healthcare: Large volumes of data are produced by the life sciences and healthcare sectors, necessitating complex statistical analysis. The need for data-driven insights in clinical trials, medical research, and healthcare administration is driving the market for statistical analysis software. Growth of Retail and E-Commerce: The retail and e-commerce industries use statistical analytic tools for inventory optimization, demand forecasting, and customer behavior analysis. The need for analytics tools is fueled in part by the expansion of online retail and data-driven marketing techniques. Government Regulations and Initiatives: Statistical analysis is frequently required for regulatory reporting and compliance with government initiatives, particularly in the healthcare and finance sectors. In these regulated industries, statistical analysis software uptake is driven by this. Big Data Analytics's Emergence: As big data analytics has grown in popularity, there has been a demand for advanced tools that can handle and analyze enormous datasets effectively. Software for statistical analysis is essential for deriving valuable conclusions from large amounts of data. Demand for Real-Time Analytics: In order to make deft judgments fast, there is a growing need for real-time analytics. Many different businesses have a significant demand for statistical analysis software that provides real-time data processing and analysis capabilities. Growing Awareness and Education: As more people become aware of the advantages of using statistical analysis in decision-making, its use has expanded across a range of academic and research institutions. The market for statistical analysis software is influenced by the academic sector. Trends in Remote Work: As more people around the world work from home, they are depending more on digital tools and analytics to collaborate and make decisions. Software for statistical analysis makes it possible for distant teams to efficiently examine data and exchange findings.