LScDC Word-Category RIG MatrixApril 2020 by Neslihan Suzen, PhD student at the University of Leicester (ns433@leicester.ac.uk / suzenneslihan@hotmail.com)Supervised by Prof Alexander Gorban and Dr Evgeny MirkesGetting StartedThis file describes the Word-Category RIG Matrix for theLeicester Scientific Corpus (LSC) [1], the procedure to build the matrix and introduces the Leicester Scientific Thesaurus (LScT) with the construction process. The Word-Category RIG Matrix is a 103,998 by 252 matrix, where rows correspond to words of Leicester Scientific Dictionary-Core (LScDC) [2] and columns correspond to 252 Web of Science (WoS) categories [3, 4, 5]. Each entry in the matrix corresponds to a pair (category,word). Its value for the pair shows the Relative Information Gain (RIG) on the belonging of a text from the LSC to the category from observing the word in this text. The CSV file of Word-Category RIG Matrix in the published archive is presented with two additional columns of the sum of RIGs in categories and the maximum of RIGs over categories (last two columns of the matrix). So, the file ‘Word-Category RIG Matrix.csv’ contains a total of 254 columns.This matrix is created to be used in future research on quantifying of meaning in scientific texts under the assumption that words have scientifically specific meanings in subject categories and the meaning can be estimated by information gains from word to categories. LScT (Leicester Scientific Thesaurus) is a scientific thesaurus of English. The thesaurus includes a list of 5,000 words from the LScDC. We consider ordering the words of LScDC by the sum of their RIGs in categories. That is, words are arranged in their informativeness in the scientific corpus LSC. Therefore, meaningfulness of words evaluated by words’ average informativeness in the categories. We have decided to include the most informative 5,000 words in the scientific thesaurus. Words as a Vector of Frequencies in WoS CategoriesEach word of the LScDC is represented as a vector of frequencies in WoS categories. Given the collection of the LSC texts, each entry of the vector consists of the number of texts containing the word in the corresponding category.It is noteworthy that texts in a corpus do not necessarily belong to a single category, as they are likely to correspond to multidisciplinary studies, specifically in a corpus of scientific texts. In other words, categories may not be exclusive. There are 252 WoS categories and a text can be assigned to at least 1 and at most 6 categories in the LSC. Using the binary calculation of frequencies, we introduce the presence of a word in a category. We create a vector of frequencies for each word, where dimensions are categories in the corpus.The collection of vectors, with all words and categories in the entire corpus, can be shown in a table, where each entry corresponds to a pair (word,category). This table is build for the LScDC with 252 WoS categories and presented in published archive with this file. The value of each entry in the table shows how many times a word of LScDC appears in a WoS category. The occurrence of a word in a category is determined by counting the number of the LSC texts containing the word in a category. Words as a Vector of Relative Information Gains Extracted for CategoriesIn this section, we introduce our approach to representation of a word as a vector of relative information gains for categories under the assumption that meaning of a word can be quantified by their information gained for categories.For each category, a function is defined on texts that takes the value 1, if the text belongs to the category, and 0 otherwise. For each word, a function is defined on texts that takes the value 1 if the word belongs to the text, and 0 otherwise. Consider LSC as a probabilistic sample space (the space of equally probable elementary outcomes). For the Boolean random variables, the joint probability distribution, the entropy and information gains are defined.The information gain about the category from the word is the amount of information on the belonging of a text from the LSC to the category from observing the word in the text [6]. We used the Relative Information Gain (RIG) providing a normalised measure of the Information Gain. This provides the ability of comparing information gains for different categories. The calculations of entropy, Information Gains and Relative Information Gains can be found in the README file in the archive published. Given a word, we created a vector where each component of the vector corresponds to a category. Therefore, each word is represented as a vector of relative information gains. It is obvious that the dimension of vector for each word is the number of categories. The set of vectors is used to form the Word-Category RIG Matrix, in which each column corresponds to a category, each row corresponds to a word and each component is the relative information gain from the word to the category. In Word-Category RIG Matrix, a row vector represents the corresponding word as a vector of RIGs in categories. We note that in the matrix, a column vector represents RIGs of all words in an individual category. If we choose an arbitrary category, words can be ordered by their RIGs from the most informative to the least informative for the category. As well as ordering words in each category, words can be ordered by two criteria: sum and maximum of RIGs in categories. The top n words in this list can be considered as the most informative words in the scientific texts. For a given word, the sum and maximum of RIGs are calculated from the Word-Category RIG Matrix.RIGs for each word of LScDC in 252 categories are calculated and vectors of words are formed. We then form the Word-Category RIG Matrix for the LSC. For each word, the sum (S) and maximum (M) of RIGs in categories are calculated and added at the end of the matrix (last two columns of the matrix). The Word-Category RIG Matrix for the LScDC with 252 categories, the sum of RIGs in categories and the maximum of RIGs over categories can be found in the database.Leicester Scientific Thesaurus (LScT)Leicester Scientific Thesaurus (LScT) is a list of 5,000 words form the LScDC [2]. Words of LScDC are sorted in descending order by the sum (S) of RIGs in categories and the top 5,000 words are selected to be included in the LScT. We consider these 5,000 words as the most meaningful words in the scientific corpus. In other words, meaningfulness of words evaluated by words’ average informativeness in the categories and the list of these words are considered as a ‘thesaurus’ for science. The LScT with value of sum can be found as CSV file with the published archive. Published archive contains following files:1) Word_Category_RIG_Matrix.csv: A 103,998 by 254 matrix where columns are 252 WoS categories, the sum (S) and the maximum (M) of RIGs in categories (last two columns of the matrix), and rows are words of LScDC. Each entry in the first 252 columns is RIG from the word to the category. Words are ordered as in the LScDC.2) Word_Category_Frequency_Matrix.csv: A 103,998 by 252 matrix where columns are 252 WoS categories and rows are words of LScDC. Each entry of the matrix is the number of texts containing the word in the corresponding category. Words are ordered as in the LScDC.3) LScT.csv: List of words of LScT with sum (S) values. 4) Text_No_in_Cat.csv: The number of texts in categories. 5) Categories_in_Documents.csv: List of WoS categories for each document of the LSC.6) README.txt: Description of Word-Category RIG Matrix, Word-Category Frequency Matrix and LScT and forming procedures.7) README.pdf (same as 6 in PDF format)References[1] Suzen, Neslihan (2019): LSC (Leicester Scientific Corpus). figshare. Dataset. https://doi.org/10.25392/leicester.data.9449639.v2[2] Suzen, Neslihan (2019): LScDC (Leicester Scientific Dictionary-Core). figshare. Dataset. https://doi.org/10.25392/leicester.data.9896579.v3[3] Web of Science. (15 July). Available: https://apps.webofknowledge.com/[4] WoS Subject Categories. Available: https://images.webofknowledge.com/WOKRS56B5/help/WOS/hp_subject_category_terms_tasca.html [5] Suzen, N., Mirkes, E. M., & Gorban, A. N. (2019). LScDC-new large scientific dictionary. arXiv preprint arXiv:1912.06858. [6] Shannon, C. E. (1948). A mathematical theory of communication. Bell system technical journal, 27(3), 379-423.

In a large network of computers or wireless sensors, each of the components (henceforth, peers) has some data about the global state of the system. Much of the system's functionality such as message routing, information retrieval and load sharing relies on modeling the global state. We refer to the outcome of the function (e.g., the load experienced by each peer) as the emph{model} of the system. Since the state of the system is constantly changing, it is necessary to keep the models up-to-date. Computing global data mining models e.g. decision trees, k-means clustering in large distributed systems may be very costly due to the scale of the system and due to communication cost, which may be high. The cost further increases in a dynamic scenario when the data changes rapidly. In this paper we describe a two step approach for dealing with these costs. First, we describe a highly efficient emph{local} algorithm which can be used to monitor a wide class of data mining models. Then, we use this algorithm as a feedback loop for the monitoring of complex functions of the data such as its k-means clustering. The theoretical claims are corroborated with a thorough experimental analysis.

LScD (Leicester Scientific Dictionary)April 2020 by Neslihan Suzen, PhD student at the University of Leicester (ns433@leicester.ac.uk/suzenneslihan@hotmail.com)Supervised by Prof Alexander Gorban and Dr Evgeny Mirkes[Version 3] The third version of LScD (Leicester Scientific Dictionary) is created from the updated LSC (Leicester Scientific Corpus) - Version 2*. All pre-processing steps applied to build the new version of the dictionary are the same as in Version 2** and can be found in description of Version 2 below. We did not repeat the explanation. After pre-processing steps, the total number of unique words in the new version of the dictionary is 972,060. The files provided with this description are also same as described as for LScD Version 2 below.* Suzen, Neslihan (2019): LSC (Leicester Scientific Corpus). figshare. Dataset. https://doi.org/10.25392/leicester.data.9449639.v2** Suzen, Neslihan (2019): LScD (Leicester Scientific Dictionary). figshare. Dataset. https://doi.org/10.25392/leicester.data.9746900.v2[Version 2] Getting StartedThis document provides the pre-processing steps for creating an ordered list of words from the LSC (Leicester Scientific Corpus) [1] and the description of LScD (Leicester Scientific Dictionary). This dictionary is created to be used in future work on the quantification of the meaning of research texts. R code for producing the dictionary from LSC and instructions for usage of the code are available in [2]. The code can be also used for list of texts from other sources, amendments to the code may be required.LSC is a collection of abstracts of articles and proceeding papers published in 2014 and indexed by the Web of Science (WoS) database [3]. Each document contains title, list of authors, list of categories, list of research areas, and times cited. The corpus contains only documents in English. The corpus was collected in July 2018 and contains the number of citations from publication date to July 2018. The total number of documents in LSC is 1,673,824.LScD is an ordered list of words from texts of abstracts in LSC.The dictionary stores 974,238 unique words, is sorted by the number of documents containing the word in descending order. All words in the LScD are in stemmed form of words. The LScD contains the following information:1.Unique words in abstracts2.Number of documents containing each word3.Number of appearance of a word in the entire corpusProcessing the LSCStep 1.Downloading the LSC Online: Use of the LSC is subject to acceptance of request of the link by email. To access the LSC for research purposes, please email to ns433@le.ac.uk. The data are extracted from Web of Science [3]. You may not copy or distribute these data in whole or in part without the written consent of Clarivate Analytics.Step 2.Importing the Corpus to R: The full R code for processing the corpus can be found in the GitHub [2].All following steps can be applied for arbitrary list of texts from any source with changes of parameter. The structure of the corpus such as file format and names (also the position) of fields should be taken into account to apply our code. The organisation of CSV files of LSC is described in README file for LSC [1].Step 3.Extracting Abstracts and Saving Metadata: Metadata that include all fields in a document excluding abstracts and the field of abstracts are separated. Metadata are then saved as MetaData.R. Fields of metadata are: List_of_Authors, Title, Categories, Research_Areas, Total_Times_Cited and Times_cited_in_Core_Collection.Step 4.Text Pre-processing Steps on the Collection of Abstracts: In this section, we presented our approaches to pre-process abstracts of the LSC.1.Removing punctuations and special characters: This is the process of substitution of all non-alphanumeric characters by space. We did not substitute the character “-” in this step, because we need to keep words like “z-score”, “non-payment” and “pre-processing” in order not to lose the actual meaning of such words. A processing of uniting prefixes with words are performed in later steps of pre-processing.2.Lowercasing the text data: Lowercasing is performed to avoid considering same words like “Corpus”, “corpus” and “CORPUS” differently. Entire collection of texts are converted to lowercase.3.Uniting prefixes of words: Words containing prefixes joined with character “-” are united as a word. The list of prefixes united for this research are listed in the file “list_of_prefixes.csv”. The most of prefixes are extracted from [4]. We also added commonly used prefixes: ‘e’, ‘extra’, ‘per’, ‘self’ and ‘ultra’.4.Substitution of words: Some of words joined with “-” in the abstracts of the LSC require an additional process of substitution to avoid losing the meaning of the word before removing the character “-”. Some examples of such words are “z-test”, “well-known” and “chi-square”. These words have been substituted to “ztest”, “wellknown” and “chisquare”. Identification of such words is done by sampling of abstracts form LSC. The full list of such words and decision taken for substitution are presented in the file “list_of_substitution.csv”.5.Removing the character “-”: All remaining character “-” are replaced by space.6.Removing numbers: All digits which are not included in a word are replaced by space. All words that contain digits and letters are kept because alphanumeric characters such as chemical formula might be important for our analysis. Some examples are “co2”, “h2o” and “21st”.7.Stemming: Stemming is the process of converting inflected words into their word stem. This step results in uniting several forms of words with similar meaning into one form and also saving memory space and time [5]. All words in the LScD are stemmed to their word stem.8.Stop words removal: Stop words are words that are extreme common but provide little value in a language. Some common stop words in English are ‘I’, ‘the’, ‘a’ etc. We used ‘tm’ package in R to remove stop words [6]. There are 174 English stop words listed in the package.Step 5.Writing the LScD into CSV Format: There are 1,673,824 plain processed texts for further analysis. All unique words in the corpus are extracted and written in the file “LScD.csv”.The Organisation of the LScDThe total number of words in the file “LScD.csv” is 974,238. Each field is described below:Word: It contains unique words from the corpus. All words are in lowercase and their stem forms. The field is sorted by the number of documents that contain words in descending order.Number of Documents Containing the Word: In this content, binary calculation is used: if a word exists in an abstract then there is a count of 1. If the word exits more than once in a document, the count is still 1. Total number of document containing the word is counted as the sum of 1s in the entire corpus.Number of Appearance in Corpus: It contains how many times a word occurs in the corpus when the corpus is considered as one large document.Instructions for R CodeLScD_Creation.R is an R script for processing the LSC to create an ordered list of words from the corpus [2]. Outputs of the code are saved as RData file and in CSV format. Outputs of the code are:Metadata File: It includes all fields in a document excluding abstracts. Fields are List_of_Authors, Title, Categories, Research_Areas, Total_Times_Cited and Times_cited_in_Core_Collection.File of Abstracts: It contains all abstracts after pre-processing steps defined in the step 4.DTM: It is the Document Term Matrix constructed from the LSC[6]. Each entry of the matrix is the number of times the word occurs in the corresponding document.LScD: An ordered list of words from LSC as defined in the previous section.The code can be used by:1.Download the folder ‘LSC’, ‘list_of_prefixes.csv’ and ‘list_of_substitution.csv’2.Open LScD_Creation.R script3.Change parameters in the script: replace with the full path of the directory with source files and the full path of the directory to write output files4.Run the full code.References[1]N. Suzen. (2019). LSC (Leicester Scientific Corpus) [Dataset]. Available: https://doi.org/10.25392/leicester.data.9449639.v1[2]N. Suzen. (2019). LScD-LEICESTER SCIENTIFIC DICTIONARY CREATION. Available: https://github.com/neslihansuzen/LScD-LEICESTER-SCIENTIFIC-DICTIONARY-CREATION[3]Web of Science. (15 July). Available: https://apps.webofknowledge.com/[4]A. Thomas, "Common Prefixes, Suffixes and Roots," Center for Development and Learning, 2013.[5]C. Ramasubramanian and R. Ramya, "Effective pre-processing activities in text mining using improved porter’s stemming algorithm," International Journal of Advanced Research in Computer and Communication Engineering, vol. 2, no. 12, pp. 4536-4538, 2013.[6]I. Feinerer, "Introduction to the tm Package Text Mining in R," Accessible en ligne: https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf, 2013.

The purpose of data mining analysis is always to find patterns of the data using certain kind of techiques such as classification or regression. It is not always feasible to apply classification algorithms directly to dataset. Before doing any work on the data, the data has to be pre-processed and this process normally involves feature selection and dimensionality reduction. We tried to use clustering as a way to reduce the dimension of the data and create new features. Based on our project, after using clustering prior to classification, the performance has not improved much. The reason why it has not improved could be the features we selected to perform clustering are not well suited for it. Because of the nature of the data, classification tasks are going to provide more information to work with in terms of improving knowledge and overall performance metrics. From the dimensionality reduction perspective: It is different from Principle Component Analysis which guarantees finding the best linear transformation that reduces the number of dimensions with a minimum loss of information. Using clusters as a technique of reducing the data dimension will lose a lot of information since clustering techniques are based a metric of 'distance'. At high dimensions euclidean distance loses pretty much all meaning. Therefore using clustering as a "Reducing" dimensionality by mapping data points to cluster numbers is not always good since you may lose almost all the information. From the creating new features perspective: Clustering analysis creates labels based on the patterns of the data, it brings uncertainties into the data. By using clustering prior to classification, the decision on the number of clusters will highly affect the performance of the clustering, then affect the performance of classification. If the part of features we use clustering techniques on is very suited for it, it might increase the overall performance on classification. For example, if the features we use k-means on are numerical and the dimension is small, the overall classification performance may be better. We did not lock in the clustering outputs using a random_state in the effort to see if they were stable. Our assumption was that if the results vary highly from run to run which they definitely did, maybe the data just does not cluster well with the methods selected at all. Basically, the ramification we saw was that our results are not much better than random when applying clustering to the data preprocessing. Finally, it is important to ensure a feedback loop is in place to continuously collect the same data in the same format from which the models were created. This feedback loop can be used to measure the model real world effectiveness and also to continue to revise the models from time to time as things change.

Mineral Resource Mines Directory

Operational Mineral Resource Mines Directory

By Homeland Infrastructure Foundation [source]

About this dataset

This dataset provides comprehensive information on operational mineral resource mines, specifically excluding sand and gravel quarries. It offers valuable insights into the locations of these mines, along with detailed data about emergency contact details, directions, and other relevant information.

The dataset covers a wide range of attributes related to the mines, including their feature type and security classification. The names of the mines are also provided, along with their respective areas. Contact information such as phone numbers and addresses can be found for each mine, including additional address details if applicable.

Furthermore, the dataset includes vital geographic information such as cities, states, ZIP codes (including ZIP+4 codes), counties, FIPS codes, directions to the mine's location using text descriptions or maps. Additionally , it reveals important emergency contact details like emergency contact titles and phone numbers.

Information on when contacts were made with each mine is available through contact dates and contact methods used. The geographic precision is also mentioned specifically in relation to locating each mine accurately.

The dataset further classifies mines according to themes set by the Homeland Security Infrastructure Program (HSIPTHEMES). Moreover,the North American Industry Classification System (NAICS) codes help identify specific industry classifications for each mine.

Precise longitude and latitude coordinates enable accurate mapping of each mine's location. Vendor data sources are identified alongside versioning information related to the dataset's content quality control/quality assurance processes qualified under QC_QA classification.

Inspection officers assigned to oversee mining operations are also included as part of this comprehensive database supporting overall security measures employed within mining sites. The Standard Industrial Classification (SIC) code designated for each site provides further clarity regarding its categorization within a specific industrial context.

Lastly,certain textual data sets provide canvass insights pertinentlty defined through Naicsdesrc correlatively providing description based granularity into various types of industries within which these miines operate dynamics chord progressed while embarking onto interphase where industry operates including MINE_TYPE as descriptive horizon including Security Classification visualizes the nature of the Master Miner Site.

The dataset includes data at various geographic levels, such as city, county, state FIPS codes (a standardized coding system for identifying counties in the United States), and multiple subunit numbers that further refine information about specific sections or units within each mine

How to use the dataset

Understanding the Columns

Before exploring the dataset, it's essential to understand the meaning of each column. Here are some key columns to note:

• FEATTYPE: The type of feature or resource mine.
• SECCLASS: The security classification of the mine.
• NAME: The name of the mine.
• AREA_: The area of the mine in numeric format.
• PHONE: The phone number of the mine.
• ADDRESS: The address of the mine.
• ADDRESS2: Additional address information for the mine.
• CITY: The city where the mine is located.
• STATE: The state where the mine is located.
• ZIP: The ZIP code of the mine's location.

• ZIPP4: The ZIP+4 code ofthe minse's location − COUNTY: The county where the mine is located − FIPS: The FIPS code (Federal Information Processing Standards) offor need country_code? − DIRECTIONS: : Directions for findingi tthepmiklet tonatiof ntheG eominformationr m concerns ieinstruction to get there adequately dierscibd by GPS − EMERGTITLE : .Themergency contact title forinounforeseen r emergency situatisononts

  yo gather Relevant dataTo will analyze gather these relevant columns that best fit your research needs tf When extracting data from this dataset using programming languages like Python or Rithcoon try xensuring collect extracteTheserengageyousr directlyebased on these columns, as they cover essential details about the mines, such as their location, contact information,pempany names, and more. The dataset also includes additional subunit information for each mine.

  Discovering Key Insights

• What are the different types of features or resource mines? ...

These three artificial datasets are for mining erasable itemset. The definition of erasable itemset is in the following reference papers. Note that the three data sets all include 200 different items. But for each item, we did not give the profit value of it. Users can generate as they require, with normal or randomly distribution.

This dataset contains the review results of the manuscript of "A Systematic Review on Privacy-Preserving Distributed Data Mining" authored by Chang Sun, Lianne Ippel, Andre Dekker, Michel Dumontier, Johan van Soest. In the datasets, there are 231 published articles about privacy-perserving distributed data mining. Variables include article DOI, title, authors, keywords, user scenarios, distributed data scenarios, privacy/security definition/proof/analysis, privacy statement, privacy-preserving methods category, privacy-preserving methods (specific), data mining problem, data mining/machine learning methods, experiment data information, accuracy of the methods, efficiency (computation and communication cost), and scalability. The search method and evaluation criteria are described in the paper "A Systematic Review on Privacy-Preserving Distributed Data Mining". The DOI and link to the paper will be provided when the paper gets published.

The dataset has been collected in the frame of the Prac1 of the subject Tipology and Data Life Cycle of the Master's Degree in Data Science of the Universitat Oberta de Catalunya (UOC).

The dataset contains 25 variables and 52478 records corresponding to books on the GoodReads Best Books Ever list (the larges list on the site).

Original code used to retrieve the dataset can be found on github repository: github.com/scostap/goodreads_bbe_dataset

The data was retrieved in two sets, the first 30000 books and then the remainig 22478. Dates were not parsed and reformated on the second chunk so publishDate and firstPublishDate are representet in a mm/dd/yyyy format for the first 30000 records and Month Day Year for the rest.

Book cover images can be optionally downloaded from the url in the 'coverImg' field. Python code for doing so and an example can be found on the github repo.

The 25 fields of the dataset are:

| Attributes | Definition | Completeness |
| ------------- | ------------- | ------------- | 
| bookId | Book Identifier as in goodreads.com | 100 |
| title | Book title | 100 |
| series | Series Name | 45 |
| author | Book's Author | 100 |
| rating | Global goodreads rating | 100 |
| description | Book's description | 97 |
| language | Book's language | 93 |
| isbn | Book's ISBN | 92 |
| genres | Book's genres | 91 |
| characters | Main characters | 26 |
| bookFormat | Type of binding | 97 |
| edition | Type of edition (ex. Anniversary Edition) | 9 |
| pages | Number of pages | 96 |
| publisher | Editorial | 93 |
| publishDate | publication date | 98 |
| firstPublishDate | Publication date of first edition | 59 |
| awards | List of awards | 20 |
| numRatings | Number of total ratings | 100 |
| ratingsByStars | Number of ratings by stars | 97 |
| likedPercent | Derived field, percent of ratings over 2 starts (as in GoodReads) | 99 |
| setting | Story setting | 22 |
| coverImg | URL to cover image | 99 |
| bbeScore | Score in Best Books Ever list | 100 |
| bbeVotes | Number of votes in Best Books Ever list | 100 |
| price | Book's price (extracted from Iberlibro) | 73 |

Dataset

This dataset was created by Balal H

Contents

The Supplementary Material of the paper "Supplementary Material: Predictive model using Cross Industry Standard Process for Data Mining" includes: 1) APPENDIX 1: SQL Statements for data extraction. Appendix 2: Interview for operating Staff. 2) The DataSet of the normalized data to define the predictive model.

Time series data for the statistic Mining Index and country Malta. Indicator Definition:Mining IndexThe indicator "Mining Index" stands at 112.34 as of 4/30/2025. Regarding the One-Year-Change of the series, the current value constitutes an increase of 7.07 percent compared to the value the year prior.The 1 year change in percent is 7.07.The 3 year change in percent is 9.48.The 5 year change in percent is -3.83.The 10 year change in percent is 11.85.The Serie's long term average value is 135.38. It's latest available value, on 4/30/2025, is 17.02 percent lower, compared to it's long term average value.The Serie's change in percent from it's minimum value, on 2/28/2011, to it's latest available value, on 4/30/2025, is +94.68%.The Serie's change in percent from it's maximum value, on 12/31/2003, to it's latest available value, on 4/30/2025, is -76.25%.

Dataset

This dataset was created by ddpr raju

Contents

The "Wikipedia SQLite Portable DB" is a compact and efficient database derived from the Kensho Derived Wikimedia Dataset (KDWD). This dataset provides a condensed subset of raw Wikimedia data in a format optimized for natural language processing (NLP) research and applications.

I am not affiliated or partnered with the Kensho in any way, just really like the dataset for giving my agents to query easily.

Key Features:

Contains over 5 million rows of data from English Wikipedia and Wikidata Stored in a portable SQLite database format for easy integration and querying Includes a link-annotated corpus of English Wikipedia pages and a compact sample of the Wikidata knowledge base Ideal for NLP tasks, machine learning, data analysis, and research projects

The database consists of four main tables:

• items: Contains information about Wikipedia items, including labels and descriptions
• properties: Stores details about Wikidata properties, such as labels and descriptions
• pages: Provides metadata for Wikipedia pages, including page IDs, item IDs, titles, and view counts
• link_annotated_text: Contains the link-annotated text of Wikipedia pages, divided into sections

This dataset is derived from the Kensho Derived Wikimedia Dataset (KDWD), which is built from the English Wikipedia snapshot from December 1, 2019, and the Wikidata snapshot from December 2, 2019. The KDWD is a condensed subset of the raw Wikimedia data in a form that is helpful for NLP work, and it is released under the CC BY-SA 3.0 license. Credits: The "Wikipedia SQLite Portable DB" is derived from the Kensho Derived Wikimedia Dataset (KDWD), created by the Kensho R&D group. The KDWD is based on data from Wikipedia and Wikidata, which are crowd-sourced projects supported by the Wikimedia Foundation. We would like to acknowledge and thank the Kensho R&D group for their efforts in creating the KDWD and making it available for research and development purposes. By providing this portable SQLite database, we aim to make Wikipedia data more accessible and easier to use for researchers, data scientists, and developers working on NLP tasks, machine learning projects, and other data-driven applications. We hope that this dataset will contribute to the advancement of NLP research and the development of innovative applications utilizing Wikipedia data.

https://www.kaggle.com/datasets/kenshoresearch/kensho-derived-wikimedia-data/data

Tags: encyclopedia, wikipedia, sqlite, database, reference, knowledge-base, articles, information-retrieval, natural-language-processing, nlp, text-data, large-dataset, multi-table, data-science, machine-learning, research, data-analysis, data-mining, content-analysis, information-extraction, text-mining, text-classification, topic-modeling, language-modeling, question-answering, fact-checking, entity-recognition, named-entity-recognition, link-prediction, graph-analysis, network-analysis, knowledge-graph, ontology, semantic-web, structured-data, unstructured-data, data-integration, data-processing, data-cleaning, data-wrangling, data-visualization, exploratory-data-analysis, eda, corpus, document-collection, open-source, crowdsourced, collaborative, online-encyclopedia, web-data, hyperlinks, categories, page-views, page-links, embeddings

Usage with LIKE queries: ``` import aiosqlite import asyncio

class KenshoDatasetQuery: def init(self, db_file): self.db_file = db_file

async def _aenter_(self):
  self.conn = await aiosqlite.connect(self.db_file)
  return self

async def _aexit_(self, exc_type, exc_val, exc_tb):
  await self.conn.close()

async def search_pages_by_title(self, title):
  query = """
  SELECT pages.page_id, pages.item_id, pages.title, pages.views, 
      items.labels AS item_labels, items.description AS item_description,
      link_annotated_text.sections
  FROM pages 
  JOIN items ON pages.item_id = items.id
  JOIN link_annotated_text ON pages.page_id = link_annotated_text.page_id
  WHERE pages.title LIKE ?
  """
  async with self.conn.execute(query, (f"%{title}%",)) as cursor:
    return await cursor.fetchall()

async def search_items_by_label_or_description(self, keyword):
  query = """
  SELECT id, labels, description 
  FROM items
  WHERE labels LIKE ? OR description LIKE ?
  """
  async with self.conn.execute(query, (f"%{keyword}%", f"%{keyword}%")) as cursor:
    return await cursor.fetchall()

async def search_items_by_label(self, label):
  query = """
  SELECT id, labels, description
  FROM items 
  WHERE labels LIKE ?
  """
  async with self.conn.execute(query, (f"%{label}%",)) as cursor:
    return await cursor.fetchall()

async def search_properties_by_label_or_desc...

Time series data for the statistic Mining Index and country Luxembourg. Indicator Definition:Mining IndexThe indicator "Mining Index" stands at 42.17 as of 4/30/2025. Regarding the One-Year-Change of the series, the current value constitutes a decrease of -22.93 percent compared to the value the year prior.The 1 year change in percent is -22.93.The 3 year change in percent is -24.35.The 5 year change in percent is 13.66.The 10 year change in percent is -66.71.The Serie's long term average value is 118.96. It's latest available value, on 4/30/2025, is 64.55 percent lower, compared to it's long term average value.The Serie's change in percent from it's minimum value, on 8/31/2021, to it's latest available value, on 4/30/2025, is +210.67%.The Serie's change in percent from it's maximum value, on 7/31/2001, to it's latest available value, on 4/30/2025, is -85.84%.

This dataset is published as Open DataAbstract The Development High Risk Area is the part of the coal mining reporting area which contains one or more recorded coal mining related features which have the potential for instability or a degree of risk to the surface from the legacy of coal mining operations. The combination of features included in this composite area includes mine entries; shallow coal workings (recorded and probable); recorded coal mining related hazards; recorded mine gas sites; fissures and breaklines and previous surface mining sites. New development in this defined area needs to demonstrate that the development will be safe and stable taking full account of former coal mining activities. This area was formally known as the Development Referral Area. Purpose The development high risk areas have been defined to enable developers and planners to understand and consider the potential for instability or degree of risk from the legacy of coal mining operations. This information is also provided to asset managers for the management of the land assets of public bodies and major landowners. Supplementary Information The National Coal Mining Database, which is based on the records held at The Coal Authority offices in Mansfield, Nottinghamshire, is updated on a regular basis. This dataset has been extracted from this dynamic database on the date stated below and therefore represents a snapshot in time. Status of the data Extract of data from the National Coal Mining Database Data update frequency: As needed

Time series data for the statistic Mining Change in Percent (period-on-period) and country Malta. Indicator Definition:Mining Change in Percent (period-on-period)The indicator "Mining Change in Percent (period-on-period)" stands at -6.30 as of 04/30/2025, the lowest value since 01/31/2025. Regarding the One-Year-Change of the series, the current value constitutes an decrease of -9.44 compared to the value the year prior.The Serie's long term average value is 2.84. It's latest available value, on 04/30/2025, is -9.14 lower, compared to it's long term average value.The Serie's change from it's minimum value, on 01/31/2004, to it's latest available value, on 04/30/2025, is +56.97 .The Serie's change from it's maximum value, on 09/30/2003, to it's latest available value, on 04/30/2025, is -155.31 .

Time series data for the statistic Mining Change in Percent (y-o-y) and country Montenegro. Indicator Definition:Mining Change in Percent (y-o-y)The indicator "Mining Change in Percent (y-o-y)" stands at 10.75 as of 3/31/2025. Regarding the One-Year-Change of the series, the current value constitutes an increase of 0.029 compared to the value the year prior.The Serie's long term average value is 17.18. It's latest available value, on 3/31/2025, is -6.43 lower, compared to it's long term average value.The Serie's change from it's minimum value, on 6/30/2009, to it's latest available value, on 3/31/2025, is +105.63 .The Serie's change from it's maximum value, on 6/30/2010, to it's latest available value, on 3/31/2025, is -861.38 .

The Development High Risk Area is the part of the coal mining reporting area which contains one or more recorded coal mining related features which have the potential for instability or a degree of risk to the surface from the legacy of coal mining operations. The combination of features included in this composite area includes mine entries; shallow coal workings (recorded and probable); recorded coal mining related hazards; recorded mine gas sites; fissures and breaklines and previous surface mining sites. New development in this defined area needs to demonstrate that the development will be safe and stable taking full account of former coal mining activities. This area was formally known as the Development Referral Area. The development high risk areas have been defined to enable developers and planners to understand and consider the potential for instability or degree of risk from the legacy of coal mining operations. This information is also provided to asset managers for the management of the land assets of public bodies and major landowners.

Abstract

The dataset was derived by the Bioregional Assessment Programme from multiple source datasets. The source datasets are identified in the Lineage field in this metadata statement.

The processes undertaken to produce this derived dataset are described in the History field in this metadata statement.

Dataset History

Stream network constructed and defined using datasets shown in the Lineage.

The stream network constructed using surface water nodes to define reaches and the the classification was assigned by using the data from the stream network from the lineage and then assigned the following classfication:

1. surface water change due to hydrology

2. no change modelled at link node within PAE

3. modelled no change at link node

4. modelled change at link node

5. assumed change due to proximity to mine pit

6. assumed change due to hydrology

Further tie-breaks were decide based on stream order or stream segment length.

Dataset Citation

Bioregional Assessment Programme (2017) GAL Surface Water Reaches for Risk and Impact Analysis 20180803. Bioregional Assessment Derived Dataset. Viewed 12 December 2018, http://data.bioregionalassessments.gov.au/dataset/64c4d16f-bdfa-4fd6-bd72-c459503003bd.

Dataset Ancestors

• Derived From Onsite and offsite mine infrastructure for the Carmichael Coal Mine and Rail Project, Adani Mining Pty Ltd 2012

• Derived From Alpha Coal Project Environmental Impact Statement

• Derived From Geofabric Surface Cartography - V2.1

• Derived From QLD Exploration and Production Tenements (20140728)

• Derived From China Stone Coal Project initial advice statement

• Derived From Kevin's Corner Project Environmental Impact Statement

• Derived From Galilee surface water modelling nodes

• Derived From Geoscience Australia GEODATA TOPO series - 1:1 Million to 1:10 Million scale

• Derived From China First Galilee Coal Project Environmental Impact Assessment

• Derived From GEODATA TOPO 250K Series 3

• Derived From Seven coal mines included in Galilee surface water modelling

Data for (i) active mine sites and (ii) inactive mine sites are stored are stored as Excel spreadsheets. NB the number of active/inactive mines shown in the spreadsheets is less than that reported in Table S1, because proprietary data sources have not been included (i.e. MRDS, BRITPITS and S&P). Each spreadsheet lists mine names (column A), mine status i.e. active or inactive (column B), the principal commodity mined (column C), and lat/long coordinates (columns D & E). Data for (iii) TSFs and (iv) TDFs are stored as zipped Shapefiles. Data should be uncompressed and then imported into any GIS programme that can read Shapefiles. Modelling was implemented procedurally in MATLAB v9.9.0 (R2020b) with the open source TopoToolbox MATLAB program for the analysis of digital elevation models (https://topotoolbox.wordpress.com). Modelling workflow is presented in SI Figure S8 with example code available in the WAPHA database (Macklin et al code.pdf). Citations to software sources are giv...