A controlled vocabulary for research and innovation in the field of Artificial Intelligence (AI)

This controlled vocabulary of keywords related to the field of Artificial Intelligence (AI) was built by SIRIS Academic in collaboration with ART-ER (the R&I and sustainable development in-house agency of the Emilia-Romagna region in Italy) and the Generalitat de Catalunya (the regional government of Catalonia, Spain), in order to identify AI research, development and innovation activities. The work was carried by consulting domain experts advice and it was ultimately applied to inform regional strategies on AI and research and innovation policy.

The aim of this vocabulary is to enable one to retrieve texts (e.g. R&D projects and scientific publications) featuring the concepts included in the present vocabulary in their titles and abstracts, assuming that these records have a certain contribution of applications, techniques and issues, in the domain of AI.

The present effort was carried out because, despite the high number of contributions and technological developments in the field of AI, there is no closed or static vocabulary of concepts that allow one to unequivocally define the boundaries of what should be considered “an Artificial Intelligence intellectual product” (or what should not). Indeed, the literature presents different definitions of the domain, with visions that could be contradictory. AI encompasses today a wide variety of sub-domains, ranging from general purpose areas such as learning and perception to more specific ones such as autonomous vehicle driving, theorem proving, or industrial process monitoring. AI synthesises and automates intellectual tasks, and is therefore potentially relevant to any area of human intellectual activity. In this sense, it is a genuinely universal and multidisciplinary field. AI draws upon disciplines as diverse as cybernetics, mathematics, philosophy, sociology and economics.

For the current definition of an AI controlled vocabulary, the initial set of terms setting the boundaries of AI were taken from different sub-domains of the ACM Computing Classification System 2012. Notably, although some relevant AI sub-domains have an independent category in the ACM taxonomy outside of AI, they have been included in the list of sub-domains. In order to align the ACM taxonomical definition with the Catalan Strategy of AI, CATALONIA.AI, the emerging area of AI Ethics has been included in the vocabulary, while some other categories which are not relevant for the objectives of this resource have been removed from the sub-domains list.

In short, AI subdomains considered in the present vocabulary are the following: (1) General, (2) Machine Learning, (3) Computer Vision, (4) Natural Language Processing, (5) Knowledge Representation and Reasoning, (6) Distributed Artificial Intelligence, (7) Expert Systems, Problem-Solving, Control Methods and Search and (7) AI Ethics.

Although a keyword rule-based approach suffers of the major two shortcomings of not being able to capture all the lexical and linguistic variants of a specific term, and of not capturing the context of the terms (in other words, keyword-based approaches would miss relevant texts if the specific pattern is not matched during the search), the present vocabulary allowed us to obtain fairly good results, due to the specificity of the concepts describing the AI domain. Furthermore, an understandable and transparent controlled vocabulary allows a better control of the final results and the final definition of the domain borders. Also, a plain list of terms allows a much easier and interactive engagement of interested stakeholders with different degree of knowledge (such as, for instance, domain experts, policy-makers and potential users) who can make use of vocabulary to retrieve pertinent literature or to enrich the resource itself.

The vocabulary has been built taking advantage of advanced language models and resources from knowledge datasets such as arXiv, Dpedia, Wikipedia and Scopus. The resulting vocabulary comprises 599 keywords, annotated by AI sub-domain, and has been validated by experts from several universities in Emilia-Romagna and Catalonia.

The first version of this resource was developed by the SIRIS Academic in 2019 in collaboration with ART-ER, Emilia Romagna (Quinquillá et al., 2020), and the current version was updated in 2020 in collaboration with the Generalitat de Catalunya.

The methodology for the construction of the controlled vocabulary is presented in the following steps:

1. An initial set of scientific publications was collected by retrieving the following records as a weakly-supervised (in the sense that records are linked to AI by their taxonomy and not by a manual label) dataset in the domain of Artificial Intelligence :

   1. Publications from Scopus with the keyword “Artificial Intelligence”

   2. Publications from arXiv in the category “Artificial Intelligence”

   3. Publications in relevant journals in the scientific domain of “Artificial Intelligence”

2. An automated algorithm was used to retrieve, from the APIs of DBpedia, a series of terms that have some categorical relationships (i.e. those that are indexed as “sub-categories of”, “equivalent to”, among other relations in DBpedia) with the Artificial Intelligence concept and with the AI categories in the ACM taxonomy. The DBpedia tree has been exploited down to the level 3, and the relevant categories have been manually selected (for instance: Classification algorithms, Machine learning or Evolutionary computation) and others were ignored (for instance: Artificial intelligence in fiction, Robots or History of artificial intelligence) because they were not relevant, or not specifically in the domain.

3. The keywords in publications in the dataset were extracted from the keyword sections and from the abstracts. The keywords with a higher TF-IDF, using an IDF matrix in the open domain, have been selected. The co-occurrence of keywords with categories in specific AI sub-domain and a clusterization of the main keywords has been used for a categorization of the keywords at the thematic level.

4. This list of keywords tagged by thematic category has been manually revised, removing the non-pertinent keywords and changing the wrong categorizations by fields.

5. The weak-supervised dataset in the domain of Artificial Intelligence is used to train a Word2Vec (Mikolov et al., 2013) word embedding model (a machine learning model based on neural networks).

6. The terms’ list is then enriched by means of automatic methods, which are run in parallel:

   1. The trained Word2Vec model is used to select, among the indexed keywords of the reference corpus, all terms “semantically close” to the initial set of words. This step is carried out to select terms that might not appear in the texts themselves, but that were deemed pertinent to label the textual records.

   2. Further, terms that are mentioned in the texts of the reference corpus and that are valued by the trained Word2Vec model as “semantically close” to the initial set of words are also retained. This step is performed to include in the controlled vocabulary a series of terms that are related to the focus of the SDGs and which are used by practitioners.

7. The final list produced by steps 2-6 is manually revised.

The definition of the vocabulary does not, per se, allow to identify STI contributions to AI: this activity in fact boils down to actually matching the terms in the controlled vocabulary to the content of the gathered STI textual records. To successfully carry out this task, a series of pattern matching rules must be defined to capture possible variants of the same concept, such as permutations of words within the concept and/or the presence of null words to be skipped. For this reason, we have carefully crafted matching rules that take into account permutations of words and that allow words within concept to be within a certain distance. Some relatively ambiguous keywords (which may match unwanted pieces of text), have a set of associated “extra” terms. These “extra” terms are defined as further terms that must co-appear, in the same sentence, together with their associated ambiguous keywords. Finally, each keyword in the vocabulary was assigned one or more AI sub-domains, so that the vocabulary can also be used to tag collections of texts within narrower AI sub-domains.

The final controlled vocabulary has been evaluated with an external test set, proposed by (Dunham et al., 2020). The test set consists of the abstract of 10,606 papers published in the arXiv repository, of which 1,076 within the Artificial Intelligence subcategories and 9,530 in arXiv categories other than Artificial Intelligence. Evaluating the controlled vocabulary on this data set, we observe accuracy of .94. However, because the pertinence of these publications to the field of AI is based solely on their taxonomic classification (i.e., on whether they are classified in the arXiv within Artificial Intelligence and not on a manual labeling), this evaluation can only yield an orientative performance assessment.

The AI controlled vocabulary has been applied in two practical cases, which have the purpose of identifying skills, stakeholders and capabilities, of a specific research ecosystem at the regional level. See the following references:

• Quinquillá, Arnau, Duran-Silva, Nicolau, Massucci, Francesco Alessandro, Fuster, Enric, Rondelli, Bernardo, Bologni, Leda, … Moretti,

Aiming at developing a UX evaluation method for AI-infused products, the researchers involved in the Meet-AI project assessed the descriptors emerging from an exploratory survey (you can find the reference at the following link: https://figshare.com/s/bf5a997e78afc351e02b) according to their (i) consistency with the dimension in which they were proposed, and to their (ii) relevance for AI-infused products. Items emerging from a previously undertaken literature review and mapping of the existing UX evaluation methods are also evaluated according to their relevance.In the dataset, the 1 to 4 evaluations by the involved researchers are displayed.

This repository contains the data from the paper, "Benchmark Generation Framework with Customizable Distortions for Image Classifier Robustness." Relevant URLs: https://hewlettpackard.github.io/trust-ml/ https://github.com/HewlettPackard/trust-ml/ Abstract: We present a novel framework for generating adversarial benchmarks to evaluate the robustness of image classification models. The RLAB framework allows users to customize the types of distortions to be optimally applied to images, which helps address the specific distortions relevant to their deployment. The benchmark can generate datasets at various distortion levels to assess the robustness of different image classifiers. Our results show that the adversarial samples generated by our framework with any of the image classification models, like ResNet-50, Inception-V3, and VGG-16, are effective and transferable to other models causing them to fail. These failures happen even when these models are adversarially retrained using state-of-the-art techniques, demonstrating the generalizability of our adversarial samples. Our framework also allows the creation of adversarial samples for non-ground truth classes at different levels of intensity, enabling tunable benchmarks for the evaluation of false positives. We achieve competitive performance in terms of net $L_2$ distortion compared to state-of-the-art benchmark techniques on CIFAR-10 and ImageNet; however, we demonstrate our framework achieves such results with simple distortions like Gaussian noise without introducing unnatural artifacts or color bleeds. This is made possible by a model-based reinforcement learning (RL) agent and a technique that reduces a deep tree search of the image for model sensitivity to perturbations, to a one-level analysis and action. The flexibility of choosing distortions and setting classification probability thresholds for multiple classes makes our framework suitable for algorithmic audits.

These files represent the data and accompanying documents of an independent research study by a student researcher examining the searchability and usability of machine learning dataset metadata.

The purpose of this exploratory study was to understand how machine learning (ML) practitioners are searching for and evaluating datasets for use in their work. This research will help inform development of the ML dataset metadata standard Croissant, which is actively being developed by the Croissant MLCommons working group, so it can aid ML practitioners' workflows and promote best practices like Responsible Artificial Intelligence (RAI).

The study consisted of a pre-interview Qualtrics survey ("Survey_questions_pre_interview.pdf") that focused on ranking various metadata elements on a Likert importance scale.

The interview consisted of open questions ("Interview_script_and_questions.pdf") on a range of topics from search of datasets to interoperability to AI used in dataset search. Additionally, participants were asked to share their screen at one point and recall a recent dataset search they had performed.

The resulting survey dataset ("Survey_p1.csv") and interview ("Interview_p1.txt") of participants are presented in open standard formats for accessibility. Identifying data has been removed from the files so there will be missing columns and rows potentially referenced in the files.

We provide a comprehensive curated catalogue of artificial intelligence datasets and benchmarks for medical decision making. At the time of first release (April 2021), the dataset contains more than 400 biomedical and clinical datasets of which 252 are publicly available or available upon request.

The dataset was compiled based on a systematic literature review covering both biomedical and computer science literature and grey literature data sources. All datasets were manually systematized and annotated for meta-information, such as:

• Availability and licensing information
• Type of source data
• Links to source publications, main references or dataset repositories

Benchmark dataset were additionally annotated for the following information:

• Associated task
• Performance metrics commonly used for evaluation
• Clinical relevance
• The availability of data splits

In addition to the versioned TSV file on Zenodo, the dataset can also be explored live via this Google Spreadsheet. The dataset is intended as a living, extendable resource. Edit suggestions and additions are encouraged and can be submitted via the comment function of the Google sheet.

File descriptions

annotated-datasets.tsv -- contains the annotated datasets

arXiv-literature-export.tsv -- contains the original literature record export from arXiv

pubmed-literature-export.tsv -- contains the original literature record export from PubMed

README.md -- contains a detailed description of all annotation fields

Nurses play a crucial role in suicide prevention, yet the integration of artificial intelligence and machine learning technologies into nursing practice remains understudied. This research examines how these technologies can enhance nurses’ ability to identify and intervene with at-risk patients. A systematic bibliometric analysis and thematic mapping approach was employed. The Web of Science database was searched for relevant publications from January 2019 to October 2024. The initial search yielded 883 publications, with 257 meeting the inclusion criteria after systematic screening. Analysis revealed six distinct research clusters, with machine learning-based behavioral prediction emerging as the dominant theme. Findings indicate significant potential for integrating artificial intelligence-supported tools into nursing workflows, particularly in risk assessment and early intervention. Natural language processing and ecological momentary assessment emerged as promising approaches for enhancing nurse-patient communication and monitoring. These findings suggest opportunities for nurses to leverage artificial intelligence technologies in suicide prevention while maintaining the essential human element of care. This study provides evidence-based guidance for nurses implementing artificial intelligence-supported suicide prevention tools while maintaining therapeutic relationships and professional judgment in clinical practice.

It is estimated that disruptions to life caused by the COVID-19 pandemic have led to an increase in the number of children and young people suffering from mental health issues globally. In England one in four children experienced poor mental health in 2022. Social prescribing is gaining traction as a systems-based approach, which builds upon person-centered methods, to refer children and young people with non-clinical mental health issues to appropriate community assets. Recognition of social prescribing benefits for children’s mental health is increasing, yet evidence is limited. Inconsistent terminology and variation of terms used to describe social prescribing practices across the literature hinders understanding and assessment of social prescribing’s impact on children’s mental health. This scoping review thus aims to systematically identify and analyse the various terms, concepts and language used to describe social prescribing with children and young people across the wider health and social care literature base. The scoping review will be undertaken using a six-stage framework which includes: identifying the research question, identifying relevant studies, study selection, charting the data, collating, summarising and reporting the results, and consultation. Electronic databases (MEDLINE, Embase, Cumulative Index to Nursing and Allied Health, PsychInfo, Social Policy Practice, Scopus, Science Direct, Cochrane library and Joanna Briggs), alongside evidence from grey literature, hand search, citation tracking, and use of expert correspondence will be included in the review to ensure published and unpublished literature is captured. Data extraction will be carried out by two reviewers using a predefined form to capture study characteristics, intervention descriptions, outcomes, and key terms used to report social prescribing for children and young people. No formal quality appraisal or risk of bias evaluation will be performed, as this scoping review aims to map and describe the literature. Data will be stored and managed using the Rayaan.ai platform and a critical narrative of the common themes found will be included.

BackgroundColorectal cancer is the third most common malignant tumor with the third highest incidence rate. Distant metastasis is the main cause of death in colorectal cancer patients. Early detection and prognostic prediction of colorectal cancer has improved with the widespread use of artificial intelligence technologies.PurposeThe aim of this study was to comprehensively evaluate the accuracy and validity of AI-based imaging data for predicting distant metastasis in colorectal cancer patients.MethodsA systematic literature search was conducted to find relevant studies published up to January, 2024, in different databases. The quality of articles was assessed using the Quality Assessment of Diagnostic Accuracy Studies 2 tool. The predictive value of AI-based imaging data for distant metastasis in colorectal cancer patients was assessed using pooled sensitivity, specificity. To explore the reasons for heterogeneity, subgroup analyses were performed using different covariates.ResultsSeventeen studies were included in the systematic evaluation. The pooled sensitivity, specificity, and AUC of AI-based imaging data for predicting distant metastasis in colorectal cancer patients were 0.86, 0.82, and 0.91. Based on QUADAS-2, risk of bias was detected in patient selection, diagnostic tests to be evaluated, and gold standard. Based on the results of subgroup analyses, found that the duration of follow-up, site of metastasis, etc. had a significant impact on the heterogeneity.ConclusionImaging data images based on artificial intelligence algorithms have good diagnostic accuracy for predicting distant metastasis in colorectal cancer patients and have potential for clinical application.Systematic review registrationhttps://www.crd.york.ac.uk/PROSPERO/, identifier PROSPERO (CRD42024516063).

AI-Shift Ameba FAQ Search

Queries and difficulty levels for AI-based FAQ search

By ai-shift (From Huggingface) [source]

About this dataset

The ai-shift/ameba_faq_search dataset provides a comprehensive collection of FAQ and query data, specifically tailored for training and evaluating an AI-based FAQ search system. This dataset is developed using a large language model, ensuring accurate results and enhanced performance.

The dataset comprises several columns containing essential information. Firstly, the Query column consists of various queries or questions that users commonly ask when seeking specific information. These queries serve as representative samples that reflect users' search patterns.

Apart from the queries, the dataset also includes a column called Difficulty, which indicates the level of complexity associated with each query. This difficulty level helps gauge how challenging it might be to find an appropriate answer for each question within the provided dataset.

To facilitate proper understanding and utilization of this dataset, it consists of multiple repetitions of these key columns: Query and Difficulty. Repetition is utilized to ensure inclusivity and provide sufficient data points to train an effective AI-based FAQ search model.

In addition to serving as a training resource, this dataset also offers separate validation files (validation.csv) to accurately measure and evaluate the performance of the AI models trained on this data. Likewise, test files (test.csv) are provided separately for testing purposes during development.

By leveraging this extensive 'ai-shift/ameba_faq_search' dataset developed explicitly for building advanced faq search systems powered by artificial intelligence technologies, developers can enhance their solutions' accuracy in providing valuable information in response to user queries

Research Ideas

• Customer Support: This dataset can be used to develop an AI-based FAQ search system for customer support. By training the model on this dataset, it can provide accurate and relevant answers to user queries, helping customers find the information they need easily.
• Knowledge Management: Companies or organizations can use this dataset to build a knowledge base that employees or users can search through to find answers to their questions. The difficulty level column can be used to prioritize certain queries or topics for better organization and accessibility of information.
• Chatbot Development: With this dataset, developers can train chatbots to understand user queries and provide appropriate responses based on the difficulty level of each query. This could enhance the efficiency and effectiveness of chatbots in providing helpful information quickly.
• Search Engine Optimization (SEO): Website owners and marketers could analyze this dataset to understand popular queries or questions users have when searching for specific information. This insight could inform content creation strategies, optimizing website content targeting frequently asked questions, improving search engine rankings and driving more traffic.
• Language Model Training: Researchers in natural language processing (NLP) could use this dataset for training AI models on question answering tasks or for evaluating their performance on understanding user queries with varying levels of difficulty.

• Competitive Analysis: Companies developing AI-based FAQ search systems or chatbots can compare their own datasets with this one as a benchmark, allowing them to identify gaps in their existing data collection process and improve upon it.

• Personalized Recommendations- This Dataset might by using some algorithms help delivering promted/popular/recommended question based upon previous searches/query patters.

These are just a few examples of how this cleverly organized dataset could be utilized!

Acknowledgements

If you use this dataset in your research, please credit the original authors. Data Source

License

License: CC0 1.0 Universal (CC0 1.0) - Public Domain Dedication No Copyright - You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission. See Other Information.

Columns

File: validation.csv | Column name | Description | |:---------------|:-------------------------------------------------------------...

Cognitive Assessment Tools Market Outlook

In 2023, the global cognitive assessment tools market size was valued at approximately USD 4.5 billion, and it is projected to reach USD 12.3 billion by 2032, growing at a compound annual growth rate (CAGR) of 11.7%. The market's growth is primarily driven by the increasing prevalence of cognitive disorders and the rising awareness about early diagnosis and intervention.

The burgeoning elderly population worldwide is a significant growth factor for the cognitive assessment tools market. As the global population ages, the prevalence of neurodegenerative diseases such as AlzheimerÂ’s and ParkinsonÂ’s is on the rise, necessitating early diagnosis and regular monitoring. Cognitive assessment tools play a crucial role in identifying the onset of these disorders, thus allowing for timely medical intervention. Furthermore, the growing awareness among healthcare providers and patients about the importance of early diagnosis in managing cognitive decline is significantly contributing to market growth.

An increased emphasis on mental health and the de-stigmatization of cognitive disorders are further propelling the demand for cognitive assessment tools. Public health campaigns and educational programs aimed at promoting mental wellness have led to a heightened awareness of cognitive health. Consequently, more individuals are seeking cognitive assessments either as a preventive measure or as part of a comprehensive health check-up. This societal shift towards proactive cognitive health management is creating substantial growth opportunities for market players.

Technological advancements and innovations in cognitive assessment tools are another key driver of market growth. Modern cognitive assessment tools are increasingly leveraging artificial intelligence (AI) and machine learning (ML) to enhance the accuracy and efficiency of cognitive evaluations. These technologies enable the development of sophisticated software that can analyze complex cognitive patterns and provide detailed insights. Moreover, the integration of digital health platforms and mobile applications is making cognitive assessments more accessible, further driving market expansion.

Cognitive Search Tools are increasingly becoming an integral part of the cognitive assessment landscape. These tools utilize advanced algorithms and machine learning techniques to sift through vast amounts of data, providing insights that were previously unattainable. By enhancing the accuracy and speed of data retrieval, cognitive search tools support healthcare professionals in making informed decisions quickly. This capability is particularly beneficial in clinical settings where timely interventions can significantly impact patient outcomes. Moreover, as the volume of cognitive health data continues to grow, the role of cognitive search tools in managing and interpreting this information becomes even more critical, driving their adoption across various sectors.

From a regional perspective, North America currently holds the largest market share in the cognitive assessment tools market, attributable to the region's advanced healthcare infrastructure and high adoption rates of innovative medical technologies. However, Asia Pacific is anticipated to witness the fastest growth during the forecast period, driven by increasing healthcare expenditure, rising awareness about cognitive health, and the growing elderly population in countries such as Japan and China.

Component Analysis

In the cognitive assessment tools market, the component segment is divided into software, hardware, and services. The software segment is a major contributor to the market, driven by the increasing preference for digital cognitive assessment tools over traditional pen-and-paper methods. Software solutions offer several advantages, including ease of use, scalability, and the ability to integrate with other healthcare systems. Furthermore, advancements in AI and ML are enhancing the capabilities of cognitive assessment software, making them more effective in diagnosing and monitoring cognitive disorders.

The hardware segment, although smaller compared to software, is also witnessing growth due to the rising adoption of advanced diagnostic devices. These devices, such as neuroimaging tools and portable EEG machines, provide valuable physiological data that complement cognitive assessments. The integration of hardware with software solutions e

Objective: Our objective is to evaluate the efficacy of ChatGPT 4 in accurately and effectively delivering genetic information, building on previous findings with ChatGPT 3.5. We focus on assessing the utility, limitations, and ethical implications of using ChatGPT in medical settings. Materials and Methods: A structured questionnaire, including the Brief User Survey (BUS-15) and custom questions, was developed to assess ChatGPT 4's clinical value. An expert panel of genetic counselors and clinical geneticists independently evaluated ChatGPT 4's responses to these questions. We also involved comparative analysis with ChatGPT 3.5, utilizing descriptive statistics and using R for data analysis. Results: ChatGPT 4 demonstrated improvements over 3.5 in context recognition, relevance, and informativeness. However, performance variability and concerns about the naturalness of the output were noted. No significant difference in accuracy was found between ChatGPT 3.5 and 4.0. Notably, the efficacy of ChatGPT 4 varied significantly across different genetic conditions, with specific differences identified between responses related to BRCA1 and HFE. Discussion and Conclusion: This study highlights ChatGPT 4's potential in genomics, noting significant advancements over its predecessor. Despite these improvements, challenges remain, including the risk of outdated information and the necessity of ongoing refinement. The variability in performance across different genetic conditions underscores the need for expert oversight and continuous AI training. ChatGPT 4, while showing promise, emphasizes the importance of balancing technological innovation with ethical responsibility in healthcare information delivery. Methods Study Design This study was conducted to evaluate the performance of ChatGPT 4 (March 23rd, 2023) Model) in the context of genetic counseling and education. The evaluation involved a structured questionnaire, which included questions selected from the Brief User Survey (BUS-15) and additional custom questions designed to assess the clinical value of ChatGPT 4's responses. Questionnaire Development The questionnaire was built on Qualtrics, which comprised twelve questions: seven selected from the BUS-15 preceded by two additional questions that we designed. The initial questions focused on quality and answer relevancy: 1. The overall quality of the Chatbot’s response is: (5-point Likert: Very poor to Very Good) 2. The Chatbot delivered an answer that provided the relevant information you would include if asked the question. (5-point Likert: Strongly disagree to Strongly agree) The BUS-15 questions (7-point Likert: Strongly disagree to Strongly agree) focused on: 1. Recognition and facilitation of users’ goal and intent: Chatbot seems able to recognize the user’s intent and guide the user to its goals. 2. Relevance of information: The chatbot provides relevant and appropriate information/answer to people at each stage to make them closer to their goal. 3. Maxim of quantity: The chatbot responds in an informative way without adding too much information. 4. Resilience to failure: Chatbot seems able to find ways to respond appropriately even when it encounters situations or arguments it is not equipped to handle. 5. Understandability and politeness: The chatbot seems able to understand input and convey correct statements and answers without ambiguity and with acceptable manners. 6. Perceived conversational credibility: The chatbot responds in a credible and informative way without adding too much information. 7. Meet the neurodiverse needs: Chatbot seems able to meet needs and be used by users independently form their health conditions, well-being, age, etc. Expert Panel and Data Collection A panel of experts (two genetic counselors and two clinical geneticists) was provided with a link to the survey containing the questions. They independently evaluated the responses from ChatGPT 4 without discussing the questions or answers among themselves until after the survey submission. This approach ensured unbiased evaluation.

This entry contains the data associated with the publication "As if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI". The shared data refer to the annotations provided by a 13 radiologists on a collection of knee MRI images and is structured in 4 files:

• target.csv: the ground truth annotations, describing for each case whether it was abnormal (either ACL tear or Meniscus tear) or not;
• classes.csv: the annotations provided by the radiologists;
• confidenze.csv: the self-perceived confidences reported by the radiologists for each of the annotated cases;
• difficolta.csv: the radiologists' perceived complexity, for each case

The abstract of the paper is:

Background

We focus on the importance of interpreting the quality of the labeling used as the input of predictive models to understand the reliability of their output in support of human decision-making, especially in critical domains, such as medicine.

Methods

Accordingly, we propose a framework distinguishing the reference labeling (or Gold Standard) from the set of annotations from which it is usually derived (the Diamond Standard). We define a set of quality dimensions and related metrics: representativeness (are the available data representative of its reference population?); reliability (do the raters agree with each other in their ratings?); and accuracy (are the raters’ annotations a true representation?). The metrics for these dimensions are, respectively, the degree of correspondence, Ψ, the degree of weighted concordance ϱ, and the degree of fineness, Φ. We apply and evaluate these metrics in a diagnostic user study involving 13 radiologists.

Results

We evaluate Ψ against hypothesis-testing techniques, highlighting that our metrics can better evaluate distribution similarity in high-dimensional spaces. We discuss how Ψ could be used to assess the reliability of new predictions or for train-test selection. We report the value of ϱ for our case study and compare it with traditional reliability metrics, highlighting both their theoretical properties and the reasons that they differ. Then, we report the degree of fineness as an estimate of the accuracy of the collected annotations and discuss the relationship between this latter degree and the degree of weighted concordance, which we find to be moderately but significantly correlated. Finally, we discuss the implications of the proposed dimensions and metrics with respect to the context of Explainable Artificial Intelligence (XAI).

Conclusion

We propose different dimensions and related metrics to assess the quality of the datasets used to build predictive models and Medical Artificial Intelligence (MAI). We argue that the proposed metrics are feasible for application in real-world settings for the continuous development of trustable and interpretable MAI systems.

Content Intelligence Market size was valued at USD 1.69 Billion in 2024 and is projected to reach USD 12.26 Billion by 2032, growing at a CAGR of 31% during the forecast period 2026-2032.

Global Content Intelligence Market Drivers

The market drivers for the Content Intelligence Market can be influenced by various factors. These may include:

Rise in Content Generation and Consumption: Tools and solutions that can evaluate and optimize content generation, delivery, and performance are in high demand due to the volume of digital material that is being produced and consumed across a variety of platforms, including websites, social media, and other digital channels.

Emphasis on Personalization: To improve user engagement and customer experiences, businesses are implementing content personalization tactics more and more. In order to deliver individualized and targeted information, content intelligence tools are essential for assessing user behavior, preferences, and interactions.

Developments in Artificial Intelligence (AI) and Machine Learning (ML): The growth of the Content Intelligence Market has been largely attributed to the advancements in AI and ML technologies. These technologies make it possible to analyze large amounts of content data in a sophisticated way, which improves insights, makes suggestions, and automates operations related to content.

Increasing Focus on Data-driven Decision Making: Decision-making processes that are based on data are receiving more attention from organizations. Content intelligence solutions enable enterprises to make well-informed decisions by offering insightful data on user interaction, content performance indicators, and other pertinent topics.

Content Optimization for SEO: Businesses are using Content Intelligence technologies to optimize their content for search engines, given the significance of search engine optimization (SEO) for online visibility. These tools assist in determining pertinent keywords, examining rival websites, and raising the standard of material in general for higher search engine ranks.

Growing Demand for Content Intelligence: technologies to Analyze and Optimize Product Descriptions, Marketing Content, and Overall Digital Strategies: The growth of e-commerce and Digital Marketing activities has driven the need for Content Intelligence technologies.

Regulatory Compliance and Content Governance: The use of content intelligence technologies to make sure that material complies with legal requirements, protects data security, and respects privacy has been fueled by the need to comply with regulations, particularly in sectors like healthcare and finance.

Emergence of Conversational AI and Chatbots: As conversational AI, chatbots, and virtual assistants are used more often, there is a growing demand for Content Intelligence solutions that can evaluate and comprehend user queries to provide more effective and contextually relevant responses.

Cross-Channel Content Management: Companies are in charge of handling content on a variety of platforms, such as social media, mobile apps, websites, and more. Solutions for content intelligence make it easier to handle several channels efficiently, guaranteeing brand consistency and consistent messaging.

Demand for Real-time Insights: The popularity of content intelligence technologies has been fueled by the demand for real-time insights regarding user interactions and content performance. Capabilities for real-time analytics and reporting enable companies to quickly adapt their content strategies.

The global patent analysis consulting services market is experiencing robust growth, driven by increasing intellectual property (IP) litigation, a rising need for strategic patent portfolio management, and the expanding complexity of technological advancements. The market, currently valued at approximately $2 billion in 2025 (estimated based on typical market sizes for similar consulting services and a reasonable CAGR), is projected to witness a Compound Annual Growth Rate (CAGR) of 12% between 2025 and 2033. This expansion is fueled by several key trends, including the growing adoption of AI and machine learning for patent analysis, a surge in outsourcing of patent-related tasks to specialized consulting firms, and the increasing demand for services that focus on patent infringement analysis and risk assessment. Large enterprises are currently the largest segment, however, the mid-sized and small company segments are showing rapid growth as awareness of the strategic importance of patents and the value of expert analysis increases. The market is geographically diverse, with North America and Europe currently holding significant shares, while Asia Pacific is expected to show the fastest growth in the coming years fueled by economic expansion and increasing R&D investment. Despite the overall positive outlook, the market faces certain constraints. These include the high cost of patent analysis services, the need for specialized expertise, and the potential for variations in service quality across providers. Competitive landscape is fragmented, with both established players such as Clarivate Analytics and CPA Global, and niche providers specializing in particular applications. To differentiate themselves, many firms are focusing on offering integrated solutions combining analysis with strategic consulting, providing clients with a complete IP management strategy. The growing complexity of patent laws across various jurisdictions is also driving demand for experienced consultants able to navigate this challenging regulatory landscape. The successful players will be those who can demonstrate consistent quality, cost-effectiveness, and deep expertise across various technical domains.

As internet use for obtaining health information increases with the additional integration of artificial intelligence, more emphasis should be placed to design with these platforms with systematically excluded populations, such as low English literacy, racial and ethnic minority groups, and those of diverse gender and sexual orientations, in mind. Online platform developers are in need of a resource that includes evidence-based inclusive practices that can be included in platform development. Rapid Realist Review (RRR) methodology was performed to identify key considerations, strategies, and barriers to incorporating health equity principles into online health information platforms. A comprehensive PubMed search yielded 334 articles, which were screened through title and abstract review, two full-text evaluation, and AI-assisted summary analysis, leading to the final inclusion of 41 articles. Five key themes emerged from the literature: (1) Enhancing the readability of online content; (2) addressing public perceptions of disease and providing behavior cues; (3) creating culturally relevant content; (4) ensuring diversity in visual aids; and (5) enhancing consumer trust in information on the platform. ... [Read More]

Health protection agencies require scientific information for evidence-based decision-making and guideline development. However, vetting and collating large quantities of published research to identify relevant high-quality studies is a challenge. One approach to address this issue is the use of adverse outcome pathways (AOPs) that provide a framework to assemble toxicological knowledge into causally linked chains of key events (KEs) across levels of biological organization to culminate in an adverse health outcome of significance to regulatory decision-making. Traditionally, AOPs have been constructed using a narrative review approach where the collection of evidence that supports each pathway is based on prior knowledge of influential studies that can also be supplemented by individually selecting and reviewing relevant references. We aimed to create a protocol for AOP weight of evidence gathering that harnesses elements of both scoping review methods and artificial intelligence (AI) tools to increase transparency while reducing bias and workload of human screeners. To develop this protocol, an existing space-health AOP in the workplan of the Organisation for Economic Co-operation and Development (OECD) AOP Programme was used as a case example. To balance the benefits of both scoping review tools and narrative approaches, a study protocol outlining a screening and search strategy was developed, and three reference collection workflows were tested to identify the most efficient method to inform weight of evidence. The workflows differed in their literature search strategies, and combinations of software tools used. Across the three tested workflows, over 59 literature searches were completed, retrieving over 34,000 references of which over 3300 were human reviewed. The most effective of the three methods used a search strategy with searches across each component of the AOP network, SWIFT Review as a pre-filtering software, and DistillerSR to create structured screening and data extraction forms. This methodology effectively retrieved relevant studies while balancing efficiency in data retrieval without compromising transparency, leading to a well-synthesized evidence base to support the AOP. The workflow is still exploratory in the context of AOP development, and we anticipate adaptations to the protocol with further experience. To further the systematicity, future iterations of the workflow could include structured quality assessment and risk of bias analysis. Overall, the workflow provides a transparent and documented approach to support AOP development, which in turn will support the need for rigorous methods to identify relevant scientific evidence while being practical to allow uptake by the broader community.

List of search terms to be applied covering the AI/ML intervention terms and PE outcome terms.

Background and objectivesCrohn’s disease (CD), a complex member of the inflammatory bowel disease spectrum, is characterized by the diversity and skipping distribution of intestinal mucosal lesions, significantly complicating its differential diagnosis with intestinal diseases such as ulcerative colitis and intestinal tuberculosis. With the increasing application of artificial intelligence (AI) in the medical field, its utilization in primary diagnosis has become more widespread. However, there is a lack of systematic evaluation regarding the specific efficacy of AI in identifying CD through capsule endoscopy.MethodsThis study conducted a comprehensive search of PubMed databases, Cochrane, EMBASE, and Web of Science up to May 21, 2024, to collect relevant literature. The Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool was used to rigorously assess the quality of included studies, and detailed information on study characteristics and AI algorithms was extracted. A bivariate mixed-effects model was employed to synthesize and analyze the sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). Additionally, meta-regression and subgroup analyses were conducted to delve into the potential sources of heterogeneity.ResultsUltimately, eight studies encompassing 11 distinct AI models were included in this meta-analysis. The overall area under the curve (AUC) for AI in identifying CD through capsule endoscopy was 99% (95% CI, 100%-0.00), indicating high diagnostic accuracy. Specifically, the pooled sensitivity was 94% (95% CI, 93–96%), specificity was 97% (95% CI, 95–98%), positive likelihood ratio (PLR) was 32.7 (95% CI, 19.9–53.6), negative likelihood ratio (NLR) was 6% (95% CI, 4–7%), and diagnostic odds ratio (DOR) reached 576 (95% CI, 295–1,127). Meta-regression analysis further revealed that AI algorithm type, study population size, and study design might be key sources of heterogeneity.ConclusionThis study demonstrates the significant potential of AI technology in assisting endoscopists in detecting and identifying CD patients through capsule endoscopy. However, given the limitations and heterogeneity of current research, more high-quality, large-sample studies are needed to comprehensively and thoroughly evaluate the practical application value of AI in CD diagnosis, thereby promoting its widespread adoption and optimization in clinical practice.

The rapid development of computational approaches for predicting the structures of T cell receptors (TCRs) and TCR-peptide-major histocompatibility (TCR-pMHC) complexes, accelerated by AI breakthroughs such as AlphaFold, has made it feasible to calculate these structures with increasing accuracy. Although these tools show great potential, their relative accuracy and limitations remain unclear due to the lack of standardized benchmarks. Here, we systematically evaluate seven tools for predicting isolated TCR structures together with six tools for predicting TCR-pMHC complex structures. The methods include homology-based approaches, general prediction tools using AlphaFold, TCR-specific tools derived from AlphaFold2, and the newly developed tFold-TCR model. The evaluation uses a post-training data set comprising 40 αβ TCRs and 27 TCR-pMHC complexes (21 Class I and 6 Class II). Model accuracy is assessed at global, local, and interface levels using a variety of metrics. We find that each tool offers distinct advantages in various aspects of its predictions. AlphaFold2, AlphaFold3, and tFold-TCR excel in overall accuracy of TCR structure prediction, and TCRmodel2 and AlphaFold2 perform well in overall accuracy of TCR-pMHC structure prediction. However, TCR-specific tools derived from AlphaFold2 show lower accuracy in the framework region than both homology-based methods and general-purpose tools such as AlphaFold, and challenges remain for all in modeling CDR3 loops, docking orientations, TCR-peptide interfaces, and Class II MHC-peptide interfaces. These findings will guide researchers in selecting appropriate tools, emphasize the importance of using multiple evaluation metrics to assess model performance, and offer suggestions for improving TCR and TCR-pMHC structure prediction tools.

Classifying scientific publications according to Field-of-Science taxonomies is of crucial importance, powering a wealth of relevant applications including Search Engines, Tools for Scientific Literature, Recommendation Systems, and Science Monitoring. Furthermore, it allows funders, publishers, scholars, companies, and other stakeholders to organize scientific literature more effectively, calculate impact indicators along Science Impact pathways and identify emerging topics that can also facilitate Science, Technology, and Innovation policy-making. As a result, existing classification schemes for scientific publications underpin a large area of research evaluation with several classification schemes currently in use. However, many existing schemes are domain-specific, comprised of few levels of granularity, and require continuous manual work, making it hard to follow the rapidly evolving landscape of science as new research topics emerge. Based on our previous work of scinobo, which incorporates metadata and graph-based publication bibliometric information to assign Field-of-Science fields to scientific publications, we propose a novel hybrid approach by further employing Neural Topic Modeling and Community Detection techniques to dynamically construct a Field-of-Science taxonomy used as the backbone in automatic publication-level Field-of-Science classifiers. Our proposed Field-of-Science taxonomy is based on the OECD fields of research and development (FORD) classification, developed in the framework of the Frascati Manual containing knowledge domains in broad (first level(L1), one-digit) and narrower (second level(L2), two-digit) levels. We create a 3-level hierarchical taxonomy by manually linking Field-of-Science fields of the sciencemetrix Journal classification to the OECD/FORD level-2 fields. To facilitate a more fine-grained analysis, we extend the aforementioned Field-of-Science taxonomy to level-4 and level-5 fields by employing a pipeline of AI techniques. We evaluate the coherence and the coverage of the Field-of-Science fields for the two additional levels based on synthesis scientific publications in two case studies, in the knowledge domains of Energy and Artificial Intelligence. Our results showcase that the proposed automatically generated Field-of-Science taxonomy captures the dynamics of the two research areas encompassing the underlying structure and the emerging scientific developments.