Dataset

This dataset was created by jastorj

Contents

Dataset

This dataset was created by Stuti

Released under CC0: Public Domain

Contents

It contains the following files:

Hindi Text Short Summarization Corpus is a collection of ~330k articles with their headlines collected from Hindi News Websites.

This is a first of its kind Dataset in Hindi which can be used to benchmark models for Text summarization in Hindi. This does not contain articles contained in Hindi Text Short and Large Summarization Corpus which is being released parallely with this Dataset.

The dataset retains original punctuation, numbers etc in the articles.

There is an increasing demand for sentiment analysis of text from social media which are mostly code-mixed. Systems trained on monolingual data fail for code-mixed data due to the complexity of mixing at different levels of the text. However, very few resources are available for code-mixed data to create models specific for this data. Although much research in multilingual and cross-lingual sentiment analysis has used semi-supervised or unsupervised methods, supervised methods still performs better. Only a few datasets for popular languages such as English-Spanish, English-Hindi, and English-Chinese are available. There are no resources available for Malayalam-English code-mixed data. This paper presents a new gold standard corpus for sentiment analysis of code-mixed text in Malayalam-English annotated by voluntary annotators. This gold standard corpus obtained a Krippendorff’s alpha above 0.8 for the dataset. We use this new corpus to provide the benchmark for sentiment analysis in Malayalam-English code-mixed texts.

Content

Sentiment analysis – otherwise known as opinion mining – is a much bandied about but often misunderstood term. In essence, it is the process of determining the emotional tone behind a series of words, used to gain an understanding of the the attitudes, opinions and emotions expressed within an online mention.

Stop words are frequent, evenly distributed, function words in any document corpus which does not add any meaning to the text content. Information retrieval from the corpus is not getting affected by the removal of these words. It has been proved that removing the stop words reduces the document size to a considerable extent and saves time in text processing in Natural Language Processing.

This dataset contains both positive and negative sentiment lexicons as well as stop words in Hindi.

Acknowledgements

If you use this in your work, please cite the following: - 10.17632/mnt3zwxmyn.2 - 10.17632/bsr3frvvjc.1

Inspiration

• To find words that essentially mean the same

As we all well aware Internet is dominated by English and finding resources for other languages (especially one's from the developing world) is hard to near impossible, so this is my small effort to bring some of the well known works from the world of Hindi to Kaggle, so people can experiment and work with the same. I am starting out with Premchand, will try to add more authors over time.

This corpus contain all the work of Munshi Premchand who is beloved figure in the world Hindi Literature, I have aggregated this dataset from multiple websites which host work of Munshi Premchand. The file is TSV, where each row is individual work, and some meta data associates with the work, i.e. Title, Work Type (Story/Novel)

First thing that comes to mind is text generation, one can start out with very naïve methods and work your way up to more complex methods. Also textual style transfer is one of the thing that can be experimented, as Premchand was very much know for his writing style as much as for the stories themselves.