Telugu corpus and cross-language POS tagging
A PES University NLP project that assembled a roughly 87 million-token Telugu corpus from books and news data, cleaned it with a Python and NLTK pipeline, and investigated part-of-speech tagging for Telugu and transfer to Kannada.
The brief
Telugu and Kannada are related Dravidian languages, and Kannada has far less labeled data. The question we were chasing: can Telugu resources help build a part-of-speech tagger for Kannada? Before any tagging, we needed a Telugu corpus big enough to be worth the effort.
What we built
- The corpus. We took Telugu books and Telugu news data from a Kaggle dataset and compiled them in Sketch Engine. It reported 87,306,725 tokens, 41,478,256 words and 651,624 sentences. That is a raw token count, not a hand-labeled training set.
- The cleaning pass. A notebook reads one sentence per line, normalizes curly quotes, long dashes and ellipses, strips punctuation and digits, drops tokens containing ASCII letters, removes words from a Telugu stop-word list using NLTK, and writes the result out as UTF-8 text.
The tokenization problem
Our first tagging attempt ran Telugu Wikipedia text through Sketch Engine. The output split Telugu words into individual characters and assigned a tag to each piece. Our project slides flag this as incorrect tokenization, and the saved output file shows it. A word-level tagger cannot produce reliable labels when the tokenizer gets the basic units wrong.
From there we looked at an external Telugu analyzer and a tag format that carries a main POS label plus morphological detail, and studied a published approach to cross-language tagging: a second-order HMM that takes transition information from Telugu and combines it with Kannada emissions or a Kannada lexicon. That method and its results belong to the paper’s authors. We read it as the reference for the transfer idea.
What is and is not here
The archive holds the corpus build, the cleaning notebook with input and output samples, and the failed tokenization experiment. It does not hold a finished Telugu-to-Kannada tagging pipeline or an evaluation, so this page makes no accuracy claim.
The cleaning code has limits I would fix today. It checks for non-ASCII characters instead of the Telugu Unicode block, reloads the stop-word list on every line, and removes every digit, including dates and quantities that might matter.
Credit
Built with Shreya Varma and Sathvik at PES University.