SciELO - Scientific Electronic Library Online

vol.14 issue2Computerized and comparative analysis of discourse in chronic organic patients author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand




  • Have no cited articlesCited by SciELO

Related links

  • Have no similar articlesSimilars in SciELO


Subjetividad y procesos cognitivos

On-line version ISSN 1852-7310


ALONSO ALEMANY, Laura. Linguistic Insights on the Lexical Normalization of user-generated content. Subj. procesos cogn. [online]. 2010, vol.14, n.2, pp.20-31. ISSN 1852-7310.

We present work in progress on word normalization for user-generated content. The approach is simple and helps in reducing the amount of manual annotation characteristic of more classical approaches. First, orthographic variants of a word, mostly abbreviations, are grouped together. From these manually grouped examples, we learn an automated classifier that, given a previously unseen word, determines whether it is an orthographic variant of a known word or an entirely new word. To do that, we calculate the similarity between the unseen word and all known words, and classify the new word as an orthographic variant of its most similar word. The classifier applies a string similarity measure based on the Levenshtein edit distance. To improve the accuracy of this measure, we assign edit operations an error-based cost. This scheme of cost assigning aims to maximize the distance between similar strings that are variants of different words. This custom similarity measure achieves an accuracy of .68, an important improvement if we compare it with the .54 obtained by the Levenshtein distance.

Keywords : Word normalization; Unseen words; Strings; Orthographic variants.

        · abstract in Spanish     · text in Spanish     · Spanish ( pdf )


Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License