Establishment of Automatic Word Segmentation Technology from Large-scale Text Data Independent of Language

In this research, we constructed a word segmentation technology that processes text data that is mixed with multiple languages expressed in Unicode with the same program. This technique is a language-independent word segmentation method based on a simple state transition model that does not require any dictionary or grammatical knowledge for each language. The research proceeded mainly in two directions: (1) extension of the language to be processed and (2) extension of application cases. Regarding (1), We confirmed that it is effective not only for Japanese but also for Japanese classics and foreign languages such as English, Chinese, and Korean. Regarding (2), we were able to propose a method for automatically creating an emotional polarity dictionary using user reviews of products and facilities.