Thursday, January 06, 2011

Natural Language Processing Tools for Chichewa

Natural language processing (NLP) is an excellent discipline of computer science focusing on developing artificial intelligence systems that are able to interact with human beings in their natural languages. The expert systems so developed try to understand patterns of human languages and process the given data (text or speech) accordingly. The expert tools are very useful in various systems. For example some feature/smart phones are able to read the name of the caller for you as the phone rings. Other applications like word processors are able to read a big document and generate for you a summary document from it. We also have nice applications (e.g. Google Translate) that read text in one  human language and translate it into another target language. All these are products of the field of NLP.

I have been working on big projects for Chichewa, a lingua franca for Malawi (formerly it's national language), Zambia and some parts of Mozambique and Zimbabwe. These systems are still in progress of perfection, but they currently are able to do great stuff at this stage. I am sure that the end of these projects will put Chichewa somewhere as far as NLP is concerned. I would like to share with you my experiences.

ChicMorph: A Morphological Analyzer for Chichewa Verbs
    Chichewa is agglutinative in nature. One word/phrase  is a combination of several sub-words (techinically called morphemes). For example sindibweranso (Lit: I am not coming again) can be broken as follows: si(not)-ndi(I)-bwer(come)-a-nso(again). Notice that the "a" has no literal meaning. It is just a final vowel to complement bwer, the stem of that verb.

    ChicMorph takes raw Chichewa verbs, discovers and isolates the verb constituent morphemes. Some Chichewa verbs are tricky in that their roots also include subwords (morphemes) that are also morphemes on their own. For example, er is an applicaticative morpheme as in gwera (gw-er-a). But it is not  a morpheme in bwera (hence bw-er-a is incorrect, but bwer-a). I have so far improved ChicMorph to evaluate correctly verbs with roots constituting morphemes that are also prefixal or suffixal allomorphs like these ones.
     ChicPOS: Part of Speech Tagger
      From August 2010, I have been working on a Chichewa part of speech tagger and it is doing great. I am hoping to make more breakthroughs in due course. Right now, chicPOS understands all Chichewa parts of speech including punctuations:
      •  Mwana womaliza uja wa a Phiri wabwera kudzagula mchere. (Lit: That last born child to Mr. Phiri has come to buy salt.) => Mwana[NN] womaliza[JJ] uja[DEM] wa[IN] a[HON] Phiri[NNP] wabwera[VB] kudzagula[VB] mchere[NN] .[.]
      Key: DEM => Demonstrative Adjective, HON => Honorific a, IN =>Preposition, JJ => Adjective, NN => Noun, NNP => Proper Noun, POSS => Possessive Adjective, PN => Pronoun, PRP => Personal Pronoun, VB =>  Verb.

      ChicPOS is also able to identify proper nouns within a given phrase. Compare usage of "Talandira" in the following phrases:
      •  Talandira ndalama kuchokera kwa a Chikale. (Lit: We have received money from Mr. Chikale.) => Talandira[VB] ndalama[NN] kuchokera[VB] kwa[ASSOC] a[HON] Chikale[NNP] .[.]
      • Ndamuona Talandira akudutsa apa. (Lit: I have seen Talandira passing by here.) => Ndamuona[VB] Talandira[NNP] akudutsa[VB] apa[DEM] .[.]
      ChicPOS fails to identify proper nouns in some positions, especially when they begin a sentence as in Talandira akudutsa apa. (Lit: Talandira is passing by here). =>  Talandira[VB] akudutsa[VB] apa[DEM] .[.] (compare it with: Akudutsa apa Talandira. (Lit: He/She is passing by here, Talandira) => Akudutsa[VB] apa[DEM] Talandira[NNP] .[.]). Proper nouns are tricky even in "natural/daily conversations" looking at the way names(Proper nouns) are formed in Chichewa. Some proper names originate from verbs/verb phrases (e.g. Talandira => (Lit: We have received), Kalinda-kadye (Lit: It waits to eat)) while others from common nouns (Chipiriro (Patience), Ulemu (Politeness/Respect)). Notice that somehow ChicPOS is also correct  in this special case: Talandira akudutsa apa. =>  Talandira[VB] akudutsa[VB] apa[DEM] .[.] The reason is since Talandira originates from a verb , by just changing the tone of the phrase Talandira akudutsa apa. will translate to We have recieived (something) while he was passing here.. In short, I should say I am still exploring this concept of proper nouns. 

      Right now, I have six thousand Chichewa words (thanks to Prof. Kevin Scannell for compiling the initial wordlist using his An Crubádan). I am in the process of tagging them, and I will be adding some more words. A note on tags, I have tried to preserve popular tags like NN, JJ but for words that I could not find one I prioritized short forms outlined in The Syntax of Chichewa by Prof. Sam Mchombo. Otherwise, I generated my own. I am hoping to create a standardized form for Chichewa (and eventually for other Malawian languages). I am also looking at some similar work in Swahili and Nguni languages.

      AffixGen: Chichewa Verb Generator
        In due course, I also developed a "Chichewa verb generator". It automatically generates 66082 prefixes, 2870 suffixes (using CARP [Causative-Applicative-Reciprocal-Passive] and RCAP suffix combination; in RCAP the reciprocal precedes the other suffixes as in menyanitsa). The suffix extension can take up to three clitics at the moment. For each single verb root, it generates 66082 x 2870 = 189,655,340 possible verb forms. This is awesome because if you have 10 Chichewa verb roots, you are able to generate close to 2 billion Chichewa verbs!! Of course some of them may not be as sensible due to some semantical encodings behind them (compare menya and bwera => akuzimenyanitsa vs akuzibweranitsa.) I am still working on this. I would like to collect all(?????) verb roots (Ha!Ha!Ha! if I can manage) and isolate them accordingly so that such funny combinations do not occur any more, or at least the error rate is reduced drastically. Right now I have 500 verb roots and the system is able to generate 94,827,670,000 (94 Billion) verbs!!!!

        I am using AffixGen output to build plugins for Hunspell spellchecker, and I have so far created two plugins, one for Firefox and another for OpenOffice (It is available online on website. Of course, the online one is not up to date yet). 

        ChiVisualize: Dynamic Visualization Tool of Chichewa Phrase Structures
          In line with ChicPOS, I am creating a visualization tool for Chichewa phrase structures. This is another great art work that I have ventured into. ChiVisualize text tagged phrases and build a syntax tree as in the following example:
          Mkango[NN] uja[DEM] ukuba[VB] mikanda[NN] yanu[POSS] (That lion is stealing your beads.)
          The syntax tree is interactive and dynamic. You can change the orientation in four directions: top, bottom, left and right. You can also emphasize on a particular level in any of the sub-trees. The system is able to "virtually" simulate the all six Chichewa phrase structures: SVO, SOV, VSO, VOS, OVS and OSV. ChiVisualize uses JavaScript InfoVis Toolkit to create these interactive visualization.

          ChiVisualize is still in its formative stages. Right now, the tagged text is processed into a base phrase structure (as defined by Chomsky's Minimalist Theory/Program) manually and given to ChiVisualize for syntax tree generation. Currently, I am working on an algorithm that will be able to automatically generate a Base Phrase Structure for given tagged text.

          Later on, I will combine ChicPOS, ChicMorph and ChiVisualize into one application. With the new system, one will just be giving it a "normal" Chichewa Phrase and it will be doing all the processing itself. ChicPOS will be generating tagged text and give it to ChiVusualize for visualization. On the other hand, ChicMorph will produce extra morphological constraints that will be displayed when one emphasizes on a certain phrase constituent in a given syntax tree. I will also add a transformational-generative grammar parser that will be able to resolve matching of argument markers to their respective nominals if present in a given phrase. Of course, I am aware of ambiguities resulting from free word ordering and NPs from same classes as depicted in the following: Galimoto ng'ombe yayigunda (galimoto => car, ng'ombe => cow, yayigunda => 'has hit'). (which one hit the other here? FYI: galimoto and ng'ombe fall in the same noun class). One way will be to leave it strictly non-configurational such that the phrase will be illustrated as S = NP + NP + V (or any of its combinations, without a VP). But I'll cross the bridges when I'll come to them :-), :-).

          By the end of everything, I would like to build a head-driven phrase structure grammar checker for Chichewa. This will be useful not only in linguistics, but also in real-world applications like word processing software.


          1. This is a revolutionary revelation for studies in Malawian Languages.My preliminary response is on the completeness of this package. As you have hinted elsewhere in your blog; tone is very crucial in Bantu languages. Without tagging tone patterns we will end up with excessive ambiguity. The good news however, is that tone has been one of the most researched areas in Malawian languages.These studies reveal interesting consistency on tone assignment especially in verbs (Noun tones are proved to be idiosyncratic ). For example; morphology plays a crucial role in tone assignment as tense prefixes assign High Tone on various positions, like penultmate vowel, antepenult vowel etc. Compare duts-a (LL) a-ku-DUts-a (LLHL-present progressive),A-KU-DU-ts-a (HHHL-past progressive).

            The point I want to make here is that this behaviour of tones is predictable and devising an algorithm for a Chichewa tone tags (ChiTonizer) would be viable . In addition to the good work you have already started consider adding this functionality to the package.

            This is an informal response so I did not acknowledge any source but they are all inspired by Mtenje's PhD thesis and various articles he has authored individually or with partners.


          2. I am really impressed with your hypothesis on the possibility of developing a tone tagging system. I am still interested to find out how one can recognize the tone difference in a written sentence like this:
            Ndilimbira akudutsa apa
            For me it is still difficult to know whether the statement should be read as:
            Ndilimbira aKUDUtsa apa (lit: Ndilimbira is passing by here [present continuous tense]) .... 1
            or as:
            Ndilimbira AKUDUtsa apa (lit: Ndilibira is going to pass by you here [immediate future]) .... 2

            Notice the change of the role of ku the verb akudutsa in both statements. In the former (1), ku is a tense marker. In the latter (2) ku is an object marker.

            This should be a nice research, though!!!

          3. I really like this information . thanks for sharing such a beautiful information with us. I would like to share some more information on Foreign Language Career . Please share your feedback too

          4. Thanks for sharing the pretty post, if you focus the certification training to enhance your skill for attaining good job in IT industry, for that you Can take the valuable certification training of Language course ( German Language Course , French language Course , Spanish Language Course ) for your career growth.

          5. Is there a github project publicly available to leverage ChicPOS and ChicMorph

          6. This is the best place to get cheats, codes, cheat codes, walkthrough, guide, FAQ, unlockables, tricks, and secrets for The Sims 4 for PC-Sims 4 cheats.

          7. Very informative blog . But i also want the world to know There is one company in my mind, its name is AFM Logistics pvt Ltd .It gives best quality logistics service ,ie is AFM Logistics Pvt Ltd is an international freight forwarding and customs clearing company established in Delhi. The company was constituted in 2012 and is indulged in providing complete logistics solution. The company has its own setup and wide network of agents throughout the world. International Logistics Companies In India . They are the best air cargo and ocean freight forwarding company in Delhi, India. AFM Logistics Pvt Ltd has been working as Import and Export Agent in India since 2012. They have been providing personal baggage shipping services in India for a very long time.