Collocation search & Mutual Information/Technical Term Extraction Criteria(Keyness)

[Dongkwang Shin]

Collocation Extract

http://pioneer.chula.ac.th/~awirote/colloc/       

setupColloc.exe

           Collocation Extract is designed to provide a list of collocations in the corpus. Users can search for collocates of a particular word in the range of 2 to 5 words, or search for all collocations of two-word chunks. Three statistical methods, namely Dunning's Log Likelihood, (pointwise) Mutual Information, and Chi-square, are used in this software. Below is the steps in using the program  

Statistical Methods

Five statistical methods for identifying collocation are available in this program. They are Dunning's log likelihood, (pointwise) mutual information, chi-square, cubic association ratio (MI3), and Frager and McGowan coefficient. (See Manning and Schütze, 1999 and Oakes 1998, for more details)

1. Dunning's log likelihood :

The test compares two hypotheses about W1 and W2
Hypothesis 1 : P(w2|w1) = p = P(w2| not w1)
Hypothesis 2 : P(w2|w1) = p1   not equal    p2 = P(w2|not w1)
Assuming binomial distribution, the log likelihood ratio is calcuated as follows:

Note that c1 is the frequency of W1, c2 is the frequency of W2, c12 is the frequency of bigram W1-W2, N is the number of total words in the corpus, p = c2/N,  p1 = c12/c1, and  p2 = (c2 – c12) / (N-c1). The higher the value, the more likely that W2 is the collocation of W1.
To see the statistical significance, multiply the result by -2, and consult the Chi-square table at the degree of freedom as one.

http://ucrel.lancs.ac.uk/llwizard.html

2. Pearson's Chi-square

Collocation between W1 and W2 is calculated in according to the frequency of bigram occurences in each cell. "W1 - W2" represents the number of occurences that W1 - W2 is found in the corpus. "not W1 - W2" represents the number of occurences that the preceding word of W2 is not W1. "W1 - not W2" represents the number of occurences that W1 is not followed by W2. "not W1 - not W2" represents the rest.
 

W1 - W2	not W1- W2
W1 - not W2	not W1 - not W2

Chi-square is calculated using the formula below, where Oi,j is the observed frequency in the table, and Ei,j is the expected frequency in each cell when W1 - W2 occur together by chance. Expect frequency on each cell is equal to (row total * column total ) / grand total

The higher the number, the more significant of collocation between W1 and W2.

http://stattrek.com/online-calculator/chi-square.aspx

3. Mutual Information

This statistical method compare the probability of finding the two words together to the probability that the two words are independent to each other. If x' and y' and collocation, it is likely that P(x' y') be highly greater than P(x') * P(y'). Thus, the higher the value, the more likely that the two words are collocation.

However, mutual information tends to give high value for rare events. For example, when W1 and W2 always occur together, MI will be higher if the frequency is lower.

When calculating statistical values for 3-,4-,5-words, pseudo-bigram transformation is used to estimate the value. (Silva and Lopes, 1999).

References

• Manning, Christopher and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. Cambridge: MIT Press.
• Oakes, Micheal P. 1998. Statistics for Corpus Linguistics. Edinburg University Press.
• Silva, Joaquim Ferreira da, and Gabriel Pereira Lopes. 1999. A Local Maxima Method and a Fair Dispersion Normalization for Extraction Multi-word Units from Corpora. In Proceedings of the 6th Meeting on the Mathematics of Language, Orlando, July 23-25.

How To Use

1. First, select all files in the corpus (“File – New File List”). These files can be either plain text or annotated files, such as html, sgml, or xml files. Select the file type that matches the data. Once the File List is defined, users can save the list for future use, by clicking on “File-Save File List”.

2. Select one of the three statistical methods: log-likelihood, mutual information, and chi-square, or select “Raw Frequency” if you want to see only frequency of occurrences. The default is log-likelihood.

3. Select the span ranged from 2 to 5. The number indicates the number of words to look for collocations. For example, if “2 words” is selected, the program will search for collocations of two-word chunks. If “3 words” is selected, the program will search for collocations of three-word chunks.

4. Set all the options. First, set the direction for searching collocations by selecting “Option-Search Collocates on”. If “Left Side” is selected, the program looks for all collocates that occur before the keyword. The default is “Both Sides”. Select the minimum frequency of n-word collocations. This will instruct the program to look for only collocations that occur at least N times, where N is the number specified. Then, select the statistical significance at the level of “p > .005”, “p > .05”, or “all occurrences”. Set the maximum number of collocations to be extracted. The default is “500” items. When searching for 2-word collocations, users can specify the distance between the two words. If set as “2”, the two words are separated by one word. This option is provided because collocations sometimes can be separated by other words, such as “hold (oppositional) views”, “hold (a similar) view”, etc. The last option, “Ignore Header Tag”, is selected by default. This will instruct the program to ignore all information in the header tag <Header> ….. </Header>, which is encoded in sgml and xml files. (Information in the header tag is usually not the contents, but the bibliographic and encoding information.)

5. Specify the search. There are two ways to specify the search. The first one is to specify the keyword (“Search – Keyword”) to be searched. The second way is to search for all 2-word collocations “Search - All 2-word Collocations”. When searching for all 2-word collocations, users can specify the distance between the two words, as explained before.

6. When searching is finished, the collocation windows will display the list of collocates that co-occur with the keyword, sorted by order of significance. If two-word collocation is searched and the distance of the two words is greater than one, a number of underscore symbols, “_”, will be marked between the two words to indicate the distance. Users can save the collocation list by selecting “File-Save Colloc List”. The output will be saved as a text file with tab delimited between each column. The output file then can be imported into the Excel program for further use.

7. If users want to see the contexts of a particular word, they can click on that word, and select the menu bar “Concord”. The specified word and its collocate will be shown. Use “*” to mark for all words. Select the order of occurrences, and the number of characters for left and right contexts.

8. The concordance output will be shown in three columns. Users can save the result for further use by selecting “File - Save Concordance”. Although this concordance feature is made available in Collocation Extract, it is suggested that concordance software should be used, if users want to work extensively on concordance results. This feature is provided for users just to take a quick look at the contexts.

9. The program can list sequences of n-words from the corpus and the frequency of occurrences. Click on “n-word Frequency” menu, and specify the number of word sequence and the minimum frequency. For example, if n-word is set as “3”, and “Min Freq” as “5”, the program will list all sequences of three-word chunks that occur at least 5 times in the corpus. The result of this search will not be shown on the screen. It will be saved as a text file.

---

Download

Click here to download Collocation Extract for Windows systems. The current version is 3.06

---

This program is further developed from Collocation Test, which can handle only two-word collocations. The development of Collocation Test was sponsored by the Development Grants for New Faculty/Researchers in 1999, and a research grant from the Research Division of the Faculty of Arts in 2000.

Copyright 2000-4. Wirote Aroonmanakun.
Dept. of Linguistics, Chulalongkorn University