Over the weekend I met some students studying linguistics. Methods from Linguistics are very important for text retrieval and data mining. That is why in my oppinion Linguistics is also a very important part of web science. I am always concerned that most people doing web science actually are computer scientists and that much of the potential in web science is being lost by not paying attention to all the disciplines that could contribute to web science!

That is why I tried to teach the linguists some basic python in order to do some basic analysis on literature. The following script which is rather hacked than beautiful code can be used to analyse texts by different authors. It will display the following statistics:

  • Count how many words are in the text
  • count how many sentences
  • Calculate average words per sentence
  • Count how many different words are in the text
  • Count how many time each word apears
  • Count how many words appear only once, twice, three times, and so on…
  • Display the longest scentence in the text

you could probably ask even more interesting questions and analyze texts from different centuries, languages and do a lot of interesting stuff! I am a computer scientist / mathmatician I don’t know what questions to ask. So if you are a linguist feel free to give me feedback and suggest some more interesting questions (-:

Some statistics I calculated

ulysses
264965 words in 27771 sentences
==> 9.54 words per sentence

30086 different words
every word was used 8.82 times on average

faust1
30632 words in 4178 sentences
==> 7.33 words per sentence

6337 different words
==> every word was used 4.83 times on average

faust2
44534 words in 5600 sentences
==> 7.95 words per sentence

10180 different words
==> every word was used 4.39 times on average

Disclaimer

I know that this is not yet a tutorial and that I don’t explain the code very well. To be honest I don’t explain the code at all. This is sad. When I was trying to teach python to the linguists I was starting like you would always start: “This is a loop and that is a list. Now let’s loop over the list and display the items…” There wasn’t much motivation left. The script below was created after I realized that coding is not supposed to be abstract and an interesting example has to be used.

If people are interested (please tell me in the comments!) I will consider to create a python tutorial for linguists that will start right a way with small scripts doing usefull stuff.

by the way you can download the texts that I used for analyzing on the following spots

 Python |  copy code |? 
01
02
# this code is licenced under creative commons licence as long as you 
03
# cite the author: Rene Pickhardt / www.rene-pickhardt.de 
04
 
05
# adds leading zeros to a string so all result strings can be ordered
06
def makeSortable(w):
07
	l = len(w)
08
	tmp = ""
09
	for i in range(5-l):
10
		tmp = tmp + "0"
11
	tmp = tmp + w
12
	return tmp
13
 
14
#replaces all kind of structures passed in l in a text s with the 2nd argument
15
def removeDelimiter(s,new,l):
16
	for c in l:
17
		s = s.replace(c, new);
18
	return s;
19
 
20
def analyzeWords(s):
21
	s = removeDelimiter(s," ",[".",",",";","_","-",":","!","?","\"",")","("])
22
	wordlist = s.split()
23
 
24
	dictionary = {}
25
	for word in wordlist:
26
		if word in dictionary:
27
			tmp = dictionary[word]
28
			dictionary[word]=tmp+1
29
		else:
30
			dictionary[word]=1
31
 
32
	l = [makeSortable(str(dictionary[k])) + " # " + k for k in dictionary.keys()]
33
 
34
	for w in sorted(l):
35
		print w
36
	count = {}
37
 
38
	for k in dictionary.keys():
39
		if dictionary[k] in count: 
40
			tmp = count[dictionary[k]]
41
			count[dictionary[k]] = tmp + 1
42
		else:
43
			count[dictionary[k]] = 1
44
	for k in sorted(count.keys()):
45
		print str(count[k]) + " words appear " + str(k) + " times"
46
 
47
def differentWords(s):
48
	s = removeDelimiter(s," ",[".",",",";","_","-",":","!","?","\"",")","("])
49
	wordlist = s.split()
50
	count = 0
51
	dictionary = {}
52
	for word in wordlist:
53
		if word in dictionary:
54
			tmp = dictionary[word]
55
			dictionary[word]=tmp+1
56
		else:
57
			dictionary[word]=1
58
			count = count + 1
59
	print str(count) + " different words"
60
	print "every word was used " + str(float(len(wordlist))/float(count)) + " times on average"	
61
	return count
62
 
63
 
64
def analyzeSentences(s):
65
	s = removeDelimiter(s,".",[".",";",":","!","?"])
66
	sentenceList = s.split(".")
67
	wordList = s.split()
68
	wordCount = len(wordList)
69
	sentenceCount = len(sentenceList)
70
	print str(wordCount) + " words in " + str(sentenceCount) + " sentences ==> " + str(float(wordCount)/float(sentenceCount)) + " words per sentence"	
71
 
72
	max = 0
73
	satz = ""
74
	for w in sentenceList:
75
		if len(w) > max:
76
			max = len(w);
77
			satz = w;
78
	print satz + "laenge " + str(len(satz))
79
 
80
texts = ["ulysses.txt","faust1.txt","faust2.txt"]
81
for text in texts:
82
	print text
83
	datei = open(text,'r')
84
	s = datei.read().lower()
85
	analyzeSentences(s)
86
	differentWords(s)
87
	analyzeWords(s)
88
	datei.close()
89

If you call this script getstats.py on a linux machine you can pass the output directly into a file on which you can work next by using

python getstats.py > out.txt

If you like this post, you might like these related posts:

  1. Download Trec (= Text Retrieval Conference) Data Set Being back in University I get to see more and...
  2. Foundations of statistical natural language processing Review of chapter 1 Due to the interesting results we found by creating Typology...

Sharing:

Tags: , , , ,

1 Comment on Data mining (text analysis) for linguists on Ulysses by James Joyce & Faust by Goethe

  1. [...] processing. Some of which I already discussed without knowing that I did in the article about text mining for linguists on ulysses. In the book of course they have been discussed in a more structured way but one can easily [...]

Leave a Reply

*

Close

Subscribe to my newsletter

You don't like mail?