Skip to main content
 首页 » 编程设计

python之使用 NLTK 无法使用 Unicode 字符标记西类牙语文本

2025年05月04日9jackei

我正在尝试解析一些包含非 ascii 字符(主要是单词中的重音...例如:película(电影)、atención(注意)等)的西类牙语句子。

我正在从使用 utf-8 编码的文件中读取行。这是我的脚本示例:

# -*- coding: utf-8 -*- 
 
import nltk 
import sys 
from nltk.corpus import cess_esp as cess 
from nltk import UnigramTagger as ut 
from nltk import BigramTagger as bt 
 
f = codecs.open('spanish_sentences', encoding='utf-8') 
results_file = codecs.open('tagging_results', encoding='utf-8', mode='w+') 
 
for line in iter(f): 
 
    output_line =  "Current line contents before tagging->" + str(line.decode('utf-8', 'replace')) 
    print output_line 
    results_file.write(output_line.encode('utf8')) 
 
    output_line = "Unigram tagger->" 
    print output_line 
    results_file.write(output_line) 
 
    s = line.decode('utf-8', 'replace') 
    output_line = tagger.uni.tag(s.split()) 
    print output_line 
    results_file.write(str(output_line).encode('utf8')) 
 
f.close() 
results_file.close() 

在这一行:

output_line = tagger.uni.tag(s.split()) 

我收到这个错误:

/usr/local/lib/python2.7/dist-packages/nltk-2.0.4-py2.7.egg/nltk/tag/sequential.py:138: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal 
return self._context_to_tag.get(context) 

这是一个简单句子的一些输出:

Current line contents before tagging->tengo una queja y cada que hablo a atención me dejan en la linea media hora y cortan la llamada!! 
 
Unigram tagger-> 
[(u'tengo', 'vmip1s0'), (u'una', 'di0fs0'), (u'queja', 'ncfs000'), (u'y', 'cc'), (u'cada', 'di0cs0'), (u'que', 'pr0cn000'), (u'hablo', 'vmip1s0'), (u'a', 'sps00'), (u'atenci\xf3n', None), (u'me', 'pp1cs000'), (u'dejan', 'vmip3p0'), (u'en', 'sps00'), (u'la', 'da0fs0'), (u'linea', None), (u'media', 'dn0fs0'), (u'hora', 'ncfs000'), (u'y', 'cc'), (u'cortan', None), (u'la', 'da0fs0'), (u'llamada!!', None)] 

如果我从 this chapter 理解正确的话...过程是正确的...我将行从 utf-8 解码为 Unicode,标记,然后再次从 Unicode 编码为 utf-8...我不明白这个错误

知道我做错了什么吗?

谢谢, 亚历杭德罗

编辑:发现问题...基本上西类牙语 cess_esp 语料库是用 Latin-2 编码编码的。请参阅下面的代码,了解如何正确训练标记器。

tagged_sents = ( 
[(word.decode('Latin2'), tag) for (word, tag) in sent] 
for sent in cess.tagged_sents() 
) 
tagger = UT(tagged_sents)  # training a tagger 

更好的方法是使用 CorpusReader类来请求语料库编码,因此您不需要事先知道它。

请您参考如下方法:

可能您的标记器对象或您的文件读取方式有问题。我重写了你的部分代码,它运行没有错误:

# -*- coding: utf-8 -*- 
 
import urllib2, codecs 
 
from nltk.corpus import cess_esp as cess 
from nltk import word_tokenize 
from nltk import UnigramTagger as ut 
from nltk import BigramTagger as bt 
 
tagger = ut(cess.tagged_sents()) 
 
url = 'https://db.tt/42Lt5M5K' 
fin = urllib2.urlopen(url).read().strip().decode('utf8') 
fout = codecs.open('tagger.out', 'w', 'utf8') 
for line in fin.split('\n'): 
    print>>fout, "Current line contents before tagging->", line 
    print>>fout, "Unigram tagger->", 
    print>>fout, tagger.tag(word_tokenize(line)) 
    print>>fout, "" 

[输出]:

http://pastebin.com/n0NK574a