python之抓取网页的策略，最大限度地收集信息

问题是:

用户注册网站后可以从 8 个工作类别中选择一个，或者选择跳过此步骤。我想根据电子邮件地址中的域名将跳过该步骤的用户分类为工作类别。

当前设置:

结合使用 Beautiful Soup 和 nltk，我抓取了主页并查找指向网站上包含单词“about”的页面的链接。我也抓取了那个页面。我在这篇文章的末尾复制了一些执行抓取的代码。

问题:

我没有获得足够的数据来制定良好的学习程序。我想知道我的抓取算法是否为成功而设置——换句话说，我的逻辑中是否有任何漏洞，或者有什么更好的方法来确保我有大量的文本来描述什么样的工作一家公司呢？

(相关)代码:

import bs4 as bs 
import httplib2 as http 
import nltk 
 
 
# Only these characters are valid in a url 
ALLOWED_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~:/?#[]@!$&'()*+,;=" 
 
 
class WebPage(object): 
    def __init__(self, domain): 
        """ 
            Constructor 
 
            :param domain: URL to look at 
            :type domain: str 
        """ 
        self.url = 'http://www.' + domain 
 
        try: 
            self._get_homepage() 
        except: # Catch specific here? 
            self.homepage = None 
 
        try: 
            self._get_about_us() 
        except: 
            self.about_us = None 
 
    def _get_homepage(self): 
        """ 
            Open the home page, looking for redirects 
        """ 
        import re 
 
        web = http.Http() 
        response, pg = web.request(self.url) 
 
        # Check for redirects: 
        if int(response.get('content-length',251)) < 250: 
            new_url = re.findall(r'(https?://\S+)', pg)[0] 
            if len(new_url): # otherwise there's not much I can do... 
                self.url = ''.join(x for x in new_url if x in ALLOWED_CHARS) 
                response, pg = web.request(self.url) 
 
        self.homepage = self._parse_html(nltk.clean_html(pg)) 
        self._raw_homepage = pg 
 
    def _get_about_us(self): 
        """ 
            Soup-ify the home page, find the "About us" page, and store its contents in a 
            string 
        """ 
        soup = bs.BeautifulSoup(self._raw_homepage) 
        links = [x for x in soup.findAll('a') if x.get('href', None) is not None] 
        about = [x.get('href') for x in links if 'about' in x.get('href', '').lower()] 
 
        # need to find about or about-us 
        about_us_page = None 
        for a in about: 
            bits = a.strip('/').split('/') 
            if len(bits) == 1: 
                about_us_page = bits[0] 
            elif 'about' in bits[-1].lower(): 
                about_us_page = bits[-1] 
 
        # otherwise assume shortest string is top-level about pg. 
        if about_us_page is None and len(about): 
            about_us_page = min(about, key=len) 
 
        self.about_us = None 
        if about_us_page is not None: 
            self.about_us_url = self.url + '/' + about_us_page 
            web = http.Http() 
            response, pg = web.request(self.about_us_url) 
            if int(response.get('content-length', 251)) > 250: 
                self.about_us = self._parse_html(nltk.clean_html(pg)) 
 
    def _parse_html(self, raw_text): 
        """ 
            Clean html coming from a web page. Gets rid of 
                - all '\n' and '\r' characters 
                - all zero length words 
                - all unicode characters that aren't ascii (i.e., &...) 
        """ 
        lines = [x.strip() for x in raw_text.splitlines()] 
        all_text = ' '.join([x for x in lines if len(x)]) # zero length strings 
        return [x for x in all_text.split(' ') if len(x) and x[0] != '&'] 

请您参考如下方法:

这超出了您的要求，但我会考虑调用已收集此信息的外部数据源。 Programmable Web 是寻找此类服务的好地方。 (例如 Mergent Company Fundamentals )。 Programmable Web 上的数据并非都是最新的，但似乎有很多 API 提供商。