Skip to main content
 首页 » 编程设计

python之在网络抓取时深入网站

2024年12月31日7zhenyulu

我正在尝试抓取一堆网站的文本,以便我可以与语料库进行交叉验证并显示特定单词在这些网站上的点击次数。 谁能帮我让我的网络抓取工具自动深入网站。

import requests 
from bs4 import BeautifulSoup 
 
url = 'https://www.theleela.com/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/' 
page = requests.get(url)        #to extract page from website 
html = page.content 
 
soup = BeautifulSoup(html) 
 
# kill all script and style elements 
for script in soup(["script", "style"]): 
    script.extract()    # rip it out 
 
# get text 
text = soup.get_text() 
 
# break into lines and remove leading and trailing space on each 
lines = (line.strip() for line in text.splitlines()) 
# break multi-headlines into a line each 
chunks = (phrase.strip() for line in lines for phrase in line.split("  ")) 
# drop blank lines 
text = '\n'.join(chunk for chunk in chunks if chunk) 

我要求网页上的所有链接都是这样的:

links=[] 
for link in soup.find_all('a'): 
  a = link.get('href') 
  if type(a) == str and "https:" not in a: 
    links.append(a) 
links 

这是我得到的:

['/en_us/offers/index', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/overview', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/rooms-and-suites', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/offers', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/meetings', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/celebrations', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/dining', 
 '/en_us/hotels-in-bengaluru/the-leela-palace-hotel-bengaluru/Spa', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/overview', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/rooms-and-suites', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/offers', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/meetings', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/celebrations', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/dining', 
 '/en_us/hotels-in-chennai/the-leela-palace-hotel-chennai/spa', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/overview', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/rooms-and-suites', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/offers', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/meetings', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/celebrations', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/dining', 
 '/en_us/hotels-in-delhi/the-leela-palace-hotel-new-delhi/spa', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/overview', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/rooms-and-suites', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/offers', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/meetings', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/celebrations', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/dining', 
 '/en_us/hotels-in-delhi/the-leela-ambience-convention-hotel-delhi/spa', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/overview', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/rooms-and-suites', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/offers', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/meetings', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/celebrations', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/dining', 
 '/en_us/hotels-in-goa/the-leela-goa-hotel/spa', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/overview', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/rooms-and-suites', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/offers', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/meetings', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/celebrations', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/dining', 
 '/en_us/hotels-in-gurugram/the-leela-ambience-hotel-gurugram/spa', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/overview', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/rooms-and-suites', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/offers', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/meetings', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/celebrations', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/dining', 
 '/en_us/hotels-in-kovalam/the-leela-kovalam-hotel/spa', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/overview', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/rooms-and-suites', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/offers', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/meetings', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/celebrations', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/dining', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/overview', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/rooms-and-suites', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/offers', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/meetings', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/celebrations', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/dining', 
 '/en_us/hotels-in-udaipur/the-leela-palace-hotel-udaipur/spa', 
 'javascript:facebookLogin();', 
 'javascript:forgot_password(this);', 
 '/application/spring/myprofile/my-profile-edit', 
 '/en_us', 
 '/application/spring/myprofile/login', 
 '/the-leela/best-rates-guaranteed', 
 '#', 
 'javascript:facebookLogin();', 
 '/application/spring/myprofile/my-profile-edit', 
 '/en_us', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/signature-spa-treatments-', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/holistic-treatments-', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/fitness', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/wellness', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/salon', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/signature-spa-treatments-', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/holistic-treatments-', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/fitness', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/wellness', 
 '/en_us/hotels-in-mumbai/the-leela-mumbai-hotel/spa/salon', 
 '/contentAsset/raw-data/d1e3f704-be84-4353-a95e-28629651db00/fileAsset', 
 '/the-leela/about-the-leela/history', 
 '/the-leela/about-the-leela/company-information', 
 '/the-leela/about-the-leela/alliances', 
 '/the-leela/about-the-leela/investor-relations', 
 '/the-leela/about-the-leela/future-openings', 
 'javascript:void(0);', 
 '/the-leela/media/media-coverage', 
 '/the-leela/media/press-releases', 
 '/the-leela/media/media-contacts', 
 '/the-leela/media/the-leela-magazine', 
 '/the-leela/media/awards', 
 '/the-leela/Loyalty/the-leela-discovery', 
 '/the-leela/Loyalty/leela-solitaire-line', 
 '/the-leela/Loyalty/connoisseur-club', 
 '/the-leela/Loyalty/the-leela-preferred-partners-membership-program', 
 '/the-leela/careers/opportunities', 
 '/the-leela/contact-us/hotels', 
 '/the-leela/contact-us/convention-centre', 
 '/the-leela/contact-us/reservations', 
 '/the-leela/contact-us/sales-marketing-offices', 
 'javascript:void(0);', 
 '/the-leela/others/art', 
 '/the-leela/others/boutique', 
 '/the-leela/termsConditions/legal', 
 '/the-leela/termsConditions/siteMap', 
 '/the-leela/termsConditions/privacy-policy', 
 '/the-leela/termsConditions/general-terms-and-conditions'] 

正如你所看到的,这里仍然有一些不相关的链接

'javascript:void(0);', 
/application/spring/myprofile/login', 
 '/the-leela/best-rates-guaranteed', 
 '#', 
 'javascript:facebookLogin();', 
 '/application/spring/myprofile/my-profile-edit', 
 '/en_us', 

我需要帮助摆脱这些,以便我可以在输出列表上循环运行抓取工具。感谢任何帮助。

请您参考如下方法:

我怀疑是否存在非特定站点的现成解决方案。根据我使用爬虫的经验,我想到了一些事情:

  • 您可以使用网站的 sitemap该页面通常供志同道合的爬虫使用,并且包含指向网站所有者希望您爬取的所有重要页面的链接。 robots.txt也可能有用。
  • 您可以尝试下载所有页面并使用mimetypes lib 和/或什至利用 Content-Type header
  • 您可能想要放置一些启发式关键字或规则,例如正则表达式,以防止您的抓取工具到达或抓取某些网址。
  • 最后(如果这是一个涉及数百或数千个网站的耗时数月的庞大项目)您可以尝试使用机器学习进一步限制 URL。