python之如何从通过 javascript 注入(inject)的视频标签中抓取视频 src url

好吧，我正在尝试抓取一个网站 http://www.popsci.com/thorium-dream用于学习目的。

我尝试抓取它来获取视频 src 但无法这样做，因为视频标签是由 javascript 注入(inject)的。

查看网络看到xhr请求看到视频的媒体文件请求。

General 
Remote Address:68.232.45.253:80 
Request URL:http://video.net2.tv/PORTICO/TECH/POPSCI/POP_84/POP_20140718_84_Thorium_A/POP_20140718_84_Thorium_A_1200.mp4 
Request Method:GET 
Status Code:206 Partial Content (from cache) 
Response Headers 
Accept-Ranges:bytes 
Cache-Control:max-age=604800 
Content-Length:24833827 
Content-Range:bytes 0-24833826/24833827 
Content-Type:video/mp4 
Date:Mon, 14 Sep 2015 02:54:29 GMT 
Etag:"734657553" 
Expires:Mon, 21 Sep 2015 02:54:29 GMT 
Last-Modified:Fri, 18 Jul 2014 21:56:46 GMT 
Server:ECAcc (cpm/F8B9) 
X-Cache:HIT 
Request Headers 
Provisional headers are shown 
Accept-Encoding:identity;q=1, *;q=0 
Range:bytes=0- 
Referer:http://player.net2.tv/?episode=53c9973ae7dbcc820502c81c&restart=true&snipe=true 
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36 

如何从抓取中获取 URL？ 如果可能，还请告诉使用默认 python 库的解决方案。

请您参考如下方法:

我已经为你编写了一些代码。它从 POPSCI 剧集页面中提取所有视频:

import re 
import requests 
from lxml import html 
 
def getVideosLinks(content): 
    videos = re.findall('(http://[\.\w/_]+\.mp[34])', content) 
    return videos 
 
def prepareJSONurl(episode_hash): 
    json_url = "http://pepto.portico.net2.tv/playlist/{hash}".format(hash=episode_hash) 
    return json_url 
 
def extractEpisodeHash(content): 
    tree = html.fromstring(content) 
    video_url = tree.xpath('//meta[contains(@http-equiv, "refresh")]/@content')[0].split('=',1)[1] 
    episode_hash = re.findall('episode=([\w]+)', video_url) 
    return episode_hash[0] 
 
def extractIframeURL(content): 
    iframe_url = None 
    tree = html.fromstring(content) 
    try: 
        iframe_url = tree.xpath('//iframe/@src')[0] 
        is_video = True 
    except: 
        is_video = False 
    return is_video, iframe_url 
 
 
POPSCI_URL = "http://www.popsci.com/thorium-dream" 
 
response = requests.get(POPSCI_URL) 
is_video, iframe_url = extractIframeURL(response.content) 
 
if is_video: 
    response_from_iframe_url = requests.get(iframe_url) 
    episode_hash = extractEpisodeHash(response_from_iframe_url.content) 
 
    json_url = prepareJSONurl(episode_hash) 
    final_response = requests.get(json_url) 
 
    for video in getVideosLinks(final_response.content): 
        print "Video: {}".format(video) 
else: 
    print "This is not a POPSCI video page :|" 

它们具有不同的视频质量和大小，因此您会看到每一集都有多个 .mp4 视频 URL。

此代码适用于任何 POPSCI 剧集页面，请尝试将 POPSCI_URL 更改为...

POPSCI_URL = "http://www.popsci.com/maker-faire-2015" 

...它仍然有效。

添加:

即便如此，也不建议使用正则表达式 (regexp) 解析 HTML 我已经为您创建了一个正则表达式版本(根据要求)。它有效，但可以改进正则表达式:

import re 
import requests 
 
def getVideosLinks(content): 
    videos = re.findall('(http://[\.\w/_]+\.mp[34])', content) 
    return videos 
 
def prepareJSONurl(episode_hash): 
    json_url = "http://pepto.portico.net2.tv/playlist/{hash}".format(hash=episode_hash) 
    return json_url 
 
def extractEpisodeHash(content): 
    episode_hash = re.findall('<meta http-equiv="refresh" content="0; url=http:\/\/player\.net2\.tv\?episode=([\w]+)&restart', content)[0] 
    return episode_hash 
 
def extractIframeURL(content): 
    iframe_url = None 
    try: 
        iframe_url = re.findall('<iframe src="(.*)" style', content)[0] 
        is_video = True 
    except: 
        is_video = False 
    return is_video, iframe_url 
 
 
POPSCI_URL = "http://www.popsci.com/thorium-dream" 
 
response = requests.get(POPSCI_URL) 
is_video, iframe_url = extractIframeURL(response.content) 
 
if is_video: 
    response_from_iframe_url = requests.get(iframe_url) 
    episode_hash = extractEpisodeHash(response_from_iframe_url.content) 
 
    json_url = prepareJSONurl(episode_hash) 
    final_response = requests.get(json_url) 
 
    for video in getVideosLinks(final_response.content): 
        print "Video: {}".format(video) 
else: 
    print "This is not a POPSCI video page :|" 

希望对你有帮助