Skip to main content
 首页 » 编程设计

python之Scrapy Splash 服务器未通过 docker run 启动

2024年10月24日27arxive

按照 alexce 的说明 here 安装了 docker、scrapyjs 和 splash。

然后,运行 docker run -p 8050:8050 scrapinghub/splash,我得到这个输出:

2016-05-08 17:17:45+0000 [-] Log opened. 
2016-05-08 17:17:45.978866 [-] Splash version: 2.1 
2016-05-08 17:17:45.979553 [-] Qt 5.5.1, PyQt 5.5.1, WebKit 538.1, sip 4.17, Twisted 16.1.1, Lua 5.2 
2016-05-08 17:17:45.980138 [-] Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4] 
2016-05-08 17:17:45.980401 [-] Open files limit: 1048576 
2016-05-08 17:17:45.981020 [-] Can't bump open files limit 
2016-05-08 17:17:46.086232 [-] Xvfb is started: ['Xvfb', ':1', '-screen', '0', '1024x768x24'] 
2016-05-08 17:17:46.161902 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles 
2016-05-08 17:17:46.260357 [-] verbosity=1 
2016-05-08 17:17:46.260607 [-] slots=50 
2016-05-08 17:17:46.261170 [-] argument_cache_max_entries=500 
2016-05-08 17:17:46.262476 [-] Web UI: enabled, Lua: enabled (sandbox: enabled) 
2016-05-08 17:17:46.264565 [-] Site starting on 8050 
2016-05-08 17:17:46.265203 [-] Starting factory <twisted.web.server.Site object at 0x7f270ec81e10> 

它卡在那里。我尝试根据 here 说明(重新安装 docker、验证 vm 正在运行、重新生成证书、设置 env)进行故障排除,但仍然一无所获。

My settings file 和 Spider

当在 yield scrapy.Request' 调用中没有 meta= 参数的情况下运行爬虫start_requests` 时,爬虫运行良好(除了动态字段不是已捕获),所以我不确定这是 docker 还是 splash 的问题。

提前致谢。

请您参考如下方法:

当使用 Docker 运行 Splash 时,控制台一开始确实只是“卡在那里”。 Splash 正在等待通过其端点之一的请求。

$ sudo docker run -p 8050:8050 scrapinghub/splash 
2016-05-09 10:21:42+0000 [-] Log opened. 
2016-05-09 10:21:42.773541 [-] Splash version: 2.1 
2016-05-09 10:21:42.774298 [-] Qt 5.5.1, PyQt 5.5.1, WebKit 538.1, sip 4.17, Twisted 16.1.1, Lua 5.2 
2016-05-09 10:21:42.774453 [-] Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4] 
2016-05-09 10:21:42.774632 [-] Open files limit: 1048576 
2016-05-09 10:21:42.774842 [-] Can't bump open files limit 
2016-05-09 10:21:42.879868 [-] Xvfb is started: ['Xvfb', ':1', '-screen', '0', '1024x768x24'] 
2016-05-09 10:21:43.072351 [-] proxy profiles support is enabled, proxy profiles path: /etc/splash/proxy-profiles 
2016-05-09 10:21:43.214478 [-] verbosity=1 
2016-05-09 10:21:43.214617 [-] slots=50 
2016-05-09 10:21:43.214703 [-] argument_cache_max_entries=500 
2016-05-09 10:21:43.215195 [-] Web UI: enabled, Lua: enabled (sandbox: enabled) 
2016-05-09 10:21:43.217494 [-] Site starting on 8050 
2016-05-09 10:21:43.217635 [-] Starting factory <twisted.web.server.Site object at 0x7f529d0fee48> 

要测试 Splash 是否正常运行,请在 http://localhost:8050/ 尝试其 Web UI

您应该会看到如下内容:

然后您可以尝试输入一些 URL 并单击“渲染我!”