Skip to main content
 首页 » 编程设计

python 和 twisted proxy,如何动态 gunzip

2024年11月01日8落叶无声

在使用 twistedmatrix ProxyClient 时,我如何压缩和处理响应部分?

我需要检查文本或 javascript 和 ajax 查询/答案。我应该使用 handleResponseEnd 吗?

我认为它在 handleResponsePart 内部,但看起来我误解了一点或其他什么,这是我的框架代码:

from twisted.python import log 
from twisted.web import http, proxy 
 
class ProxyClient(proxy.ProxyClient): 
    """Mange returned header, content here. 
 
    Use `self.father` methods to modify request directly. 
    """ 
    def handleHeader(self, key, value): 
            # change response header here 
            log.msg("Header: %s: %s" % (key, value)) 
            proxy.ProxyClient.handleHeader(self, key, value) 
 
    def handleResponsePart(self, buffer): 
    # this part below do not work,  
    # looks like @ this moment i do not have 'Content-Encoding' or 'Content-Type' 
    # what am i misunderstading? 
            cEncoding = self.father.getAllHeaders().get('Content-Encoding', '') 
            cType = self.father.getAllHeaders().get('Content-Type', '') 
            print >> sys.stderr, 'Content-Encoding', cEncoding 
            print >> sys.stderr, 'Content-Type', cType 
            if ('text' in cType.lower() or 'javascript' in cType.lower()) and 'gzip' in cEncoding.lower(): 
 
                buf = StringIO(buffer) 
                s = gzip.GzipFile(mode="rb", fileobj=buf) 
                content = s.read(len(buffer)) 
 
                # here process content as it should be gunziped 
 
    proxy.ProxyClient.handleResponsePart(self, buffer) 
 
class ProxyClientFactory(proxy.ProxyClientFactory): 
    protocol = ProxyClient 
 
class ProxyRequest(proxy.ProxyRequest): 
    protocols = dict(http=ProxyClientFactory) 
 
class Proxy(proxy.Proxy): 
    requestFactory = ProxyRequest 
 
class ProxyFactory(http.HTTPFactory): 
    protocol = Proxy 

从我的记录中我有:

2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Date: Tue, 11 Jun 2013 12:07:25 GMT 
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Server: Apache 
... 
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Content-Type: text/html;charset=ISO-8859-1 
... 
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Content-Encoding: gzip 
... 
2013-06-11 14:07:33+0200 [ProxyClient,client] Header: Connection: close 

所以我应该具备这两个条件吧!请问我错过了什么?

即使我对第二种方式不感兴趣,也就是删除对请求的接受,像这样,是否可以这样做: (顺便说一句,它看起来不起作用,或者经过测试的网络服务器不关心我们不想接收 gzip 压缩内容这一事实)

class ProxyRequest(proxy.ProxyRequest): 
    protocols = dict(http=ProxyClientFactory) 
 
    def process(self): 
        # removing the accept so that we do not tell "i'm ok with gzip encoded content" and should receive only not gzip-ed 
        self.requestHeaders.removeHeader('accept') 
        self.requestHeaders.removeHeader('accept-encoding') 

请您参考如下方法:

您必须在 handleResponsePart 中将数据 block 收集到 StringIO 缓冲区中,然后在 handleResponseEnd 中使用 GzipFile 进行解码。