Skip to main content
 首页 » 编程设计

python之pySpark Kafka Direct Streaming 更新 Zookeeper/Kafka Offset

2024年08月20日8yyy_WW

目前我正在使用 Kafka/Zookeeper 和 pySpark (1.6.0)。 我已经成功创建了一个使用 KafkaUtils.createDirectStream() 的 kafka 消费者。

所有流媒体都没有问题,但我认识到,在我消费了一些消息后,我的 Kafka 主题没有更新到当前偏移量。

由于我们需要更新主题以在此处进行监控,这有点奇怪。

在 Spark 的文档中我发现了这条评论:

   offsetRanges = [] 
 
     def storeOffsetRanges(rdd): 
         global offsetRanges 
         offsetRanges = rdd.offsetRanges() 
         return rdd 
 
     def printOffsetRanges(rdd): 
         for o in offsetRanges: 
             print "%s %s %s %s" % (o.topic, o.partition, o.fromOffset, o.untilOffset) 
 
     directKafkaStream\ 
         .transform(storeOffsetRanges)\ 
         .foreachRDD(printOffsetRanges) 

You can use this to update Zookeeper yourself if you want Zookeeper-based Kafka monitoring tools to show progress of the streaming application.

这是文档: http://spark.apache.org/docs/1.6.0/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers

我在 Scala 中找到了一个解决方案,但找不到 python 的等效解决方案。 这是 Scala 示例:http://geeks.aretotally.in/spark-streaming-kafka-direct-api-store-offsets-in-zk/

问题

但问题是,我如何才能从那时起更新 zookeeper?

请您参考如下方法:

我用 python 编写了一些函数来保存和读取 Kafka 偏移量 kazoo图书馆。

获取Kazoo客户端单例的第一个函数:

ZOOKEEPER_SERVERS = "127.0.0.1:2181" 
 
def get_zookeeper_instance(): 
    from kazoo.client import KazooClient 
 
    if 'KazooSingletonInstance' not in globals(): 
        globals()['KazooSingletonInstance'] = KazooClient(ZOOKEEPER_SERVERS) 
        globals()['KazooSingletonInstance'].start() 
    return globals()['KazooSingletonInstance'] 

然后是读写偏移量的函数:

def read_offsets(zk, topics): 
    from pyspark.streaming.kafka import TopicAndPartition 
 
    from_offsets = {} 
    for topic in topics: 
        for partition in zk.get_children(f'/consumers/{topic}'): 
            topic_partion = TopicAndPartition(topic, int(partition)) 
            offset = int(zk.get(f'/consumers/{topic}/{partition}')[0]) 
            from_offsets[topic_partion] = offset 
    return from_offsets 
 
def save_offsets(rdd): 
    zk = get_zookeeper_instance() 
    for offset in rdd.offsetRanges(): 
        path = f"/consumers/{offset.topic}/{offset.partition}" 
        zk.ensure_path(path) 
        zk.set(path, str(offset.untilOffset).encode()) 

然后在开始流式传输之前,您可以从 zookeeper 读取偏移量并将它们传递给 createDirectStream 对于 fromOffsets 参数。:

from pyspark import SparkContext 
from pyspark.streaming import StreamingContext 
from pyspark.streaming.kafka import KafkaUtils 
 
 
def main(brokers="127.0.0.1:9092", topics=['test1', 'test2']): 
    sc = SparkContext(appName="PythonStreamingSaveOffsets") 
    ssc = StreamingContext(sc, 2) 
 
    zk = get_zookeeper_instance() 
    from_offsets = read_offsets(zk, topics) 
 
    directKafkaStream = KafkaUtils.createDirectStream( 
        ssc, topics, {"metadata.broker.list": brokers}, 
        fromOffsets=from_offsets) 
 
    directKafkaStream.foreachRDD(save_offsets) 
 
 
if __name__ == "__main__": 
    main()