Skip to main content
 首页 » 编程设计

Python:异步 Cassandra 插入

2025年05月04日9小虾米

Cassandra Python 驱动程序的问题是返回的“ future ”对象通过副作用 添加回调。这意味着“ future ”不是可组合的,就像来自 Javascript 或 Scala 的 Future 是可组合的一样。我想知道是否有一种模式可用于将不可组合的 future 转变为可组合的 future (最好没有内存泄漏。)

   my_query_object.insert(1, 2, 3, 'Fred Flinstone') 
     .insert(1, 2, 3, 'Barney Rubble') 
     .insert(5000, 2, 3, 'George Jetson') 
     .insert(5000, 2, 3, 'Jane his wife') 

查看 performance在 Datastax 的 Cassandra Python 驱动程序部分,我看到了一个示例,说明他们如何创建一系列可连续链接的插入查询。即此模式的稍微复杂的版本:

def insert_next(previous_result=sentinel): 
    if previous_result is not sentinel: 
        if isinstance(previous_result, BaseException): 
            log.error("Error on insert: %r", previous_result) 
 
    future = session.execute_async(query) 
    # NOTE: this callback also handles errors 
    future.add_callbacks(insert_next, insert_next) 

作为玩具示例效果很好。一个查询完成后,另一个等效查询将再次执行。该方案允许他们实现 7k 写入/秒,而不尝试“链接”回调的版本上限约为 2k 写入/秒。

我一直在努力创造某种机制,让我重新获得那个确切的机制,但无济于事。有人想出类似的东西吗?

请您参考如下方法:

我花了一些时间思考如何以某种形式保护 future :

import logging 
from Queue import Queue #queue in python 3 
from threading import Event #hmm... this needed? 
 
 
insert_logger = logging.getLogger('async_insert') 
insert_logger.setLevel(logging.INFO) 
 
def handle_err(err): 
  insert_logger.warning('Failed to insert due to %s', err) 
 
 
#Designed to work in a high write environment. Chained callbacks for best performance and fast fail/stop when error 
#encountered. Next insert should re-up the writing. Potential loss of failed write. Some guarantee on order of write 
#preservation. 
class CappedQueueInserter(object): 
  def __init__(self, session, max_count=0): 
    self.__queue = Queue(max_count) 
    self.__session = session 
    self.__started = Event() 
 
  @property 
  def started(self): 
    return self.__started.is_set() 
 
  def insert(self, bound_statement): 
    if not self.started: 
      self._begin(bound_statement) 
    else: 
      self._enqueue(bound_statement) 
 
  def _begin(self, bound_statement): 
    def callback(): 
      try: 
        bound = self.__queue.get(True) #block until an item is added to the queue 
        future = self.__session.execute_async(bound) 
        future.add_callbacks(callback, handle_err) 
      except: 
        self.__started.clear() 
 
    self.__started.set() 
    future = self.__session.execute_async(bound_statement) 
    future.add_callbacks(callback, handle_err) 
 
  def _enqueue(self, bound_statement): 
    self.__queue.put(bound_statement, True) 
 
 
#Separate insert statement binding from the insertion loop 
class InsertEnqueue(object): 
  def __init__(self, prepared_query, insert, consistency_level=None): 
    self.__statement = prepared_query 
    self.__level = consistency_level 
    self.__sink = insert 
 
  def insert(self, *args): 
    bound = self.bind(*args) 
    self.__sink.insert(bound) 
 
  @property 
  def consistency_level(self): 
    return self.__level or self.__statement.consistency_level 
 
  @consistency_level.setter 
  def adjust_level(self, value): 
    if value: 
      self.__level = value 
 
  def bind(self, *args): 
    bound = self.__statement.bind(*args) 
    bound.consistency_level = self.consistency_level 
 
    return bound 

Queue 和Event 的组合来触发事件。假设写入可以“最终”发生,这应该可行。