python之从 PySpark 的 RDD 中的数据中查找最小和最大日期

我将 Spark 与 Ipython 一起使用，并且有一个 RDD 打印时包含这种格式的数据:

打印 rdd1.collect()

[u'2010-12-08 00:00:00', u'2010-12-18 01:20:00', u'2012-05-13 00:00:00',....] 

每个数据都是一个datetimestamp，我想在这个RDD中找到最小值和最大值。我该怎么做？

请您参考如下方法:

例如，您可以使用 aggregate 函数(有关其工作原理的解释，请参见:What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?)

from datetime import datetime     
 
rdd  = sc.parallelize([ 
    u'2010-12-08 00:00:00', u'2010-12-18 01:20:00', u'2012-05-13 00:00:00']) 
 
def seq_op(acc, x): 
    """ Given a tuple (min-so-far, max-so-far) and a date string 
    return a tuple (min-including-current, max-including-current) 
    """ 
    d = datetime.strptime(x, '%Y-%m-%d %H:%M:%S') 
    return (min(d, acc[0]), max(d, acc[1])) 
 
def comb_op(acc1, acc2): 
    """ Given a pair of tuples (min-so-far, max-so-far) 
    return a tuple (min-of-mins, max-of-maxs) 
    """ 
    return (min(acc1[0], acc2[0]), max(acc1[1], acc2[1])) 
 
# (initial-min <- max-date, initial-max <- min-date) 
rdd.aggregate((datetime.max, datetime.min), seq_op, comb_op) 
 
## (datetime.datetime(2010, 12, 8, 0, 0), datetime.datetime(2012, 5, 13, 0, 0)) 

或DataFrame:

from pyspark.sql import Row 
from pyspark.sql.functions import from_unixtime, unix_timestamp, min, max 
 
row = Row("ts") 
df = rdd.map(row).toDF() 
 
df.withColumn("ts", unix_timestamp("ts")).agg( 
    from_unixtime(min("ts")).alias("min_ts"),  
    from_unixtime(max("ts")).alias("max_ts") 
).show() 
 
## +-------------------+-------------------+ 
## |             min_ts|             max_ts| 
## +-------------------+-------------------+ 
## |2010-12-08 00:00:00|2012-05-13 00:00:00| 
## +-------------------+-------------------+