scala 如何根据列值从 df 中删除行

debugcn 发表于 Dev

拉维库玛·雷迪·耶鲁瓦

我有这些值的数据框我需要过滤最小日期（groupby（id，count）和摘要应该更改为等于更多

id secid count date   summary
1   2     9    20170608  equal
1   3     9    20160608  equal
2   3     8    20170608  less
3   3     9    20160608  equal

我需要展示

id secid count date   summary
1   2     9    20170608  more
2   3     8    20170608  less
3   3     9    20160608  equal

文德夫

您可以使用groupByto group idand counttogether，然后使用whenandotherwise将您的汇总字段更改为more，以防您有更多date相同的idand count。

//create your original DF
val df = Seq((1, 2, 9, 20170608, "equal"),
      (1, 3, 9, 20160608, "equal"),
      (2, 3, 8, 20170608, "less"),
      (3, 3, 9, 20160608, "equal"),
      (1, 2, 8, 20170608, "random"),
      (1, 2, 8, 20170608, "random"))
      .toDF("id", "secid", "count", "date", "summary")

//Create a UDF to find the length of datelist after grouping
val isMoreThanOne = udf((lst: Seq[Int], summary: String) => lst.size > 1 && summary.equals("equal"))

//apply groupby and other operations to get the result
df.groupBy("id", "count")
  .agg(collect_list("date").as("datelist"),
    max("date").as("date"),
    first("secid").as("secid"),
    first("summary").as("summary"))
  .withColumn("summary",
    when(isMoreThanOne($"datelist", $"summary"), "more").otherwise($"summary"))
  .drop("datelist")
  .show()

//    output
//    +---+-----+--------+-----+-------+
//    | id|count|    date|secid|summary|
//    +---+-----+--------+-----+-------+
//    |  1|    9|20170608|    2|   more|
//    |  1|    8|20170608|    2| random|
//    |  3|    9|20160608|    3|  equal|
//    |  2|    8|20170608|    3|   less|
//    +---+-----+--------+-----+-------+

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。