Map Spark UDAF (Java)

·1 min read

I run Spark code on Java. I had data with the following schema - ```text root |-- userId: string (nullable = true) |-- dt: string (nullable = true) |-- result: map (nullable = true) |    |-- key: string |    |-- value: long (valueContainsNull = true) ``` And I wanted to get a single record for a user which has the following schema - ```text root |-- userId: string (nullable = true) |-- result: map (nullable = true) |    |-- key: string |    |-- value: map (valueContainsNull = true) |    |    |-- key: string |    |    |-- value: long (valueContainsNull = true) ``` Attached the user defined aggregation function I wrote to achieve it. Before that - ```java MergeMapUDAF mergeMapUDAF = new MergeMapUDAF(); df.groupBy("userId").agg(mergeMapUDAF.apply(df.col("dt"), df.col("result")).as("result")); ```