Map Spark UDAF (Java)
I run Spark code on Java. I had data with the following schema -
[code language="bash"]
root
|-- userId: string (nullable = true</span>
|-- dt: string (nullable = true)</span>
|-- result: map (nullable = true)</span>
| |-- key: string
| |-- value: long (valueContainsNull = true)
[/code]
And I wanted to get a single record for a user which has the following schema -
[code language="bash"]
root
|-- userId: string (nullable = true)</span>
|-- result: map (nullable = true)
| |-- key: string
| |-- value: map (valueContainsNull = true)
| | |-- key: string
| | |-- value: long (valueContainsNull = true)
[/code]
Attached the user defined aggregation function I wrote to achieve it. Before that -
[code language="bash"]
MergeMapUDAF mergeMapUDAF = new MergeMapUDAF();
df.groupBy("userId").agg(mergeMapUDAF.apply(df.col("dt"), df.col("result")).as("result"));
[/code]
https://gist.github.com/tomron/36fd3c1b41169fc40acaeb4dbe95067d