Skip to content

[BUG]: UDF returning json as string changing the decimal format inside json #984

@guruvonline

Description

@guruvonline

Describe the bug
UDF returning json as string, but when reading it in main function, decimal format is getting change

sample json returned by UDF

{
  "Organization":"Org",
  "Employee": 
         {
            "Id":  2345678945.1234512    // this is decimal in json but while reading from UDF, decimal part is not coming correctly
             "name" : "somename"
         }
}
//UDF with signature
string myFunc(string)
{
   return json.serialize(myObject);
}
//myObject is complex object and  looks like
Class myObject
{
   string Organization;
   Employee Emp;
}

class Employee
{
  decimal Id;   //if i change it to string, i get full value
  string name;
}

In my main function i am calling the UDF as

df = df.Select(myFunc(col('col1')).As(jsonResult));

// json schema
schema = {StructField(Organization, StringType),
                  StructFiled(Employee, StringType)}    // reading json as String type

//explode json
df = df.Select(FromJson(Col("jsonResult"), schema).As("Result")) 

df.Select("Result.*").

df.Select("Employee")  // reading json attribute as string, but decimal is getting truncated.

If i change my decimal to string value in json, then i am getting the full decimal value. Which means UDF is returning the correct value, but while reading in main function it is truncating decimal (even though i am reading as string).

Thanks

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions