Skip to content

rushti & taskfiles encoded UTF-8 with BOM #140

@hermie64

Description

@hermie64

Hi,

After upgrading rushti, we tested with a TI process that generates the task file. That worked fine.
However, when testing another TI process, the task file was created, but its processes were not executed.

We then tested manually and encountered the following error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\Python311\Scripts\rushti.exe\__main__.py", line 5, in <module>
  File "D:\Python311\Lib\site-packages\rushti\cli.py", line 929, in main
    tm1_service_by_instance, preserve_connections = setup_tm1_services(
                                                    ^^^^^^^^^^^^^^^^^^^
  File "D:\Python311\Lib\site-packages\rushti\execution.py", line 157, in setup_tm1_services
    tm1_instances_in_tasks = get_instances_from_tasks_file(tasks_file_path)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python311\Lib\site-packages\rushti\parsing.py", line 193, in get_instances_from_tasks_file
    detected_mode = detect_execution_mode(tasks_file_path)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python311\Lib\site-packages\rushti\taskfile.py", line 496, in detect_execution_mode
    raise ValueError(
ValueError: Cannot determine execution mode from file content. Expected line starting with 'id=' (opt) or 'instance=' (norm), got: instance="XXXX_XXX" process="T_00_2421_Blabla...

Upon closer inspection, we found that the task file that fails is encoded in UTF-8 with BOM, whereas the one that runs successfully is encoded in UTF-8 without BOM.
After resaving the failing file as UTF-8 without BOM, the processes in the task file were executed correctly.

Both TI processes use:

  • SetOutputCharacterSet(s_TaskFile, 'TM1CS_UTF8')
  • TextOutput
  • Unique (timestamp-based) task file names

It appears that within TI, we cannot control whether UTF-8 is written with or without BOM? Have you seen this before?

As a workaround, we could preprocess the file using PowerShell or Python to remove the BOM. However, shouldn’t this be handled by the Rushti parser? In other words, should it not be able to correctly read task files regardless of whether they are encoded with or without BOM?

What are your thoughts?

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions