tests fail on s390x (big-endian) since 0.2.0.2

[rathann]

While building 0.2.0.2 and 0.2.0.3 on Fedora, I noticed that the tests have started failing on s390x arch:

$ cd openapv-0.2.0.3
$ /usr/bin/ctest --test-dir redhat-linux-build --output-on-failure --force-new-ctest-process -j2
Internal ctest changing into directory: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build
Test project /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build
      Start  3: encode
 1/16 Test  #3: encode ...........................   Passed    0.58 sec
      Start  1: Encoder_runs
      Start  2: Decoder_runs
 2/16 Test  #1: Encoder_runs .....................   Passed    0.00 sec
      Start  5: decode_qp_A
 3/16 Test  #2: Decoder_runs .....................   Passed    0.00 sec
      Start  6: decode_qp_B
 4/16 Test  #6: decode_qp_B ......................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  0.99 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/qp_B.apv -v 3 --hash 
================================================================================
AU 0         2885421-bytes    1-frame(s)        287 msec
- FRM 0  GID 1     PRIMARY       2885339-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         2889559-bytes    1-frame(s)        287 msec
- FRM 0  GID 1     PRIMARY       2889477-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         2899159-bytes    1-frame(s)        289 msec
- FRM 0  GID 1     PRIMARY       2899077-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 863 msec, 0.863 sec
Average decoding time for a frame = 287 msec
Average decoding speed            = 3.476 frames/sec
================================================================================
      Start  7: decode_qp_C
 5/16 Test  #5: decode_qp_A ......................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  0.99 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/qp_A.apv -v 3 --hash 
================================================================================
AU 0         3197997-bytes    1-frame(s)        286 msec
- FRM 0  GID 1     PRIMARY       3197915-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         3197997-bytes    1-frame(s)        286 msec
- FRM 0  GID 1     PRIMARY       3197915-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         3198648-bytes    1-frame(s)        286 msec
- FRM 0  GID 1     PRIMARY       3198566-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 857 msec, 0.857 sec
Average decoding time for a frame = 285 msec
Average decoding speed            = 3.501 frames/sec
================================================================================
      Start  8: decode_qp_D
 6/16 Test  #8: decode_qp_D ......................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  0.78 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/qp_D.apv -v 3 --hash 
================================================================================
AU 0          614268-bytes    1-frame(s)        193 msec
- FRM 0  GID 1     PRIMARY        614186-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1          614268-bytes    1-frame(s)        267 msec
- FRM 0  GID 1     PRIMARY        614186-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2          614220-bytes    1-frame(s)        281 msec
- FRM 0  GID 1     PRIMARY        614138-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 740 msec, 0.740 sec
Average decoding time for a frame = 246 msec
Average decoding speed            = 4.054 frames/sec
================================================================================
      Start  9: decode_qp_E
 7/16 Test  #9: decode_qp_E ......................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  1.13 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/qp_E.apv -v 3 --hash 
================================================================================
AU 0         3119144-bytes    1-frame(s)        328 msec
- FRM 0  GID 1     PRIMARY       3119062-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         3119144-bytes    1-frame(s)        338 msec
- FRM 0  GID 1     PRIMARY       3119062-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         3119769-bytes    1-frame(s)        334 msec
- FRM 0  GID 1     PRIMARY       3119687-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 1001 msec, 1.001 sec
Average decoding time for a frame = 333 msec
Average decoding speed            = 2.997 frames/sec
================================================================================
      Start 10: decode_syn_A
 8/16 Test #10: decode_syn_A .....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  0.29 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/syn_A.apv -v 3 --hash 
================================================================================
AU 0         3206458-bytes    1-frame(s)         92 msec
- FRM 0  GID 1     PRIMARY       3206376-bytes  1920x1080 4:2:2-10  hash:mismatch
================================================================================
AU 1          601443-bytes    1-frame(s)         67 msec
- FRM 0  GID 1     PRIMARY        601361-bytes  1920x1080 4:2:2-10  hash:mismatch
================================================================================
AU 2          601443-bytes    1-frame(s)         66 msec
- FRM 0  GID 1     PRIMARY        601361-bytes  1920x1080 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 225 msec, 0.225 sec
Average decoding time for a frame = 75 msec
Average decoding speed            = 13.333 frames/sec
================================================================================
      Start 11: decode_syn_B
 9/16 Test  #7: decode_qp_C ......................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  2.26 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/qp_C.apv -v 3 --hash 
================================================================================
AU 0        14493687-bytes    1-frame(s)        533 msec
- FRM 0  GID 1     PRIMARY      14493605-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1        14493687-bytes    1-frame(s)        551 msec
- FRM 0  GID 1     PRIMARY      14493605-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2        14492775-bytes    1-frame(s)        522 msec
- FRM 0  GID 1     PRIMARY      14492693-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 1607 msec, 1.607 sec
Average decoding time for a frame = 535 msec
Average decoding speed            = 1.867 frames/sec
================================================================================
      Start 12: decode_tile_A
10/16 Test #11: decode_syn_B .....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  0.32 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/syn_B.apv -v 3 --hash 
================================================================================
AU 0         2043287-bytes    1-frame(s)        102 msec
- FRM 0  GID 1     PRIMARY       2043205-bytes  1920x1080 4:2:2-10  hash:mismatch
================================================================================
AU 1         2043287-bytes    1-frame(s)         71 msec
- FRM 0  GID 1     PRIMARY       2043205-bytes  1920x1080 4:2:2-10  hash:mismatch
================================================================================
AU 2         2043287-bytes    1-frame(s)         71 msec
- FRM 0  GID 1     PRIMARY       2043205-bytes  1920x1080 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 244 msec, 0.244 sec
Average decoding time for a frame = 81 msec
Average decoding speed            = 12.295 frames/sec
================================================================================
      Start 13: decode_tile_B
11/16 Test #13: decode_tile_B ....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  0.92 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/tile_B.apv -v 3 --hash 
================================================================================
AU 0         3119144-bytes    1-frame(s)        257 msec
- FRM 0  GID 1     PRIMARY       3119062-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         3119144-bytes    1-frame(s)        257 msec
- FRM 0  GID 1     PRIMARY       3119062-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         3119769-bytes    1-frame(s)        275 msec
- FRM 0  GID 1     PRIMARY       3119687-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 789 msec, 0.789 sec
Average decoding time for a frame = 263 msec
Average decoding speed            = 3.802 frames/sec
================================================================================
      Start 14: decode_tile_C
12/16 Test #12: decode_tile_A ....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  1.29 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/tile_A.apv -v 3 --hash 
================================================================================
AU 0         3109535-bytes    1-frame(s)        409 msec
- FRM 0  GID 1     PRIMARY       3109453-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         3109535-bytes    1-frame(s)        371 msec
- FRM 0  GID 1     PRIMARY       3109453-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         3110194-bytes    1-frame(s)        336 msec
- FRM 0  GID 1     PRIMARY       3110112-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 1116 msec, 1.116 sec
Average decoding time for a frame = 372 msec
Average decoding speed            = 2.688 frames/sec
================================================================================
      Start 15: decode_tile_D
13/16 Test #15: decode_tile_D ....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  1.06 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/tile_D.apv -v 3 --hash 
================================================================================
AU 0         3151784-bytes    1-frame(s)        297 msec
- FRM 0  GID 1     PRIMARY       3151702-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         3151784-bytes    1-frame(s)        339 msec
- FRM 0  GID 1     PRIMARY       3151702-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         3152409-bytes    1-frame(s)        278 msec
- FRM 0  GID 1     PRIMARY       3152327-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 914 msec, 0.914 sec
Average decoding time for a frame = 304 msec
Average decoding speed            = 3.282 frames/sec
================================================================================
      Start 16: decode_tile_E
14/16 Test #16: decode_tile_E ....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  1.03 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/tile_E.apv -v 3 --hash 
================================================================================
AU 0         3120164-bytes    1-frame(s)        310 msec
- FRM 0  GID 1     PRIMARY       3120082-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 1         3120164-bytes    1-frame(s)        295 msec
- FRM 0  GID 1     PRIMARY       3120082-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
AU 2         3120789-bytes    1-frame(s)        289 msec
- FRM 0  GID 1     PRIMARY       3120707-bytes  3840x2160 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 894 msec, 0.894 sec
Average decoding time for a frame = 298 msec
Average decoding speed            = 3.356 frames/sec
================================================================================
15/16 Test #14: decode_tile_C ....................***Failed  Required regular expression not found. Regex=[.*hash:match.*
]  3.03 sec
  ____                ___   ___ _   __
 / __ \___  ___ ___  / _ | / _ \ | / / Decoder (v0.2.0.3)
/ /_/ / _ \/ -_) _ \/ __ |/ ___/ |/ / 
\____/ .__/\__/_//_/_/ |_/_/   |___/  
    /_/                               
Command line: /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/redhat-linux-build/bin/oapv_app_dec -i /builddir/build/BUILD/openapv-0.2.0.3-build/openapv-0.2.0.3/test/bitstream/tile_C.apv -v 3 --hash 
================================================================================
AU 0         6278681-bytes    1-frame(s)        998 msec
- FRM 0  GID 1     PRIMARY       6278599-bytes  7680x4320 4:2:2-10  hash:mismatch
================================================================================
AU 1         6278681-bytes    1-frame(s)        994 msec
- FRM 0  GID 1     PRIMARY       6278599-bytes  7680x4320 4:2:2-10  hash:mismatch
================================================================================
AU 2         6278181-bytes    1-frame(s)        736 msec
- FRM 0  GID 1     PRIMARY       6278099-bytes  7680x4320 4:2:2-10  hash:mismatch
================================================================================
End of file
--> end of bitstream
=== Summary ====================================================================
Processed access units            = 3
Decoded frame count               = 3
Total decoding time               = 2727 msec, 2.727 sec
Average decoding time for a frame = 909 msec
Average decoding speed            = 1.100 frames/sec
================================================================================
      Start  4: decode
16/16 Test  #4: decode ...........................   Passed    0.25 sec
25% tests passed, 12 tests failed out of 16
Total Test time (real) =   8.30 sec
The following tests FAILED:
	  5 - decode_qp_A (Failed)
	  6 - decode_qp_B (Failed)
	  7 - decode_qp_C (Failed)
	  8 - decode_qp_D (Failed)
	  9 - decode_qp_E (Failed)
	 10 - decode_syn_A (Failed)
	 11 - decode_syn_B (Failed)
	 12 - decode_tile_A (Failed)
	 13 - decode_tile_B (Failed)
	 14 - decode_tile_C (Failed)
	 15 - decode_tile_D (Failed)
	 16 - decode_tile_E (Failed)
Errors while running CTest
$ 

[dariusz-f]

It looks that hash option in oapv isn't properly working on s390x architecture.
Anyway, all of those testes are passing on x86_64 arch.

[kpchoi]

This issue seems to require PR from folks using s390x machine.
Do we have any simple way to make this enrionment?

[rathann]

@sharkcz could you assist here?

[sharkcz]

Yes, I can provide access to our public s390x development systems. Please send me an email at <sharkcz [at] fedoraproject.org>. Another option is requesting a system via https://openmainframeproject.org/news/developer-resources-for-linux-on-s390x/, but my route should be faster/easier.

[Anemptyship]

I've submitted a PR to fix this issue: #177

Root Cause: The MD5 implementation was not endian-safe. It cast u8* to u32*, causing incorrect hash computation on big-endian systems.

Fix: Made md5_trans() explicitly decode bytes to little-endian u32 words, with a compile-time optimization to preserve zero-copy performance on little-endian systems.

[rathann]

Fix: Made md5_trans() explicitly decode bytes to little-endian u32 words, with a compile-time optimization to preserve zero-copy performance on little-endian systems.

The PR fixes the issue for me, thanks!