Skip to content

Aarch64 support #569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
NicolasHug opened this issue Mar 17, 2025 · 8 comments
Open

Aarch64 support #569

NicolasHug opened this issue Mar 17, 2025 · 8 comments
Labels
enhancement New feature or request

Comments

@NicolasHug
Copy link
Member

Opening this issue to track requests for support of aarch64 (Linux ARM).

I think our build/test/release infra should be able to support those fairly naturally. We'd need to build and push more non-GPL FFmpeg binaries as well.
@imstevenpmwork
Copy link

We, at the lerobot OSS project from Hugging Face, recently decided to use torchcode as our default codec. However, some of our 'deployed' robots are running on aarch64, so the installation is currently broken for some users. To address this, we’ve implemented a platform check: if torchcode is available, we use it; otherwise, we gracefully fall back to another one.
Happy to test it once it is packaged for aarch64 🙂
Here are the logs:

(lerobot) raspberrypi@raspberrypi:~/lerobot $ uname -a
Linux raspberrypi 6.6.74+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64 GNU/Linux
(lerobot) raspberrypi@raspberrypi:~/lerobot $ pip install torchcodec==0.2.1
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
ERROR: Could not find a version that satisfies the requirement torchcodec==0.2.1 (from versions: 0.0.0.dev0)
ERROR: No matching distribution found for torchcodec==0.2.1

@hmellor hmellor mentioned this issue Mar 18, 2025
Closed
@scotts scotts added the enhancement New feature or request label Apr 23, 2025
@traversaro traversaro mentioned this issue Apr 27, 2025
Merged
5 tasks
@traversaro
Copy link
Contributor

While packaging torchcodec on conda-forge, it was trivial to also add the aarch64 build. However, it seems that both C++ and Python tests are failing on linux-aarch64 . I initially tought this may be a problem of conda-forge's pytorch package on linux-aarch64, but I was able to reproduce the same failures even linking PyTorch installed via PyPI.

Log of tests failures: https://github.com/traversaro/torchcodec-test-debug/actions/runs/14695881468/job/41237423866 .

Output C++ test failing on linux-aarch64 
 Internal ctest changing into directory: /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi
Test project /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi
      Start  1: SingleStreamDecoderTest.MissingVideoFileThrowsException
 1/23 Test  #1: SingleStreamDecoderTest.MissingVideoFileThrowsException ...................................   Passed    0.31 sec
      Start  2: SingleStreamDecoderTest.RespectsWidthAndHeightFromOptions
 2/23 Test  #2: SingleStreamDecoderTest.RespectsWidthAndHeightFromOptions .................................   Passed    0.32 sec
      Start  3: SingleStreamDecoderTest.RespectsOutputTensorDimensionOrderFromOptions
 3/23 Test  #3: SingleStreamDecoderTest.RespectsOutputTensorDimensionOrderFromOptions .....................   Passed    0.32 sec
      Start  4: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFpsAndDurationForVideoInMetadata/false
 4/23 Test  #4: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFpsAndDurationForVideoInMetadata/false ...   Passed    0.31 sec
      Start  5: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFpsAndDurationForVideoInMetadata/true
 5/23 Test  #5: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFpsAndDurationForVideoInMetadata/true ....   Passed    0.31 sec
      Start  6: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/false
 6/23 Test  #6: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/false ..............***Failed    0.32 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/0
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:190: Failure
Value of: torch::equal(tensor0FromOurDecoder, tensor0FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:191: Failure
Value of: torch::equal(tensor1FromOurDecoder, tensor1FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/0, where GetParam() = false (17 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (17 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (17 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/0, where GetParam() = false

 1 FAILED TEST

      Start  7: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/true
 7/23 Test  #7: FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/true ...............***Failed    0.33 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/1
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/1
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:190: Failure
Value of: torch::equal(tensor0FromOurDecoder, tensor0FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:191: Failure
Value of: torch::equal(tensor1FromOurDecoder, tensor1FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/1, where GetParam() = true (19 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (19 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (19 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/1, where GetParam() = true

 1 FAILED TEST

      Start  8: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/false
 8/23 Test  #8: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/false ...............***Failed    0.34 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/0
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:224: Failure
Value of: torch::equal(tensor[0], tensor0FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:225: Failure
Value of: torch::equal(tensor[1], tensorTime6FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/0, where GetParam() = false (30 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (30 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (30 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/0, where GetParam() = false

 1 FAILED TEST

      Start  9: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/true
 9/23 Test  #9: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/true ................***Failed    0.35 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/1
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/1
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:224: Failure
Value of: torch::equal(tensor[0], tensor0FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:225: Failure
Value of: torch::equal(tensor[1], tensorTime6FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/1, where GetParam() = true (31 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (31 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (31 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/1, where GetParam() = true

 1 FAILED TEST

      Start 10: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/false
10/23 Test #10: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/false ...............***Failed    0.34 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/0
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:249: Failure
Value of: torch::equal(tensor[0], tensor0FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:250: Failure
Value of: torch::equal(tensor[1], tensorTime6FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/0, where GetParam() = false (29 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (29 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (29 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/0, where GetParam() = false

 1 FAILED TEST

      Start 11: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/true
11/23 Test #11: FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/true ................***Failed    0.33 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/1
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/1
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:249: Failure
Value of: torch::equal(tensor[0], tensor0FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:250: Failure
Value of: torch::equal(tensor[1], tensorTime6FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/1, where GetParam() = true (29 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (29 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (29 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/1, where GetParam() = true

 1 FAILED TEST

      Start 12: FromFileAndMemory/SingleStreamDecoderTest.SeeksCloseToEof/false
12/23 Test #12: FromFileAndMemory/SingleStreamDecoderTest.SeeksCloseToEof/false ...........................   Passed    0.33 sec
      Start 13: FromFileAndMemory/SingleStreamDecoderTest.SeeksCloseToEof/true
13/23 Test #13: FromFileAndMemory/SingleStreamDecoderTest.SeeksCloseToEof/true ............................   Passed    0.33 sec
      Start 14: FromFileAndMemory/SingleStreamDecoderTest.GetsFramePlayedAtTimestamp/false
14/23 Test #14: FromFileAndMemory/SingleStreamDecoderTest.GetsFramePlayedAtTimestamp/false ................   Passed    0.37 sec
      Start 15: FromFileAndMemory/SingleStreamDecoderTest.GetsFramePlayedAtTimestamp/true
15/23 Test #15: FromFileAndMemory/SingleStreamDecoderTest.GetsFramePlayedAtTimestamp/true .................   Passed    0.38 sec
      Start 16: FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/false
16/23 Test #16: FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/false ...............***Failed    0.43 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/0
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:307: Failure
Value of: torch::equal(tensor6FromOurDecoder, tensor6FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:323: Failure
Value of: torch::equal(tensor61FromOurDecoder, tensor61FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:343: Failure
Value of: torch::equal(tensor10FromOurDecoder, tensor10FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:358: Failure
Value of: torch::equal(tensor6FromOurDecoder, tensor6FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:375: Failure
Value of: torch::equal(tensor7FromOurDecoder, tensor7FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/0, where GetParam() = false (118 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (118 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (118 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/0, where GetParam() = false

 1 FAILED TEST

      Start 17: FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/true
17/23 Test #17: FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/true ................***Failed    0.45 sec
Running main() from /home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/buildpypi/_deps/googletest-src/googletest/src/gtest_main.cc
Note: Google Test filter = FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/1
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest
[ RUN      ] FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/1
/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:307: Failure
Value of: torch::equal(tensor6FromOurDecoder, tensor6FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:323: Failure
Value of: torch::equal(tensor61FromOurDecoder, tensor61FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:343: Failure
Value of: torch::equal(tensor10FromOurDecoder, tensor10FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:358: Failure
Value of: torch::equal(tensor6FromOurDecoder, tensor6FromFFMPEG)
  Actual: false
Expected: true

/home/runner/work/torchcodec-test-debug/torchcodec-test-debug/.pixi_ws/torchcodec/test/VideoDecoderTest.cpp:375: Failure
Value of: torch::equal(tensor7FromOurDecoder, tensor7FromFFMPEG)
  Actual: false
Expected: true

[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/1, where GetParam() = true (137 ms)
[----------] 1 test from FromFileAndMemory/SingleStreamDecoderTest (137 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (137 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/1, where GetParam() = true

 1 FAILED TEST

      Start 18: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorFilterGraph/false
18/23 Test #18: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorFilterGraph/false .............   Passed    0.36 sec
      Start 19: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorFilterGraph/true
19/23 Test #19: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorFilterGraph/true ..............   Passed    0.32 sec
      Start 20: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorSwscale/false
20/23 Test #20: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorSwscale/false .................   Passed    0.32 sec
      Start 21: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorSwscale/true
21/23 Test #21: FromFileAndMemory/SingleStreamDecoderTest.PreAllocatedTensorSwscale/true ..................   Passed    0.31 sec
      Start 22: FromFileAndMemory/SingleStreamDecoderTest.GetAudioMetadata/false
22/23 Test #22: FromFileAndMemory/SingleStreamDecoderTest.GetAudioMetadata/false ..........................   Passed    0.30 sec
      Start 23: FromFileAndMemory/SingleStreamDecoderTest.GetAudioMetadata/true
23/23 Test #23: FromFileAndMemory/SingleStreamDecoderTest.GetAudioMetadata/true ...........................   Passed    0.30 sec

65% tests passed, 8 tests failed out of 23

Total Test time (real) =   8.07 sec

The following tests FAILED:
Errors while running CTest
	  6 - FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/false (Failed)
	  7 - FromFileAndMemory/SingleStreamDecoderTest.ReturnsFirstTwoFramesOfVideo/true (Failed)
	  8 - FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/false (Failed)
	  9 - FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNCHW/true (Failed)
	 10 - FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/false (Failed)
	 11 - FromFileAndMemory/SingleStreamDecoderTest.DecodesFramesInABatchInNHWC/true (Failed)
	 16 - FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/false (Failed)
	 17 - FromFileAndMemory/SingleStreamDecoderTest.SeeksToFrameWithSpecificPts/true (Failed)
Error: Process completed with exit code 8.

This was referenced Apr 27, 2025
Closed
Closed
@NicolasHug
Copy link
Member Author

NicolasHug commented Apr 28, 2025 •
edited
Loading

Thank you for the logs @traversaro . It looks like most of the errors look like

Mismatched elements: 170001 / 388800 (43.7%)
Greatest absolute difference: 3 at index (0, 27, 274)
Greatest relative difference: inf at index (0, 0, 104)
FAILED decoders/test_decoders.py::TestVideoDecoder::test_getitem_int[exact-cpu-4] - AssertionError: Tensor-likes are not equal!

which don't seem too bad. We do exact checks against linux x86, but we do have to relax the tolerances on MacOS and on CUDA already:

torchcodec/test/utils.py

Lines 38 to 51 in ed13ac5

def assert_frames_equal(*args, **kwargs):
if sys.platform == "linux":
if args[0].device.type == "cuda":
atol = 2
if get_ffmpeg_major_version() == 4:
assert_tensor_close_on_at_least(
args[0], args[1], percentage=95, atol=atol
)
else:
torch.testing.assert_close(*args, **kwargs, atol=atol, rtol=0)
else:
torch.testing.assert_close(*args, **kwargs, atol=0, rtol=0)
else:
torch.testing.assert_close(*args, **kwargs, atol=3, rtol=0)

So relaxing the tolerances on aarch64 seems reasonable. I'm not sure how to best do that robustly though, especially if those new architectures are supported out-of-core. This is something we may have to enable soon for XPU support as well: CC @scotts @dvrogozh for ideas?

@traversaro traversaro mentioned this issue Apr 28, 2025
Merged
5 tasks
@traversaro
Copy link
Contributor

Thanks, that is clear! So if I understood correctly the C++ tests are only expected to pass on Linux on x86_64?

@NicolasHug
Copy link
Member Author

NicolasHug commented Apr 28, 2025 •
edited
Loading

They are ideally meant to pass on all platforms, but if they don't, it's safe to disable them. Most of the C++ tests are legacy at this point anyway. As long as the Python tests pass (possibly slightly increasing the tolerance), an architecture can be considered safe to release.

@scotts
Copy link
Contributor

scotts commented Apr 28, 2025

@traversaro, thanks for digging into this! Did you need to make any changes outside of testing?

@traversaro
Copy link
Contributor

@traversaro, thanks for digging into this! Did you need to make any changes outside of testing?

No, with just the test-related patch suggested by @NicolasHug (https://github.com/conda-forge/torchcodec-feedstock/blob/fe701d8f1f535381f4779310bc158584b8420c7c/recipe/use_strict_threshold_only_for_linux_x86_64.patch) all the Python test run fine on CI on linux-aarch64: conda-forge/torchcodec-feedstock#12 .

@dvrogozh
Copy link
Contributor

So relaxing the tolerances on aarch64 seems reasonable. I'm not sure how to best do that robustly though<...>: CC @scotts @dvrogozh for ideas?

Just relaxing the tolerance won't work actually. Problem is with color conversion algorithms which might significantly differ per implementation and per-platform. You can't expect bit-to-bit matches. And that's not something you can overcome with the current threshold which expects difference in no more than N pixels. There could be bigger substantial differences. I did step into that working on XPU support for torchcodec.

For the media domain that's known problem. It's being solved by comparing frames by metrics such as PSNR, SSIM, etc. That was the part of my initial #558, see change in test/utils.py:

        elif args[0].device.type == "xpu":
            if not torch.allclose(*args, atol=0, rtol=0):
                from torcheval.metrics import PeakSignalNoiseRatio

                metric = PeakSignalNoiseRatio()
                metric.update(args[0], args[1])
                assert metric.compute() >= 40

As you can see I did that for XPU, but we can reuse the idea and code for other cases. I will extract this to separate PR today, but I will need help to test this on aarch64 since I don't have this platform at hand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@scotts @NicolasHug @traversaro @dvrogozh @imstevenpmwork