1. Forum
  2. Usenet
  3. LINUX.DEBIAN.DEVEL.MENTOR

  • how to manage packages that require native acceleration code

    From Aaron Boxer@21:1/5 to All on Mon Sep 12 16:10:01 2022
    Hello,
    My codec project uses SIMD code for x86 and AArch64 architectures. Also, as there are different versions of SIMD i.e. SSE vs AVX vs AVX2, the project
    uses a library that builds multiple versions of the accelerated code and chooses which version to use at runtime.

    https://github.com/google/highway

    My package currently has a `march-native` error logged for certain architectures.

    https://qa.debian.org/bls/packages/l/libgrokj2k.html

    What would be the best approach for packaging such a project and avoiding
    this error ?

    Thanks very much,
    Aaron

    <div dir="ltr">Hello,<div>My codec project uses SIMD code for x86 and AArch64 architectures. Also, as there are different versions of SIMD i.e. SSE vs AVX vs AVX2, the project uses a library that builds multiple versions of the accelerated code and
    chooses which version to use at runtime.</div><div><br></div><div><a href="https://github.com/google/highway">https://github.com/google/highway</a><br></div><div><br></div><div>My package currently has a `march-native` error logged for certain
    architectures.</div><div><br></div><div><a href="https://qa.debian.org/bls/packages/l/libgrokj2k.html">https://qa.debian.org/bls/packages/l/libgrokj2k.html</a><br></div><div><br></div><div>What would be the best approach for packaging such a project and
    avoiding this error ?</div><div><br></div><div>Thanks very much,</div><div>Aaron</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Aaron Boxer on Mon Sep 12 16:40:01 2022
    On Mon, Sep 12, 2022 at 10:05:36AM -0400, Aaron Boxer wrote:
    My codec project uses SIMD code for x86 and AArch64 architectures. Also, as there are different versions of SIMD i.e. SSE vs AVX vs AVX2, the project uses a library that builds multiple versions of the accelerated code and chooses which version to use at runtime.
    This sounds good.

    My package currently has a `march-native` error logged for certain architectures.
    -march=native is not compatible with the approach you just described. And
    you indeed must not use it when building packages.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmMfRCQtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh ri8P/0RBBSypiZFDTbh6etqBUi2u4hRUp5a6qoyVr+T7n1NpuKPBjF1xccdZlbVl C45E7owwVzTUWn3ZvJXmShqu4/r0PISbeIl7XR9HDO5yAJqfhYxa0MvLIne8yrmo qcXJIs3sTcFUve8QHkUgllrskX/L9AMRwn1ZN7EwVlmH1t74kFeeadFoyC9M5ynQ JbS7UHxOV/1gXnQlVaS3hfXhdDZHruGbhcm5MCXuZBlf6PLg/VLV69RwfvunsnNa eDn1gQI3VzVSRUHsJtuFp+eIbykTDH3p2vIqUZQrorPc5B+ZAW/2TT2aLyXpVpqW CKEY6slmuVbSi47GqWQC7WA/ii+7wCwqT/PWzigz5QTNH112ZrgWgcx8q3sbbXKl W3Hzo8aIRNUq3MX+s1xkhT24epmGhvnqjIVFMmCiZRzpIqwXVbjfRvancaKzCAEw PDuVfnJqqeh0gEdmF+maQdn1Q9oVByyopblVEVVuKLEzsC+ojrqBBznxqqk0f/Lr u6fNRyOIC8En34+GqTcXrrGrFKpiyBSvyBAx4OuGMvHpGTb44X6ikbGdGkyerOiI lkob1oMI7Mc2c+txQdLIz8Nsnu2dOWXQ3lTghZgYf/3FTE4JckTWd8PB9O+JZ01X kDLxfs+YfC9X5Ufjk2LPM4+bjnz1gGhI8SHfRqYQPm3nULiO
    =zdj7
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Aaron Boxer@21:1/5 to [email protected] on Mon Sep 12 23:30:01 2022
    On Mon, Sep 12, 2022 at 10:37 AM Andrey Rahmatullin <[email protected]> wrote:

    On Mon, Sep 12, 2022 at 10:05:36AM -0400, Aaron Boxer wrote:
    My codec project uses SIMD code for x86 and AArch64 architectures. Also,
    as
    there are different versions of SIMD i.e. SSE vs AVX vs AVX2, the project uses a library that builds multiple versions of the accelerated code and chooses which version to use at runtime.
    This sounds good.

    My package currently has a `march-native` error logged for certain architectures.
    -march=native is not compatible with the approach you just described. And
    you indeed must not use it when building packages.



    Thanks, Andrey. I have fixed this problem. v10.0,0 has just been uploaded
    to unstable - would you
    recommend releasing a new version 10.0.1 with these fixes, or is there a
    way of updating v10.0.0 ?




    --
    WBR, wRAR


    <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 12, 2022 at 10:37 AM Andrey Rahmatullin &lt;<a href="mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquote class="
    gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Sep 12, 2022 at 10:05:36AM -0400, Aaron Boxer wrote:<br>
    &gt; My codec project uses SIMD code for x86 and AArch64 architectures. Also, as<br>
    &gt; there are different versions of SIMD i.e. SSE vs AVX vs AVX2, the project<br>
    &gt; uses a library that builds multiple versions of the accelerated code and<br>
    &gt; chooses which version to use at runtime.<br>
    This sounds good.<br>

    &gt; My package currently has a `march-native` error logged for certain<br> &gt; architectures.<br>
    -march=native is not compatible with the approach you just described. And<br> you indeed must not use it when building packages.<br></blockquote><div><br></div><div><br></div><div>Thanks, Andrey. I have fixed this problem. v10.0,0 has just been uploaded to unstable - would you</div><div>recommend releasing a new version 10.0.1
    with these fixes, or is there a way of updating v10.0.0 ?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">

    -- <br>
    WBR, wRAR<br>
    </blockquote></div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Wise@21:1/5 to Aaron Boxer on Tue Sep 13 05:50:01 2022
    On Mon, 2022-09-12 at 10:05 -0400, Aaron Boxer wrote:

    My codec project uses SIMD code for x86 and AArch64 architectures.
    Also, as there are different versions of SIMD i.e. SSE vs AVX vs
    AVX2, the project uses a library that builds multiple versions of the accelerated code and chooses which version to use at runtime.

    Runtime selection of instructions is the best solution indeed.

    Other solutions include automatic porting between SIMD instructions,
    emulating atomic instructions, manual runtime code path selection,
    manual runtime function selection, compiler function multi-versioning,
    glibc hwcaps library selection, runtime binary selection, blocking
    installation and blocking running binaries. All are summarised here:

    https://wiki.debian.org/InstructionSelection

    --
    bye,
    pabs

    https://wiki.debian.org/PaulWise

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEYQsotVz8/kXqG1Y7MRa6Xp/6aaMFAmMf/LYACgkQMRa6Xp/6 aaNxFQ//a6EnR8rpvbO4vyVkKAg4w/RKjGWvdliy7GuG7/1UR2GGF3JaO9XxXJ6Z j5FG6Tcr+T7oPLjz6g37EA6XYvTBgE3RZpzGrK1DqOGBBnv+lusV/xY8lDMw00D9 OZmsmpF0HDa2HTzHkQfpOMH4s0t1+3EUlbY+BDWewXoq2oCF6cA2gqKIEimfzodL bXMhTP64farb2CxlpXes7RNetoA8gZqdUs1J4GZaVAlff9pLv2uFgqeSXVTlhgtU C4CgAR1i1xzwrIZ9U/Xy90LXD+y4P9HaTAY+6uNQhsE7/B4ZQ9L4KiW2mgu75uEP +TaIfEXBlNgec38XTcS7NuYCQ4ke/gH+zXYerYBVGj2jcVnH9HArRpLW1hItDrF2 nSvFBJJPEsmTrMVk6uKKRpmw0+ocQ3wuw54Ru3TDzrZh7jQEOsHwMQ3a0ZqSficK LIsBQDfkxHgcZzuLhwrKxJ0Obw+3qeRCOzVVU8cafD9ZM5LFELxaj8aeXRxekllW EnA3PwOciB/dEXw9S3VpsatEFo1zi4l2CiUyIS6UD3mMQZaVDlsbTUIdK46Fzd0b 3oZuPtfTyZZWB7wC+4uHaTsv/p3izkXciIZ3Iw20riOSoCjT6VffpbqXU94zKp6M r5RXoHA+aZJVLjBlwmss7GU2y7ugbA3tT9mVeI/8l9qI8y+/QDM=
    =Q19+
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rahmatullin@21:1/5 to Aaron Boxer on Tue Sep 13 09:00:02 2022
    On Mon, Sep 12, 2022 at 05:25:27PM -0400, Aaron Boxer wrote:
    Thanks, Andrey. I have fixed this problem. v10.0,0 has just been uploaded
    to unstable - would you
    recommend releasing a new version 10.0.1 with these fixes, or is there a
    way of updating v10.0.0 ?
    You can upload 10.0.0-2.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmMgKeItFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh h8gP/0qwsfxq6VRDeyoWrSR/7dPqw6JMlBCAxuh+3E9vylbql+xlTgL74jWPeOuO dL4AsimSrUsHpOJTgpTRciYpOd8qDxoubKlYx5mJqkZfW7i828XrjpkMYAOTfv2K nEeeNt11AYr+Oc9G4Q/XlAhgcVGJ13Ft4Y3TJEKisXKmcmvWSoff32+krjnMUaNt nVx9g7zl0prvYeHGmfeKiSBBKgdfLxBpW+er25FubcxLAie6kDFItBcU4KRbcLXG yP9NXGekE2BsF46MM7/eVtHHyrM7+va7KXaUGcdKZ7ZPfGh2n4+VzBwl6C7NDHFi VDvaNgThBjO6uFLTiGlkEwIYJ7mjHnpFagomtcKbi+pjtE+tjXwYOcIvwVb9jhzv fyUFR0MRQS0CkBz4CkNDBQ8qMw5Sj/JdXB79OMEGnvdWgK/wgSckSiI9zH3jAPOy 4M4xbBBzJHuew9GRuLs7/Z+pvDrYjt6ksqHvnUkJr9q50VdnSEpe0ymatdU1/TT5 hGf62c2YTMaRkcNWVmi9btxB4VROE3zpxhj0O1XMZk+a8utckLZvbcfw5K/GV/Fa JTQ3vI6OuaBbXks367gkoGVU4AavEkw/DFXWcEKaQ0vaIkmehaCtzQMjogLH7i3o YkCFlq2QZ12hPuOaf7K7LXXw7d+SJZ8BPBMM3ZHzx4LkbeA3
    =M9R7
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Aaron Boxer@21:1/5 to [email protected] on Tue Sep 13 15:30:01 2022
    On Tue, Sep 13, 2022 at 2:58 AM Andrey Rahmatullin <[email protected]> wrote:

    On Mon, Sep 12, 2022 at 05:25:27PM -0400, Aaron Boxer wrote:
    Thanks, Andrey. I have fixed this problem. v10.0,0 has just been uploaded to unstable - would you
    recommend releasing a new version 10.0.1 with these fixes, or is there a way of updating v10.0.0 ?
    You can upload 10.0.0-2.


    Thanks. As code in master has changed, I decided to make a new point
    release, now uploaded to mentors

    https://mentors.debian.net/package/libgrokj2k/

    If this could be uploaded to unstable, it would be much appreciated!

    Aaron




    --
    WBR, wRAR


    <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 13, 2022 at 2:58 AM Andrey Rahmatullin &lt;<a href="mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquote class="
    gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Sep 12, 2022 at 05:25:27PM -0400, Aaron Boxer wrote:<br>
    &gt; Thanks, Andrey. I have fixed this problem. v10.0,0 has just been uploaded<br>
    &gt; to unstable - would you<br>
    &gt; recommend releasing a new version 10.0.1 with these fixes, or is there a<br>
    &gt; way of updating v10.0.0 ?<br>
    You can upload 10.0.0-2.<br></blockquote><div><br></div><div>Thanks. As code in master has changed, I decided to make a new point release, now uploaded to mentors</div><div><br></div><div><a href="https://mentors.debian.net/package/libgrokj2k/">https://
    mentors.debian.net/package/libgrokj2k/</a><br></div><div><br></div><div>If this could be uploaded to unstable, it would be much appreciated!</div><div><br></div><div>Aaron</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px
    0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">

    -- <br>
    WBR, wRAR<br>
    </blockquote></div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Aaron Boxer@21:1/5 to [email protected] on Tue Sep 13 15:30:01 2022
    On Mon, Sep 12, 2022 at 11:45 PM Paul Wise <[email protected]> wrote:

    On Mon, 2022-09-12 at 10:05 -0400, Aaron Boxer wrote:

    My codec project uses SIMD code for x86 and AArch64 architectures.
    Also, as there are different versions of SIMD i.e. SSE vs AVX vs
    AVX2, the project uses a library that builds multiple versions of the accelerated code and chooses which version to use at runtime.

    Runtime selection of instructions is the best solution indeed.

    Other solutions include automatic porting between SIMD instructions, emulating atomic instructions, manual runtime code path selection,
    manual runtime function selection, compiler function multi-versioning,
    glibc hwcaps library selection, runtime binary selection, blocking installation and blocking running binaries. All are summarised here:

    https://wiki.debian.org/InstructionSelection


    Very interesting, thanks!




    --
    bye,
    pabs

    https://wiki.debian.org/PaulWise


    <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 12, 2022 at 11:45 PM Paul Wise &lt;<a href="mailto:[email protected]">[email protected]</a>&gt; wrote:<br></div><blockquote class="gmail_quote"
    style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 2022-09-12 at 10:05 -0400, Aaron Boxer wrote:<br>

    &gt; My codec project uses SIMD code for x86 and AArch64 architectures.<br> &gt; Also, as there are different versions of SIMD i.e. SSE vs AVX vs<br>
    &gt; AVX2, the project uses a library that builds multiple versions of the<br> &gt; accelerated code and chooses which version to use at runtime.<br>

    Runtime selection of instructions is the best solution indeed.<br>

    Other solutions include automatic porting between SIMD instructions,<br> emulating atomic instructions, manual runtime code path selection,<br>
    manual runtime function selection, compiler function multi-versioning,<br> glibc hwcaps library selection, runtime binary selection, blocking<br> installation and blocking running binaries. All are summarised here:<br>

    <a href="https://wiki.debian.org/InstructionSelection" rel="noreferrer" target="_blank">https://wiki.debian.org/InstructionSelection</a></blockquote><div><br></div><div>Very interesting, thanks!</div><div> </div><blockquote class="gmail_quote" style="
    margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>

    -- <br>
    bye,<br>
    pabs<br>

    <a href="https://wiki.debian.org/PaulWise" rel="noreferrer" target="_blank">https://wiki.debian.org/PaulWise</a><br>
    </blockquote></div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)