Skip to content

confused with the latest changes - one branch? #46

Closed

Description

@perexg

It seems that this repo moved again from the branch system (one branch = one version) to all-in-one branch. It's a bit unclear to me, how I can fetch one version for packaging. And it's also unclear, how you'll tag the releases.

Activity

  1. lgirdwood commented on Jun 3, 2021

    @lgirdwood
    Member

    @perexg sorry again, we've had a lot of discussion on how to improve this with the SOF developers, and @marc-hb has been improving how we move forward here. We have a pending Readme patch to be merged before we tag today, i.e. the binaries are all correct but we will tag once the readme update is merged today.

  2. perexg commented on Jun 3, 2021

    @perexg
    Author

    Ok, but how I can fetch only v1.8 with the current scheme? If you tag the main branch, I'll fetch both 1.7 and 1.8. And if there are 10 releases, I'll end to fetch all of them.

    The solution may be to filter the files in the release assets, but they have wrong names (v1.7.tar.bz instead sof-bin-1.7.tar.gz) or return back to the release branches.

  3. lgirdwood commented on Jun 3, 2021

    @lgirdwood
    Member

    @marc-hb please also tag the release. You have maintainer access so should be able to do this via GH or git. The known issues are here https://github.com/thesofproject/sof/milestone/15 for linking to in release.

  4. lgirdwood commented on Jun 3, 2021

    @lgirdwood
    Member

    Ok, but how I can fetch only v1.8 with the current scheme? If you tag the main branch, I'll fetch both 1.7 and 1.8. And if there are 10 releases, I'll end to fetch all of them.

    The solution may be to filter the files in the release assets, but they have wrong names (v1.7.tar.bz instead sof-bin-1.7.tar.gz).

    I think @marc-hb has some flows for installation. Lets wait until he's online later today.

  5. marc-hb commented on Jun 3, 2021

    @marc-hb
    Collaborator

    It seems that this repo moved again from the branch system (one branch = one version) to all-in-one branch.

    That's correct.

    It's a bit unclear to me, how I can fetch one version for packaging.

    You can't do that anymore. The longer term fix for that is to move away from using git to release binaries. For now the entire .git/ folder is 10M.

    And it's also unclear, how you'll tag the releases.

    As you noted, tagging the releases in sof-bin does not really make sense anymore.

    The 1.7 version in the README.md is obviously an example.

  6. lgirdwood commented on Jun 3, 2021

    @lgirdwood
    Member

    @marc-hb thanks for your inputs here, are there any docs in sof-docs or GH feature request that describe the release process roadmap and your current plans. I assume the end goal is to have a script that can create a tar.bz2 of FW, ldcs and topolgies for someone to attach to GH releases ?

  7. perexg commented on Jun 3, 2021

    @perexg
    Author

    You can't do that anymore. The longer term fix for that is to move away from using git to release binaries. For now the entire .git/ folder is 10M.

    Just a note: From the storage perspective, it does not matter, if you store different source binary files to one branch or multiple branches. Perhaps, git merging may help, too.

    I'm using the auto-generated tar balls by github now (using the commit-id not your tag which was floating previously). In this case, only the necessary files are downloaded. Your move to artifacts means that you plan to use own scripts for the release, which is also fine, but the storage can be used efficiently with the branches, too like:

    • main branch (without binary files, only license files and scripts)
    • version branch (with binary files for the specific version + merged main branch)

    In this case, you can just tag the version branch and you'll get the automatic tar ball from github.

    Example of commit-id URL to get the tar ball from github for v1.7: https://github.com/thesofproject/sof-bin/archive/6bd5501e6ac3f29dde42a28ffe03a2942aa60640/sof-bin-6bd5501.tar.gz .

  8. perexg commented on Jun 3, 2021

    @perexg
    Author

    Or you may create meta branch with scripts and doc files which will be merged to main (all branches) and to separate release branches. There are multiple possibilities.

  9. marc-hb commented on Jun 3, 2021

    @marc-hb
    Collaborator

    I assume the end goal is to have a script that can create a tar.bz2 of FW, ldcs and topolgies ...

    Most of this script already exists and I use it many times everyday: https://github.com/thesofproject/sof/tree/main/installer

    It's still missing a few features: final tarball, checksums, compilation flags,... I was hoping to get some user feedback to help prioritize further development but I'm afraid most people still use xtensa-build-all.sh + build-tools.sh + their own, private scripts on top. Cause why change? This is also the case of our internal Jenkins-based build and validation; this mismatch makes each release time-consuming and error-prone.

    It's been unsurprisingly hard to get people excited about packaging and releases so the only way forward has been slow: try something (not too) different and wait a bit to see if anyone is confused or complains. There :-)

    ... for someone to attach to GH releases ?

    Yes, my idea was to eventually just upload the output of the installer directly to https://docs.github.com/en/github/administering-a-repository/releasing-projects-on-github/managing-releases-in-a-repository and deprecate the entire sof-bin repo: everything required to perform releases would be self-contained in the main sof repo, no external dependencies and no other branches or tag besides the "real" sof ones. Hopefully simple enough not to require any Powerpoint: configure and run the installer/, upload to github releases, done. No secret sauce hidden in Jenkins or wherever.

    Example: https://github.com/zephyrproject-rtos/sdk-ng/releases

    I'm using the auto-generated tar balls by github now (using the commit-id not your tag which was floating previously). In this case, only the necessary files are downloaded.

    First, thanks for this clarification and your expert feedback in general: even more useful considering how rare you are.

    I didn't even know github had such a convenient git archive implementation, sorry I "broke" that. I understand the convenience of tagging sof-bin now, I will do today.

    Once I tag sof-bin this git archive feature should not really be "broken" any more, just less efficient. Do you mind ignoring the older versions until we move to github releases? Just a few spurious megabytes every few months.

    For the record I "announced" the move to a single branch in January 2021 (among many other things, some abandoned). I understand that didn't make any headline :-) thesofproject/sof#3775 (comment)

    Cc: @aiChaoSONG , @keyonjie , @fredoh9 , @plbossart , @mrhpearson @cujomalainey , please forward to any other (rare!) person who might have an interest for this topic.

  10. plbossart commented on Jun 3, 2021

    @plbossart
    Member

    FWIW we added recently a pointer to sof-bin in the drivers:

    sound/soc/sof/loader.c:                 "you may need to download the firmware from https://github.com/thesofproject/sof-bin/\n");
    sound/soc/sof/topology.c:                       "you may need to download the firmware from https://github.com/thesofproject/sof-bin/\n");

    you may change directions on how you want to deliver the binaries but that site has to stay.

  11. marc-hb commented on Jun 3, 2021

    @marc-hb
    Collaborator

    Good point thanks! We can have a simple "redirection" in the top-level sof-bin/README.md

    As you made no other comment, I assume you agree with all the rest? :-)

  12. plbossart commented on Jun 3, 2021

    @plbossart
    Member

    As long as @perexg is happy I am happy.

  13. 1 remaining item

  14. marc-hb commented on Jun 3, 2021

    @marc-hb
    Collaborator

    I think that you can upload the manually generated assets to the release tag until you have the final solution.

    Good idea and close enough to the final solution anyway; just in a different repo for now. Will do after README.md #37 is merged and after I create the v1.8-rc2 tag in sof-bin.

    From my packager perspective, I would like just to change the source package URL in my rpm spec file

    Thanks, I think I understand better now. In fact I just found https://koji.fedoraproject.org/koji/packageinfo?packageID=30869. I ran dnf download --source alsa-sof-firmware on some Fedora box and had a look at the .spec and other files. I should have done this earlier.

    Do not forget that we store a copy of the source packages in our build system forever, too.

    Are you expressing a space concerning here and/or other(s)?

    Just to be extra sure: in this particular case, "source package" means "sof-bin sources" which means firmware binaries, correct?

  15. perexg commented on Jun 4, 2021

    @perexg
    Author

    I think that you can upload the manually generated assets to the release tag until you have the final solution.

    Good idea and close enough to the final solution anyway; just in a different repo for now. Will do after README.md #37 is merged and after I create the v1.8-rc2 tag in sof-bin.

    From my packager perspective, I would like just to change the source package URL in my rpm spec file

    Thanks, I think I understand better now. In fact I just found https://koji.fedoraproject.org/koji/packageinfo?packageID=30869. I ran dnf download --source alsa-sof-firmware on some Fedora box and had a look at the .spec and other files. I should have done this earlier.

    The Fedora package sources are available also via web portal: https://src.fedoraproject.org/rpms/alsa-sof-firmware

    Here is a straight change from the commit-id to the version style for 1.7 (sitting in my private repo until we know the final source URL for 1.8):

    --- a/alsa-sof-firmware.spec
    +++ b/alsa-sof-firmware.spec
    @@ -4,8 +4,6 @@
     %global _firmwarepath  /usr/lib/firmware
     
     %global sof_version 1.7
    -%global sof_commit 6bd5501e6ac3f29dde42a28ffe03a2942aa60640
    -%global sof_shortcommit %(c=%{sof_commit}; echo ${c:0:7})
     
     %global tplg_version 1.2.4
     
    @@ -16,7 +14,7 @@ Release:        1%{?dist}
     # See later in the spec for a breakdown of licensing
     License:        BSD
     URL:            https://github.com/thesofproject/sof-bin
    -Source:         https://github.com/thesofproject/sof-bin/archive/%{sof_commit}/sof-bin-%{sof_shortcommit}.tar.gz
    +Source:         https://github.com/thesofproject/sof-bin/archive/v%{sof_version}/sof-bin-%{sof_version}.tar.gz
     BuildRequires:  alsa-topology >= %{tplg_version}
     BuildRequires:  alsa-topology-utils >= %{tplg_version}
     Conflicts:      alsa-firmware <= 1.2.1-6
    @@ -36,7 +34,7 @@ License:        BSD
     This package contains the debug files for the Sound Open Firmware project.
     
     %prep
    -%autosetup -n sof-bin-%{sof_commit}
    +%autosetup -n sof-bin-%{sof_version}
     
     mkdir -p firmware/intel/sof
    
    

    Do not forget that we store a copy of the source packages in our build system forever, too.

    Are you expressing a space concerning here and/or other(s)?

    I only mean that the source package contents propagate to all distributions which need to deal with it somehow. So removing duplicates is really welcome.

    Just to be extra sure: in this particular case, "source package" means "sof-bin sources" which means firmware binaries, correct?

    Yes, I mean the source for the packaging here (thus the sof-bin tar or zip archive) - Source: line in the rpm spec file.

    Thank you for your attention and your work on this.

  16. lgirdwood commented on Jun 4, 2021

    @lgirdwood
    Member

    @mwasko who from your team can work with @marc-hb on aligning the signing process with the release process to make it less manual and error prone. I would imagine that some helper scripts could be used to take the signed output from the signing system and upload/copy to the release destination. This can be done for v1.9 so we have 3 months.

  17. marc-hb commented on Jun 4, 2021

    @marc-hb
    Collaborator
  18. perexg commented on Jun 4, 2021

    @perexg
    Author
    diff --git a/alsa-sof-firmware.spec b/alsa-sof-firmware.spec
    index e61e51b..cc65f3b 100644
    --- a/alsa-sof-firmware.spec
    +++ b/alsa-sof-firmware.spec
    @@ -3,7 +3,8 @@
     %define _binaries_in_noarch_packages_terminate_build   0
     %global _firmwarepath  /usr/lib/firmware
     
    -%global sof_version 1.7
    +%global sof_version 1.8
    +%global sof_version_extra -rc2
     
     %global tplg_version 1.2.4
     
    @@ -14,7 +15,7 @@ Release:        1%{?dist}
     # See later in the spec for a breakdown of licensing
     License:        BSD
     URL:            https://github.com/thesofproject/sof-bin
    -Source:         https://github.com/thesofproject/sof-bin/archive/v%{sof_version}/sof-bin-%{sof_version}.tar.gz
    +Source:         https://github.com/thesofproject/sof-bin/releases/download/v%{sof_version}%{sof_version_extra}/sof-bin-v%{sof_version}%{sof_version_extra}.tar.gz
     BuildRequires:  alsa-topology >= %{tplg_version}
     BuildRequires:  alsa-topology-utils >= %{tplg_version}
     Conflicts:      alsa-firmware <= 1.2.1-6
    @@ -34,15 +35,15 @@ License:        BSD
     This package contains the debug files for the Sound Open Firmware project.
     
     %prep
    -%autosetup -n sof-bin-%{sof_version}
    +%autosetup -n sof-bin-v%{sof_version}%{sof_version_extra}
     
     mkdir -p firmware/intel/sof
     
     # we have the version in the package name
    -mv sof-v%{sof_version}/* firmware/intel/sof
    +mv v%{sof_version}.x/sof-v%{sof_version}%{sof_version_extra}/* firmware/intel/sof
     
     # move topology files
    -mv sof-tplg-v%{sof_version} firmware/intel/sof-tplg
    +mv v%{sof_version}.x/sof-tplg-v%{sof_version}%{sof_version_extra} firmware/intel/sof-tplg
     
     # remove NXP firmware files
     rm LICENCE.NXP
    

    The paths are so complex, but it does work. Also, see the difference between v1.7 (easy) and v1.8-rc2 (complex).

  19. marc-hb commented on Jun 4, 2021

    @marc-hb
    Collaborator

    I just verified that I can change the "v1.8-rc2 only" tarball like this with changing neither the .install.sh nor the README.md:

    mv v1.8.x/* .
    rmdir v1.8.x
    

    EDIT: automated and submitted in #48

    So this should look exactly like v1.7 and should let you get back to a single %{sof_version} variable, please confirm.

    I just updated the sof-bin-v1.8-rc2.tar.gz accordingly. Note: I normally never change anything released and I'm surprised github let me do that without even a warning. However this is a release candidate and it's a new approach that I uploaded less than one hour ago. So I'm pretty confident you've been the only user so far :-)

  20. perexg commented on Jun 5, 2021

    @perexg
    Author

    Much better now (it's diff from 1.7 release):

    diff --git a/alsa-sof-firmware.spec b/alsa-sof-firmware.spec
    index e61e51b..5ae4f3b 100644
    --- a/alsa-sof-firmware.spec
    +++ b/alsa-sof-firmware.spec
    @@ -3,18 +3,21 @@
     %define _binaries_in_noarch_packages_terminate_build   0
     %global _firmwarepath  /usr/lib/firmware
     
    -%global sof_version 1.7
    +%global sof_ver 1.8
    +%global sof_ver_pre rc2
    +%global sof_ver_rel %{?sof_ver_pre:.%{sof_ver_pre}}
    +%global sof_ver_pkg v%{sof_ver}%{?sof_ver_pre:-%{sof_ver_pre}}
     
     %global tplg_version 1.2.4
     
     Summary:        Firmware and topology files for Sound Open Firmware project
     Name:           alsa-sof-firmware
    -Version:        %{sof_version}
    -Release:        1%{?dist}
    +Version:        %{sof_ver}
    +Release:        1%{?sof_ver_rel}%{?dist}
     # See later in the spec for a breakdown of licensing
     License:        BSD
     URL:            https://github.com/thesofproject/sof-bin
    -Source:         https://github.com/thesofproject/sof-bin/archive/v%{sof_version}/sof-bin-%{sof_version}.tar.gz
    +Source:         https://github.com/thesofproject/sof-bin/releases/download/%{sof_ver_pkg}/sof-bin-%{sof_ver_pkg}.tar.gz
     BuildRequires:  alsa-topology >= %{tplg_version}
     BuildRequires:  alsa-topology-utils >= %{tplg_version}
     Conflicts:      alsa-firmware <= 1.2.1-6
    @@ -34,15 +37,15 @@ License:        BSD
     This package contains the debug files for the Sound Open Firmware project.
     
     %prep
    -%autosetup -n sof-bin-%{sof_version}
    +%autosetup -n sof-bin-%{sof_ver_pkg}
     
     mkdir -p firmware/intel/sof
     
     # we have the version in the package name
    -mv sof-v%{sof_version}/* firmware/intel/sof
    +mv sof-%{sof_ver_pkg}/* firmware/intel/sof
     
     # move topology files
    -mv sof-tplg-v%{sof_version} firmware/intel/sof-tplg
    +mv sof-tplg-%{sof_ver_pkg} firmware/intel/sof-tplg
     
     # remove NXP firmware files
     rm LICENCE.NXP
    
  21. mwasko commented on Jun 7, 2021

    @mwasko

    @mwasko who from your team can work with @marc-hb on aligning the signing process with the release process to make it less manual and error prone. I would imagine that some helper scripts could be used to take the signed output from the signing system and upload/copy to the release destination. This can be done for v1.9 so we have 3 months.

    @slawblauciak and me are coordinating sof releases. I would recommend to prepare/extend script to validate release content and report if anything is missing. We can also work on some helper scripts to upload signed binaries, but there will be still some manual work dependency.

  22. marc-hb commented on Jun 22, 2021

    @marc-hb
    Collaborator

    1.8 released at https://github.com/thesofproject/sof-bin/releases

    @perexg is everything good or good enough now?

  23. lgirdwood commented on Jun 22, 2021

    @lgirdwood
    Member

    Thanks @marc-hb - Adding release notes shortly - give me a few hours.

  24. perexg commented on Jun 23, 2021

    @perexg
    Author

    It seems fine. Fedora 34 package update: https://bodhi.fedoraproject.org/updates/FEDORA-2021-c838e6f856

    I'm closing this now. It seems that all was settled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions