[arch-dev-public] [RFC] Remove svn propset id's

[Jelle van der Waa]

Most of our PKGBUILDs svn propset's break reproducible builds and the
pkgbuild_sha256sum in the BUILDINFO file. When building a package before
commiting the PKGBUILD the propset $Id will differ since the $Id is set on
commit.

This has a few implications, pkgbuild_sha256sum is useless and we can't
reproduce packages due to the BUILDINFO not matching. Also the reproduce tool
uses ASP to retrieve the PKGBUILD and therefore can't verify that it got the
correct PKGBUILD (it relies on pkgbuild_sha256sum).

To resolve this issue we could simply remove the propset id's, since for
me, although not sure about others they don't seem particulary useful.

The proof that the sha256sums's don't match:

$ extra-x86_64-build
$ grep sha256 .BUILDINFO
pkgbuild_sha256sum = 8748d60d2c782f477cb7e692a3dad30be90491cdc13fe8951340da4c0bc7f19e
$ $repopkg

$ sha256sum PKGBUILD
d8ab51a983026dd4a6e2f48e9dc66177eca8cf6c1c0ffefb950b093db299e304 PKGBUILD

# The git checkout

[***@helium][/tmp/bar/community/python-psutil/trunk]%sha256sum PKGBUILD
ce7f1e68a3b426412a24f46016817d30721860c8ef6b3d0a2dddac8ff2448b84 PKGBUILD

[***@helium][/tmp/bar/community/python-psutil/trunk]%diff PKGBUILD /tmp/python-psutil/trunk/PKGBUILD
1c1
< # $Id$
---

# $Id: PKGBUILD 375007 2018-08-28 17:24:26Z jelle $

[Levente Polyak via arch-dev-public]

Post by Jelle van der Waa
Most of our PKGBUILDs svn propset's break reproducible builds and the
pkgbuild_sha256sum in the BUILDINFO file. When building a package before
commiting the PKGBUILD the propset $Id will differ since the $Id is set on
commit.
This has a few implications, pkgbuild_sha256sum is useless and we can't
reproduce packages due to the BUILDINFO not matching. Also the reproduce tool
uses ASP to retrieve the PKGBUILD and therefore can't verify that it got the
correct PKGBUILD (it relies on pkgbuild_sha256sum).
To resolve this issue we could simply remove the propset id's, since for
me, although not sure about others they don't seem particulary useful.
$ extra-x86_64-build
$ grep sha256 .BUILDINFO
pkgbuild_sha256sum = 8748d60d2c782f477cb7e692a3dad30be90491cdc13fe8951340da4c0bc7f19e
$ $repopkg
$ sha256sum PKGBUILD
d8ab51a983026dd4a6e2f48e9dc66177eca8cf6c1c0ffefb950b093db299e304 PKGBUILD
# The git checkout
ce7f1e68a3b426412a24f46016817d30721860c8ef6b3d0a2dddac8ff2448b84 PKGBUILD
1c1
< # $Id$
---

# $Id: PKGBUILD 375007 2018-08-28 17:24:26Z jelle $

I know there are some people who like them because $reason, but even
with svn its not rocket science to get the last author.

+1 from me because on top of your reason:

- IMO such meta data belongs to the repo history and not the file
content itself.

- we will purge it anyway if we finally finish the transition to git


cheers,
Levente

[Eli Schwartz via arch-dev-public]

Post by Jelle van der Waa
Most of our PKGBUILDs svn propset's break reproducible builds and the
pkgbuild_sha256sum in the BUILDINFO file. When building a package before
commiting the PKGBUILD the propset $Id will differ since the $Id is set on
commit.
This has a few implications, pkgbuild_sha256sum is useless and we can't
reproduce packages due to the BUILDINFO not matching. Also the reproduce tool
uses ASP to retrieve the PKGBUILD and therefore can't verify that it got the
correct PKGBUILD (it relies on pkgbuild_sha256sum).
To resolve this issue we could simply remove the propset id's, since for
me, although not sure about others they don't seem particulary useful.

I've never been entirely clear on their motivating purpose, in fact.


Also to expand on the general issue for people who aren't in
#archlinux-reproducible:

When you run extra-x86_64-build, you're using the PKGBUILD you're about
to commit, which svn will set to the expanded propset of the previous
commit... which matches no file ever seen by svn.

If you svn commit, and *then* extra-x86_64-build, then svn will actually
have the right file. What's the likelihood of people making sure to svn
commit before making sure the package actually builds as expected...

IIRC at least some packages seem to have been built by the svntogit
exported PKGBUILD (e.g. via asp) since their pkgbuild_sha256sum can be
obtained from asp.

This results in far too many ways to maybe get the actual file used to
build, and in the most likely scenario it requires deep forensics of the
svn repository.

...

svn propsets will die either way whenever we finally manage to migrate
away from svn and onto git.

[Florian Pritz via arch-dev-public]

Post by Jelle van der Waa
To resolve this issue we could simply remove the propset id's, since for
me, although not sure about others they don't seem particulary useful.

+1 for getting rid of them.

Florian

[David Runge]

Post by Jelle van der Waa
This has a few implications, pkgbuild_sha256sum is useless and we
can't reproduce packages due to the BUILDINFO not matching. Also the
reproduce tool uses ASP to retrieve the PKGBUILD and therefore can't
verify that it got the correct PKGBUILD (it relies on
pkgbuild_sha256sum).

</ insert kill-it-with-fire-meme >

[Jelle van der Waa]

Post by Jelle van der Waa
Most of our PKGBUILDs svn propset's break reproducible builds and the
pkgbuild_sha256sum in the BUILDINFO file. When building a package before
commiting the PKGBUILD the propset $Id will differ since the $Id is set on
commit.

So far, I've only gotten positive reactions. If no one objects I propose
to remove the propsets treewide after a week, so everyone has time
enough to object. Removal will be done as following:

$ sed -ri '/\$Id/d' */trunk/PKGBUILD
$ svn propdel svn:keywords */trunk/PKGBUILD

[Jelle van der Waa]

Post by Jelle van der Waa

Post by Jelle van der Waa
Most of our PKGBUILDs svn propset's break reproducible builds and the
pkgbuild_sha256sum in the BUILDINFO file. When building a package before
commiting the PKGBUILD the propset $Id will differ since the $Id is set on
commit.

So far, I've only gotten positive reactions. If no one objects I propose
to remove the propsets treewide after a week, so everyone has time
$ sed -ri '/\$Id/d' */trunk/PKGBUILD
$ svn propdel svn:keywords */trunk/PKGBUILD

The propsets have been removed from both community and packages.

We can now continue the reproducible build effort knowing that the
PKGBUILD hash in trunk and in BUILDINFO matches :)

[Thore Bödecker via arch-dev-public]

Post by Jelle van der Waa
The propsets have been removed from both community and packages.
We can now continue the reproducible build effort knowing that the
PKGBUILD hash in trunk and in BUILDINFO matches :)
--
Jelle van der Waa

Nice job!

Could someone please update the info here as well:

https://wiki.archlinux.org/index.php/DeveloperWiki:HOWTO_Be_A_Packager

"Simple" TUs are lacking the permissions to edit the dev wiki, as most
of you know ;)

I propose removing the svn propset command from this sections:
https://wiki.archlinux.org/index.php/DeveloperWiki:HOWTO_Be_A_Packager#Adding_a_new_Package

and removing that $Id$ subsection here:
https://wiki.archlinux.org/index.php/DeveloperWiki:HOWTO_Be_A_Packager#SVN_.24Id.24_tags


Cheers,
Thore