Building the OpenBSD packages (ports) from source and updating packages
The OpenBSD use it's own format for the ports tree to be easy to build your own packages from the source. Alternatively you can update the ports tree instead using the stable branch. Find out by reading this article how the OpenBSD (stable) packages are built and how can you use other ports branches than stable.
Recently the OpenBSD ports have dropped support for CVS. Internally the OpenBSD development team is using a git similar and compatible version control system named got
Game of Trees. You can use git instead of got as they are almost similar in functionality and got does follows git specifications. The ports tree have been moved to github.com.
In the original design, all the servers were running their separate cron job, updating their own
cvs ports tree and doing a very long cvs
diff. The result was working but not very practical for the people signing who were receiving mails from each machine for each batch.
The new design only changed one thing: One machine was chosen to run the cron job, produce the package list and then will copy that list to the other machines which update their ports tree and run the build. Once all machines finished to build, the initiator machine will gather outputs and send an unique mail with a summary of each machine. This became easier to compare the output of each architecture and once you receive the email this means every machine finished their job and the signing can be done.
Having the summary of all the building machines resulted in another improvement: In the logic of the script, it is possible to send an email telling absolutely no package has been built while the process was triggered, which means, something went wrong. From here, I need to check the logs to understand why the last commit didn't produce a package. This can be failures like a distinfo file update forgotten in the commit.
Also, this permitted fixing one issue: As the
distfilesare shared through a common NFS mount point, if multiples machines try to fetch a distfile at the same time, both will fail to build. Now, the initiator machine will download all the required distfiles before starting the build on every node.
All of the previous scripts were reused, except the one sending the email which had to be rewritten.
The original design of the process is listed below. It has to be done separately on each machine: amd64, arm64, i386, sparc64.
First step is to update the ports tree using
cvs up from a cron job and capture its output. If there is a result, the process continues into the next steps and we discard the result.
Building ports from the source
Make a list of ports to compile
The script uses cvs rdiff between the cvs release and stable branches to show what changed since release, and its output is passed through a few grep and awk scripts to only retrieve the
pkgpaths(the pkgpath of
curlis net/curl) of the packages that were updated since the last release.
File ports/net/dhcpcd/Makefile changed from revision 1.80 to 220.127.116.11 File ports/net/dhcpcd/distinfo changed from revision 1.48 to 18.104.22.168 File ports/net/dnsdist/Makefile changed from revision 1.19 to 22.214.171.124 File ports/net/dnsdist/distinfo changed from revision 1.7 to 126.96.36.199 File ports/net/icinga/core2/Makefile changed from revision 1.104 to 188.8.131.52 File ports/net/icinga/core2/distinfo changed from revision 1.40 to 184.108.40.206 File ports/net/synapse/Makefile changed from revision 1.13 to 220.127.116.11 File ports/net/synapse/distinfo changed from revision 1.11 to 18.104.22.168 File ports/net/synapse/pkg/PLIST changed from revision 1.10 to 22.214.171.124
The script will produce:
net/dhcpcd net/dnsdist net/icinga/core2 net/synapse
From here, for each pkgpath that have been sorted out, the sqlports database is queried to get the full list of pkgpaths of each packages, this will include all packages like flavors, subpackages and multipackages.
This is important because an update in editors/vim pkgpath will trigger this long list of packages:
editors/vim,-lang editors/vim,-main editors/vim,gtk2 editors/vim,gtk2,-lang [...40 results hidden for readability...] editors/vim,no_x11,ruby editors/vim,no_x11,ruby,-lang editors/vim,no_x11,ruby,-main
Once all the pkgpaths are gathered, to build and stored them in a file, next step can start.
Preparing the environment
As the compilation is done on the real system (using PORTS_PRIVSEP though) and not in a chroot we need to clean all packages installed except the minimum required for the build infrastructure, which are rsync and sqlports.
dpb(1) can't be used because it didn't gave good results for building the delta of the packages between release and stable.
The various temporary directories used by the ports infrastructure are cleaned to be sure the build starts in a clean environment.
Compiling and creating the packages
This step is really simple. The ports infrastructure is used to build the packages list we produced at step 2.
env SUBDIRLIST=package_list BULK=yes make package
In the script there is some code to manage the logs of the previous batch but there is nothing more.
Every new run of the process will pass over all the packages which received a commit, but the ports infrastructure is smart enough to avoid rebuilding ports which already have a package with the correct version.
Transfer the package to the signing team
Once the packages are built, we need to pass only the built packages to the person who will manually sign the packages before publishing them and have the mirrors to sync.
From the package list, the package file lists are generated and reused by rsync to only copy the packages generated.
env SUBDIRLIST=package_list show=PKGNAMES make | grep -v "^=" | \ grep ^. | tr ' ' '\n' | sed 's,$,\.tgz,' | sort -u
The system has all the -release packages in
/usr/ports/packages/amd64/all) to avoid rebuilding all dependencies required for building a package update, thus we can't copy all the packages from the directory where the packages are moved after compilation.
Send a notification
Last step is to send an email with the output of rsync to send an email telling which machine built which package to tell the people signing the packages that some packages are available.
As this process is done on each machine and that they don't necessarily build the same packages (no firefox on sparc64) and they don't build at the same speed (arm64 is slower), mails from the four machines could arrive at very different time, which led to a small design change.
The whole process is automatic from building to delivering the packages for signature. The signature step requires a human to be done though, but this is the price for security and privilege separation.