Modularization of our reproducible build system

About

tor-browser-build contains many necessary pieces in our reproducible builds system that are generic and not tied to Tor Browser specifically (e.g. container creation, dealing with old artifacts etc.). Other projects like Namecoin, Ricochet-Refresh etc. need to work around that by forking that repository and kind of keeping it up-to-date with latest upstream changes.

To make the adoption of our reproducible builds system easier we should split up the tor-browser-build parts into a generic project and Tor Browser specific parts. See: https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40350 for the tracking bug and discussion so far.

Team (add yourself here if you want to join)

Geko (you can find him in irc)

boklm

Skills needed:

having worked with a rbm/tor-browser-build build system before or willing to do so

willing to touch Perl code

Where can we be found: we are hanging around on #tor-dev on OFTC's IRC network (or the respective Matrix channel) and when we need to meet we can use either #tor-meeting or #tor-meeting2.

How can I join: joining the project is easy, just say "Hi!" on #tor-dev or ping me (GeKo). We'll keep notes as we go on this pad trying to enumerate potential tasks which folks can then pick up. Otherwise, just ask.

Potential tasks to work on:

map out what we need to do for getting the modularization off the ground

Add support for multiple rbm.conf files

https://gitlab.torproject.org/tpo/applications/rbm/-/issues/40027

Add support for multiple projects directories

https://gitlab.torproject.org/tpo/applications/rbm/-/issues/40028

Splitting up rbm.conf

Add default remote_start/remote_exec/remote_put/remote_get/remove_finish for containers

https://gitlab.torproject.org/tpo/applications/rbm/-/issues/40029

what about (this is tor-browser-build#40542 now)

debug

has already default option in RBM

compress_tar

default option in RBM

output_dir

has already default option in RBM

tmp_dir

has already default option in RBM

build_log

has already default option in RBM

pkg_type

has already default option in RBM

buildconf: num_procs

default option in RBM (getting it via `nproc`)

buidconf: git_signtag_opts

seems pretty tor-browser-build-specific for now

ENV

maybe we can move that to the default rbm config

gpg_wrapper

maybe we can move that to the default rbm config (rbm/lib/RBM/DefaultConfig.pm), adding an option to enable/disable expired keys

targets: no_build_id

targets: no_containers

I think we can move this to the container module

var: sort_deps

this is used in projects/container-image/config, so I think we should move it to the container module (modules/container/rbm.module.conf)

var: build_id

var: build_id_txt

this currently depends on container options, so maybe we should move that to the container module. Although it would be nice to have it in the default rbm config. Maybe we can have some basic version in default rbm config, with some way for modules like container to extend it. (happens in rbm#40041)

var: container

I think we should move that to the container module, renaming it to something like var_container (depending on which convention we use for naming options from modules)

var: input_files_list

it looks like we don't use input_files_list anywhere, so we might be able to remove it

var: faketime

maybe in default rbm config

var: touch

maybe in default rbm config

var: sign_build

var: sign_build_gpg_opts

this is only used in projects/release, so maybe keep it in tor-browser-build. Maybe that can be moved to projects/release/config.

var: rezip

currently we only use that in projects/firefox/build. Maybe we should move it to projects/firefox/config, unless we have plans to use it in other places

var: set_default_env

this is similar to var/build_id. It currently depends on a container option, so easiest is to move it to the container module. Unless we find some mechanism to allow modules to extend it, in which case we can move it to the default rbm config.

some way to allow extending set_default_env in modules would be adding something like this at the end of set_default_env, which would allow setting container specific things in `module_container/set_default_env`:

[% FOREACH module IN c("modules").keys.sort; GET c("module_" _ module _ '/set_default_env'); END -%] (work happens in rbm#40041

I think we can use some naming convention like this to make it more clear where options come from [+1; your proposal sounds good to me]:

var/name for options defined in tor-browser-build.git or other non-module repos

var_$module/name for options defined in a module (for example we would rename var/deps to var_container/deps)

name for options defined in default rbm config

Where do we document the whole new setup now that tor-browser-build becomes just a project among others? There will be tor-browser-build related documentation changes, sure. But we'd benefit from a general outline for someone who wanted to get started and use rbm for their project in a modularized world. While I like the idea of having "Reproducible builds at Tor", say, on a wiki page in /applications/team it might not be the right place for that general part. (we have now tor-browser-build#40543)

Where do we keep rbm.local.conf.example? [nothing to do here; it's up the respective project whether to make some config example available and, if so, how]

Split up the keyring in case we have modules (like the container one) that use keys currently being in tor-browser-build

we need to add support in rbm for keyring directories in modules (happens in rbm#40028 as well)

Where do we keep non-Tor Browser specific tools (in tor-browser-build/tools)?

keyring management scripts XXX

cleaning up old builds (clean-old, prune-old-builds (do we still need that one at all?)

for the clean-old script, I think we can replace it with rbm#25719 and add some rbm command to clean files not used in the last X days/months (sounds good to me)

for prune-old-builds I think it's only used in automated nightly builds, so we can keep it in tor-browser-build for now

get project X adapted to the modularized world

working on metrics-build (hackweek branch) for that right now

https://gitlab.torproject.org/gk/metrics-build

decide which parts of tor-browser-build.git should be moved to a separate module (I guess that's to be decided by the team but starting with a container module is probably unproblematic)

wishlist: add more comments and internals-documentation to RBM internals [Do you have anything in particular in mind? That is: is there a part that is dense in particular? Or is everything in a similar need of comments/documentation? --GeKo]