Running Repo Renderer on OpenBSD

# Running Repo Renderer on OpenBSD

This guide sets up an OpenBSD 8.0 (or later) server that hosts your git
repositories and serves a website for them. People push to the server over
SSH, each push re-renders the site, and httpd serves the result, with
relayd in front of it for TLS.

You'll need two machines:

- **Your machine**, with a checkout of this repository and Go installed.
  You build the server's programs here, and run the scripts in `tools/`.
- **The server**, running OpenBSD.

Each set of commands below is labelled with the machine it runs on and the
user it runs as. The paths used here are the usual OpenBSD ones, but none
of them are built into the renderer, so you can change any of them in the
config.

When you're done, the server will have:

| Path | Contents | Owner |
|---|---|---|
| `/var/git/NAME.git` | the bare repositories | `git` |
| `/var/www/htdocs/repo-render` | the rendered website | `git` |
| `/var/db/repo-render` | the renderer's saved state and job queue | `git` |
| `/etc/repo-render/repo-render.yaml` | the renderer's config | root |
| `/etc/repo-render/httpd.conf` | httpd's config for the site | root |
| `/usr/local/bin/repo-render`, `/usr/local/bin/repo-render-new-repo` | the programs | root |
| `/usr/local/share/repo-render/post-receive` | the git hook, which every repository links to | root |
| `/usr/local/share/examples/repo-render/` | example configs | root |

A user called `git` owns the repositories, accepts the pushes, and runs
the renderer. When someone pushes, the git hook adds the push to the
renderer's queue and starts the renderer in the background. If the
repository has mirrors (for example on GitHub), the hook also pushes the
changes on to them.

## 1. Install the programs

On your machine, in the checkout, build the server package and copy it to
the server. For an arm64 server, add `SERVER_ARCH=arm64` to the `make`
command.

```sh
make server
scp dist/repo-render-openbsd-amd64.tar.gz SERVER:
```

On the server, as root, unpack it and run the install script, which
installs the programs, the hook and the example configs under
`/usr/local`:

```sh
tar -xzf repo-render-openbsd-amd64.tar.gz
repo-render-openbsd-amd64/install.sh
```

## 2. Create the git user

On the server, as root:

```sh
pkg_add git
echo /usr/local/bin/git-shell >> /etc/shells
useradd -d /var/git -g =uid -s /usr/local/bin/git-shell git
install -d -o git -g git -m 755 /var/git
install -d -o git -g git -m 700 /var/git/.ssh
```

Add the SSH public key of everyone who should be able to push to
`/var/git/.ssh/authorized_keys`, and make the file owned by `git` with
mode 600. The `git-shell` login shell only allows git commands, so these
keys can push and fetch but can't log in.

The git user's home directory is the directory of repositories, so the
address of a repository is simply `git@code.example.com:NAME.git`.

## 3. Configure the renderer

On the server, as root, copy the example config into place and edit it:

```sh
install -d /etc/repo-render
install -m 644 /usr/local/share/examples/repo-render/repo-render.yaml /etc/repo-render/
vi /etc/repo-render/repo-render.yaml
```

The settings that aren't commented out are required. Change
`code.example.com` to your host name, and change the paths if you want
things somewhere else. The commented-out settings show their default
values, and you can uncomment any of them to change it, for example to
set the site's title or to add sections to the index page. Each section
has a short name, like `tools`; which repositories are in it isn't in
this file, but in each repository's own git config, which `git selfhost`
and `repo-render add` set (step 6).

The config file belongs to root, so people who can push can't edit it.

Then create the directories the renderer writes to, and check the config
by running the renderer's `check` command as the git user. It should
finish by printing `ok`.

```sh
install -d -o git -g git -m 755 /var/www/htdocs/repo-render /var/db/repo-render
su -s /bin/sh git -c 'repo-render check'
```

## 4. Configure httpd

The package includes a complete httpd config for the site, with comments
explaining each part. On the server, as root, copy it into place and edit
it:

```sh
install -m 644 /usr/local/share/examples/repo-render/httpd.conf /etc/repo-render/
vi /etc/repo-render/httpd.conf
```

In the editor:

- Change `code.example.com` in the `server` line to your host name.
- Delete the section marked as the private site, unless you're setting
  one up (step 8).
- If `/etc/httpd.conf` already has a `types` block, delete the `types`
  block in this file. httpd only allows one.

If you changed `htdocs` in the renderer's config, change `root` and
`errdocs` to match. The file explains how.

If `/etc/httpd.conf` already has a `server` block for your host name,
remove it. Then include the new file from `/etc/httpd.conf` and start
httpd:

```sh
echo 'include "/etc/repo-render/httpd.conf"' >> /etc/httpd.conf
httpd -n && rcctl enable httpd && rcctl restart httpd
```

httpd now serves the site over plain HTTP on port 8080. The next step puts
relayd in front of it to handle HTTPS.

## 5. Set up HTTPS with relayd

On the server, as root, add a relay to `/etc/relayd.conf` that accepts
HTTPS on port 443 and forwards it to httpd:

```
relay "https" {
        listen on $external_address port 443 tls
        protocol "https"
        forward to 127.0.0.1 port 8080 check tcp interval 30
}
```

You'll need a TLS certificate for relayd to use. The site's httpd config
answers acme-client(1) challenges from `/var/www/acme`, so you can get a
certificate from Let's Encrypt with acme-client.

## 6. Add repositories

The easiest way to add a repository is from your machine, with `git
selfhost`. In your checkout of Repo Renderer, run `make install-tools`,
which links `tools/git-selfhost` into `~/.local/bin` (set `BINDIR` for
somewhere else on your PATH). Then tell it about your server, once:

```sh
git config --global selfhost.server SERVER
```

Then, in the working copy of any project, run:

```sh
git selfhost
```

It asks for a one-line description, which section of the index page to
put the repository in, and its tags, then shows what it will do on the
server and asks first. On the server, `repo-render add` does it, as root:
it creates the repository, named after the working copy's directory, and
sets its section and tags in the repository's own git config
(`repo-render.section` and `repo-render.tag`), where the renderer reads
them. The renderer's config isn't changed. Then `git selfhost` adds the
server as the `origin` remote, and pushes. A repository in no section is
private and doesn't appear on the public site. Within a section,
repositories are listed alphabetically.

To see which repositories are in which section, with their tags, run
this on the server:

```sh
repo-render list
```

The branch it pushes, and makes the server's default, is the one the
server's or the mirrors' `HEAD` names (step 7), or with neither,
`init.defaultBranch` (`main`), whichever branch you have checked out.
It stops, before changing anything, if it can't tell which branch that
is, if you don't have the branch, if you have no commits, if the
server's or a mirror's copy of the branch has commits yours doesn't
(pull them first), or if a tag differs between here and there.

If it stops part way, run it again: it picks up where it was. Run it
again later to move the repository to another section, or change its
tags or description. To skip the questions, give the answers:

```sh
git selfhost --yes -d "A one-line description" --section "Section title" --tag go
```

`git selfhost -h` lists all of its options, and the settings it reads
from git config. Step 7 sets up mirrors.

To copy all of your GitHub repositories to the server at once, run this
on your machine. Try it with `--dry-run` first to see what it will do.

```sh
tools/migrate-from-github --dry-run SERVER
tools/migrate-from-github SERVER
```

Both scripts connect to the server over SSH as your own user and use doas
for the parts that need root. The comment at the top of each script lists
all of its options.

You can also add a repository on the server, as root, with `repo-render
add`, which is what `git selfhost` runs there. Try it with `--dry-run`
first:

```sh
repo-render add NAME --branch main \
    -d "A one-line description" --section tools --tag go
```

`--section` takes a section's name, from the config. Then push to
`git@SERVER:NAME.git`. Run it again to change any of that; what you leave
out stays as it is. Leave out `--section` and the repository stays where
it is (in no section, for a new one); give `--no-section` to take it out
of the index. For a repository with commits, `add` renders it again,
and the index, so a move or a new description shows at once.

## 7. Mirror repositories to GitHub or elsewhere (optional)

A repository can have one or more mirrors. After every push, the hook
pushes the repository's default branch and all of its tags to each
mirror. It never deletes branches or tags on the mirror. If a mirror
push fails, the error is logged to `/var/log/messages` and the next push
tries again.

A force push to the server (after an amend or a rebase) is forced on the
mirrors too, over the commits it replaced. But the hook never throws
away commits made on a mirror, such as a pull request merged on GitHub:
if the mirror's branch is at a commit the server has never had, the
mirror push is refused, and logged. Pull those commits into your working
copy and push them to the server, and the next push mirrors again. To
throw them away instead, on the server, as root:

```sh
repo-render mirror NAME --force
```

To see each repository's mirrors, with any whose last push failed
marked, run `repo-render list`. `repo-render check` warns about a mirror push that failed last
time.

Tags are never forced: a tag moved on the server isn't moved on the
mirrors. Mirroring only goes one way, so it's a good idea to turn off
merging on your mirrors.

Each mirrored repository has an SSH key of its own on the server, in
`/etc/repo-render/keys/NAME`, which you add to each of its mirrors as a
deploy key with write access. A key that leaks can then push to that
one repository, not to everything your account can. GitHub, Codeberg,
GitLab and other Forgejo and Gitea servers all have deploy keys, in the
repository's settings.

The hook runs without anyone there to answer SSH's questions, so the
server needs each mirror host's host key. `repo-render add` adds a host
key it doesn't have yet to `/etc/ssh/ssh_known_hosts`, from
`ssh-keyscan`, and says so. To check them yourself first, or to add
them by hand:

```sh
ssh-keyscan github.com codeberg.org >> /etc/ssh/ssh_known_hosts
```

### With git selfhost

On your machine, name each place you mirror to, with its URL and,
optionally, a link to its deploy key settings. `{repo}` stands for the
repository's name. Then list the ones a new repository gets:

```sh
git config --global selfhost.github.url 'git@github.com:me/{repo}.git'
git config --global selfhost.github.keys 'https://github.com/me/{repo}/settings/keys'
git config --global selfhost.codeberg.url 'git@codeberg.org:me/{repo}.git'
git config --global selfhost.codeberg.keys 'https://codeberg.org/me/{repo}/settings/keys'
git config --global --add selfhost.mirror github
git config --global --add selfhost.mirror codeberg
```

`git selfhost` then makes the repository's key, prints it with the
links, and waits while you add it, and then runs the mirror push and
says which mirrors worked. It doesn't make repositories on the mirrors:
a mirror is only used if its repository is there already, as `git
ls-remote` sees it from your machine. If the working copy's `origin`
is one of the mirrors, it's renamed after the mirror, and the server
becomes `origin`.

For one repository, `--mirror NAME` (repeatable) picks among the mirrors
instead, `--mirror-url URL` adds one that isn't in your config, and
`--no-mirrors` gives none.

### By hand

On the server, as root, give `repo-render add` the mirror:

```sh
repo-render add NAME --branch main \
    --mirror git@github.com:me/NAME.git
```

That makes the repository's key, `/etc/repo-render/keys/NAME`, if it
hasn't one, and sets the repository to push to its mirrors with it. Add
`/etc/repo-render/keys/NAME.pub` to the mirror as a deploy key with
write access. Then push to the mirrors now, to check them:

```sh
repo-render mirror NAME
```

A repository's key is used for all of its mirrors, so for another
mirror, give `add` another `--mirror` and add the same key there. Remove
a mirror with
`git -C /var/git/NAME.git config --unset repo-render.mirror URL`.

`tools/migrate-from-github` doesn't make keys for each repository: give
it `--key FILE`, a key on your GitHub account, which every repository it
mirrors uses.

## 8. Set up a private site (optional)

Repositories that aren't in any section of the index are private. By
default, private repositories aren't rendered at all. You can instead
render them to a separate site at `/private/` that asks for a password.

On the server, as root, create a directory for the private site, add it
to the renderer's config, and create a login:

```sh
install -d -o git -g www -m 750 /var/www/private/repo-render
echo 'private_htdocs: /var/www/private/repo-render' >> /etc/repo-render/repo-render.yaml
htpasswd /var/www/conf/repo-render.htpasswd YOURNAME
chown root:www /var/www/conf/repo-render.htpasswd
chmod 640 /var/www/conf/repo-render.htpasswd
```

Make sure `/etc/repo-render/httpd.conf` still has the private site
section from the example, with your host name in it. Then reload httpd
and render everything:

```sh
httpd -n && rcctl reload httpd
su -s /bin/sh git -c 'repo-render render --all'
```

To add more logins later, run `htpasswd` again. httpd reads the file on
every request, so there's no need to reload it. Browsers send the
password with every request, so only make the private site available
over HTTPS.

## Day-to-day use

These commands run on the server, as root:

- After editing the renderer's config, check it with:

  ```sh
  su -s /bin/sh git -c 'repo-render check'
  ```

  Any change to the config makes the next push re-render every repository.

- To re-render everything immediately:

  ```sh
  su -s /bin/sh git -c 'repo-render render --all'
  ```

  This also removes the pages of any repository you've deleted from
  `/var/git`.

- The renderer logs to `/var/log/daemon`, and mirror pushes log to
  `/var/log/messages`.

## Upgrading

On your machine, update the checkout and build a new package:

```sh
git pull
make server
scp dist/repo-render-openbsd-amd64.tar.gz SERVER:
```

On the server, as root, unpack it and run the install script again, as
in step 1. Every repository links to the same installed hook, so this
upgrades the hook for all of them.

Then compare `/etc/repo-render/httpd.conf` with the new example at
`/usr/local/share/examples/repo-render/httpd.conf`, copy over anything
that changed, and reload httpd:

```sh
httpd -n && rcctl reload httpd
```