RemoteRepo
Summary
This document describes a way to configure hosting git/got repos for users. The goal is to provide shared hosting of repos, from multiple users, without needing to grant shared group privileges or even accounts on the public server. This maximises the isolation, for security reasons, between users and the public facing server. Users will store their repos in their home directory (~/public/gitgot/) and push/pull changes to it locally or via ssh as normal (details below).
Two scripts will be used to accomplish this; gitgot-staging.pl will run on the User Host, and gitgot-deploy.pl will run on the Public Host. They will need to run as root via a cron job at regular intervals (5 - 15 min). The gitgot-staging.pl script will rsync repos from each user's home directory to the staging area on the User Host. It'll then rsync the User Host staging area to the Public Host's staging area. The gitgot-deploy.pl script will rsync the files from the staging area to the web server's chroot for deployment.
The staging area is needed for security reasons. The script needs access to rsync the files between hosts without a password which would be a security concern if it can directly modify other areas on the Public Host. It's on both the the User and Public Host to allow the same scripts to be used even for user accounts hosted on the Public Host. The scripts will need an account (gitgot) on each machine and have an ssh key allowing the User Host to connect to the Public Host without a password. The staging area on both hosts will be owned by the gitgot user. The repos will be owned by the web server user once deployed.
The repo will be staged in /var/gitgot/user/repo directory structure.
1. Setup the Hosts
This configuration assumes two hosts, but can also be run on a single or multiple hosts. The host where the user accounts and home directories are will be known as the User Host, and the host where the web server runs will be known as the Public Host in this configuration. You can have multiple User Hosts that all use the same Public Host, and the Public Host can also have user accounts on it. Advanced configurations could also have multiple public hosts with a load balancing configuration, but that is beyond the scope of this document.
1.1 User Host
The configuration of the User Host consists of creating the ~/public/gitgot directory where users will put the repos they wish to publish. The gitgot-staging.pl script can be configured to create this folder for you if it is missing for existing users or you can use other means to setup the directory.
1.1.1 Modify /etc/skel
For new accounts, setting up the ~/public/gitgot directory can be automated by adding it to /etc/skel directory like this:
doas mkdir -p /etc/skel/public/gitgot doas chown root.wheel /etc/skel/public/gitgot doas chmod 755 /etc/skel/public/gitgot
1.1.2 Create the gitgot user
The gitgot-staging.pl script will move repos from the users directory to the staging area and change ownership to the gitgot user. It skips empty directories. Both the user Host and Public Host will need a gitgot user created like this:
doas useradd -m gitgot doas su - gitgot
1.1.3 Generate an SSH Key
The User Host will need an ssh key so it can send the changes to the Public Host.
ssh-keygen -t ed25519 exit
Use the default path for the key file and leave the password empty. The output will look something like this:
Generating public/private ed25519 key pair. Enter file in which to save the key (/home/gitgot/.ssh/id_ed25519): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/gitgot/.ssh/id_ed25519 Your public key has been saved in /home/gitgot/.ssh/id_ed25519.pub The key fingerprint is: SHA256:lyrfN6pDSRuVI112nfZiMh02aOoHxvgq5Y5jyY2QKpI gitgot@planetofnix.com The key's randomart image is: +--[ED25519 256]--+ | . oo....| | . =.o.+o.| | = + o.o.| | + =.o + .| | . .SBo. + . | | o =oo . | | . . o.B.. . | |E . B+=. o | |.. ..++ooo . | +----[SHA256]-----+
You'll need to do the same on the Public Host and copy the public key over. Run the following, but change <adminUser> and <publicHost> as needed. Doas is broken and wont let you cat >> so a few extra steps to do this as a copy/paste...
NOTE: The <adminUser> has to exist on the Public Host and needs to have write permission on /home/gitgot/.ssh/authorized_keys for this to work. You can use other methods to get the gitgot public key onto the Public Host.
1.1.4 Create gitgot User on Public Host
ssh_as='<adminUser>@<publicHost>' ssh $ssh_as doas useradd -m gitgot doas cat /home/gitgot/.ssh/id_ed25519.pub | ssh $ssh_as 'cat >>~/doasbroken.tmp'
1.1.5 Copy User Host Public Key to Public Host
# Log in to the Public Host ssh $ssh_as # Copy paste these separately after logging in doas su cat doasbroken.tmp >>/home/gitgot/.ssh/authorized_keys rm doasbroken.tmp exit exit
1.1.6 The gitgot-staging.pl script
We'll put the script for copying the files in /home/gitgot/bin/gitgot-staging.pl This script is also available via a git repo.
mkdir /home/gitgot/bin vim /home/gitgot/bin/gitgot-staging.pl
#!/usr/bin/perl
use strict;
use warnings;
my $config={
remoteHost=>'got.ircnow.org',
remoteUser=>'gitgot',
remoteStage=>'/var/gitgot',
repoDir=>'public/gitgot',
createRepoDir=>1,
stageDir=>'/var/gitgot',
gitgotUser=>'gitgot',
key_path=>'/home/gitgot/.ssh/id_ed25519',
rsync_cmd=>'openrsync',
};
# verify we have a local staging area
if (not -d $config->{stageDir}) {
my (undef,undef,$uid,$gid) = getpwnam($config->{gitgotUser})
or die "User " . $config->{gitgotUser} . " doesn't exist";
mkdir $config->{stageDir}, 0750;
chown $uid,$gid,$config->{stageDir};
}
opendir (my $DH, '/home/') || die "Can't opendir /home : $!";
while (my $user = readdir($DH)) {
if (-d "/home/$user/" . $config->{repoDir}) {
# Skip if no repos
opendir (my $RDH, "/home/$user/" . $config->{repoDir})
or die "Can't opendir /home/$user/ : $!";
my @repos = readdir($RDH);
next unless (scalar @repos > 2);
# Make sure user has a staging directory
if(not -d $config->{stageDir} . "/$user") {
mkdir ($config->{stageDir} . "/$user");
}
# rsync files to local staging area
my $src=qq(/home/$user/$config->{repoDir}/*);
my $dst=qq($config->{stageDir}/$user/);
my $rv=system(qq{
openrsync --rsync-path=openrsync -a --delete $src $dst
});
warn "rsync failed $user\n$!\n" if ($rv != 0);
} elsif ($config->{createRepoDir}) { # create missing repo dirs
# skip dirs without users.
my (undef,undef,$uid,$gid) = getpwnam($user) or next;
my $repoPath="/home/$user";
for my $dir (split('/', $config->{repoDir})) {
$repoPath.="/$dir";
unless (-e $repoPath) {
mkdir $repoPath, 0755;
chown $uid,$gid,$repoPath;
}
}
}
}
# rsync to remote staging area if we have a remote host
if (defined $config->{remoteHost}) {
# Only sync if we have repos to deploy
opendir (my $SDH, $config->{stageDir})
or die "Can't opendir ". $config->{stageDir} . " : $!";
my @repos = readdir($SDH);
if(scalar @repos > 2) {
my $src=$config->{stageDir} . "/*";
my $dst=$config->{gitgotUser}. "@" . $config->{remoteHost}
. ":" . $config->{remoteStage} . "/";
my $rv=system(qq{
openrsync --rsync-path=openrsync -ae
"ssh -i $config->{key_path}" --delete $src $dst
});
}
}
1.2 Public Host
The Public Host is where the web server runs. It will run the gitgot-deploy.pl script from a cron job as root to copy the files from the staging area to the proper location for the web server.
NOTE: This part of the documentation is a bit in flux because the current server doesn't have a way to publish repos in user folders. This is a project for another day, hopefully not too far into the future. As a work around, the users name is added as a prefix to the repo name to avoid one user's repo clobbering another users repo with the same name. Hopefully this will be fixed soon.
1.2.1 The gitgot-deploy.pl Script
#!/usr/bin/perl
use strict;
use warnings;
my $config={
stageDir=>'/var/gitgot',
gitgotUser=>'gitgot',
gotwebDir=>'/var/www/got/public',
};
# Sync changes from stageDir to the webdir
opendir (my $DH, $config->{stageDir})
or die "Can't opendir " . $config->{stageDir} . " : $!";
while (my $user = readdir($DH)) {
next if ($user =~ /^\./); # skip . files
if (-d $config->{stageDir} . "/$user") {
# Need the name of each repo so we can add username ass prefix
opendir (my $uDH, $config->{stageDir} . "/$user")
or die "Can't opendir ". $config->{stageDir} . "/$user/ : $!";
while (my $repo = readdir($uDH)) {
next if ($repo =~ /^\./); # skip . files
# need destination folder name to change to prefix
# the username to it to prevent one user from clobbering
# another users repos. Can't do a rename of the folder
# with rsync so have to create the destination folder then
# rsync the contents of it.
my @stat=stat($config->{stageDir} . "/$user/$repo");
my $prefixName="$user-$repo";
my $dst=qq($config->{gotwebDir}/$prefixName/);
mkdir $dst,$stat[2]; #stat[2] is mode of original dir
opendir (my $rDH, $config->{stageDir} . "/$user/$repo")
or die "Can't opendir ". $config->{stageDir} . "/$user/$repo : $!";
while (my $git = readdir($rDH)) {
next if ($git eq '.' or $git eq '..'); # skip . and ..
my $src=qq($config->{stageDir}/$user/$repo/$git);
my $rv=system(qq{
openrsync --rsync-path=openrsync -a --delete $src $dst
});
warn "rsync failed $prefixName\n$!\n" if ($rv != 0);
}
}
}
}
2.0 Using the Repo
NOTE: For most of these commands, git and got can be used interchangeably. Got is a clone of git after all.
Git repos exist in two forms; working or bare. A bare repo contains no working tree and is generally used for published versions of the repo. The bare version is really just the contents of the .git directory in your working tree. It is usually the repo name with a .git extension by convention. So a repo named my myProgram would be published in the directory ~/public/gitgot/myProgram.git.
Your working repo is where you make your changes. It has a hidden directory of .git which contains the bare repo contents. If you are using your shell account on the User Host machine for your working repo, you are a local user. If you are using a remote computer for your working repo then you'll use the remote User commands to access the public repo. The examples below will assume you are using a working directory of ~/myProjects/ where you'll put the working versions of your repos.
2.1 Creating a New Repo
A new repo is created using the git init command. Its easiest if you create the bare repo first since cloning sets up the remote details for you.
2.1.1 Local User
cd ~/myProjects/ git init --bare ~/public/gitgot/myProgram.git git clone ~/public/gitgot/myProgram.git git push
2.1.2 Remote User
cd ~/myProjects/ ssh username@example.com git init --bare ~/public/gitgot/myProgram.git git clone ssh://username@example.com/~/public/gitgot/myProgram.git git push
2.2 Pushing an Existing Repo
If you have an existing working repo that you want to publish you first need to create an empty bare repo then add a remote url to your working repo to push to like this:
2.2.1 Local User
cd ~/myProjects/myProgram git init --bare ~/public/gitgot/myProgram.git git remote add origin ~/public/gitgot/myProgram.git git push --set-upstream origin master # After the first push you only need this git push
2.2.2 Remote User
cd ~/myProjects/myProgram ssh username@example.com git init --bare ~/public/gitgot/myProgram.git git remote add origin ssh://username@example.com/~/public/gitgot/myProgram.git git push --set-upstream origin master # After the first push you only need this git push
2.3 Clone a Bare Repo
Another way to create a bare repo of an existing repo is to clone it as a --bare repo like this:
2.3.1 Local User
cd ~/public/gitgot/ git clone --bare ~/myProjects/myProgram cd ~/myProjects/myProgram git remote add origin ~/public/gitgot/myProgram.git git push --set-upstream origin master # After the first push you only need this git push
2.3.2 Remote User
Its harder to do a clone of your working repo from remote since you probably don't have an easy way to access it from remote. We'll assume you have the ability to ssh
cd ~/public/gitgot/ git clone --bare ssh://username@example.com/~/myProjects/myProgram # On your local machine cd ~/myProjects/myProgram git remote add origin ssh://username@UserHost.com/~/public/gitgot/myProgram.git git push --set-upstream origin master # After the first push you only need this git push