Mirror and sync Gitlab and Github repositories

Monday 5 October 2020 (2020-10-05)

Wednesday 15 May 2024 (2024-05-15)

git, github, gitlab

🇬🇧

Use case#

For serious and big projects I use Gitlab over Github because it's much more easy, efficient and featureful. But you know what Gitlab lacks of? For no good reason people hypes Github over Gitlab so if you want to have contributors to your project it will be 100 times easier if your repository is on Github. Not even counting (even if it's rarer nowaday) the countless services that are available only for Github.

So for those reasons I made a Gitlab repository but needed a Github mirror for the sake of visibility.

Mirroring repositories#

The easiest (and certainly the best) way is setting up a push mirror from GitLab to GitHub. So any commit pushed on any branch on the Gitlab source repository will just be instantly pushed on the Github mirror repository too.

Sync issues#

For a moment I used Zapier to sync new issues from Github to Gitlab since Gitlab kanban-style issue board is endlessly better than Github issue tracker plus Github "Project boards". You just have to plug Zapier GitLab Integrations zaps with GitHub Integrations zaps.

For example you can:

create a new issue on Gitlab for any issue created on Github
create a new issue on Gitlab for any issue created on Github if it has a specific label
create a new issue on Gitlab for any Pull Request (PR) created on Github
etc.

So you don't even need to check Github or enable Github notifications. But as I'm very active on Github too I just disabled that to avoid to have to close issues twice or configure complex sync to auto-close them.

Merge Requests / Pull Requests#

Of course since we're using a push mirroring we wil never want to commit anything on the Github mirror to avoid mirroring accidents.

So what if someone open a PR on Github? Because you won't be able to simply merge it on Github, you'll need to merge it on the Gitlab repository. And there is not Zapier zap to sync, copy or create a new MR on Gitlab when a PR is created on Github.

The best way I have found to handle that is to configure your local repository to have both remotes and to enable MR/PR references so you can easily merge MR/PR locally.

So when I'll have configured that I'll be able to follow this workflow:

Fetch Github PR refs
Merge the Github PR locally
Push the commits from the Github PR to the origin remote dev branch
Auto-mirroring will push the commits on the mirror remote dev branch
Github will detect the commits were merged and so will auto-close the PR

Example setup#

I'll give an example of the setup and workflow with my Rawsec Cybersecurity Inventory:

Source repository git@gitlab.com:rawsec/rawsec-cybersecurity-list.git (origin) on Gitlab
Mirror repository https://github.com/noraj/rawsec-cybersecurity-inventory.git (mirror) on Github

Let's clone the source repository (Gitlab):

$ git clone git@gitlab.com:rawsec/rawsec-cybersecurity-list.git rawsec-cybersecurity-inventory
$ cd rawsec-cybersecurity-inventory

Now add the remote mirror (Github):

$ git remote add mirror https://github.com/noraj/rawsec-cybersecurity-inventory.git

Now we need to configure the git repository to fetch not only branches but also MRs/PRs. So in .git/config add those lines:

# For a Gitlab repository
fetch = +refs/merge-requests/*/head:refs/remotes/<remote_name>/merge-requests/*
# For a Github repository
fetch = +refs/pull/*/head:refs/remotes/<remote_name>/pr/*
# Of course replace <remote_name> with the actual remote name

So with my full .git/config that gives us this:

[core]
  repositoryformatversion = 0
  filemode = true
  bare = false
  logallrefupdates = true
[remote "origin"]
  url = git@gitlab.com:rawsec/rawsec-cybersecurity-list.git
  fetch = +refs/heads/*:refs/remotes/origin/*
  fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*
[branch "master"]
  remote = origin
  merge = refs/heads/master
[branch "dev"]
  remote = origin
  merge = refs/heads/dev
[pull]
  ff = only
[remote "mirror"]
  url = https://github.com/noraj/rawsec-cybersecurity-inventory.git
  fetch = +refs/heads/*:refs/remotes/mirror/*
  fetch = +refs/pull/*/head:refs/remotes/mirror/pr/*

Now we can fetch PR from Gitub with:

$ git fetch mirror

Let's say I move to the dev branch to handle this.

$ git checkout dev

Then I can simply merge a PR (here id 14) like that:

$ git merge mirror/pr/14

And finally push the dev branch to Gitlab:

$ git push origin dev

If the PR branch is outdated or based on the wrong branch, a rebase may be needed.

To do so, move on the PR branch.

git checkout mirror/pr/52

Make the rebase onto the dev branch. In interactive mode we can choose which commit pick, drop or squash.

git rebase --interactive dev

However, it's a detached head, so changes here are temporary. To use them we need to copy the changes into a new temporary branch.

git switch -c tmp

Then we can come back on the dev branch and merge the temporary branch.

git checkout dev
git merge --ff-only tmp

Then remove the merged temporary branch.

git branch -d tmp

Use cases#

That was very useful during Hacktoberfest when people would only make a PR on Github since only Github is eligible for Hacktoberfest.

In many other case people have a Github account and no Gitlab account on don't know they can sign in Gitlab with their Github account thanks to oAuth. So they will naturally PR on Github instead of MR on Gitlab.

Merging MRs/PRs on CLI is great anyway!

misc