A common approach to modifying repositories without writing permissions is to fork the repository. However, updating the fork with the latest changes from the original repository is often challenging due to merge conflicts.
Instead of directly merging the codebases, this repository breaks down the merging process into simpler subtasks. By iterating over the commit history it allows to adjust the code changes on commit and file level.
To implement this strategy, this repository uses patch files that reflect the current code adaptations. It contains scripts to simplify tasks like creating, updating or applying a set of patch files.
It is recommended to define an environment variable pointing to the location where Repository Patcher resides on disk.
export PathToRepositoryPatcher="path/to/RepositoryPatcher"
Add the previous command to .bashrc to make it persistent.
Clone the repository with:
git clone https://github.com/SBCV/RepositoryPatcher $PathToRepositoryPatcher
Ensure that the scripts have execute permissions. For example, by running:
chmod +x $PathToRepositoryPatcher/create_patches.sh
chmod +x $PathToRepositoryPatcher/apply_patches.sh
chmod +x $PathToRepositoryPatcher/migrate_rename_commit.sh
chmod +x $PathToRepositoryPatcher/clean_reject_files.sh
Verify the permissions - for example with:
ls -l $PathToRepositoryPatcher/apply_patches.sh
Define two (temporary) variables storing the locations of the patch files and the target repository:
export PathToPatchRepository="path/to/ColmapForVisSatPatches"
export PathToTargetRepository="path/to/ColmapToBePatched"
As an example we will use the repositories Colmap and ColmapForVisSatPatches. Clone the repositories with:
git clone https://github.com/colmap/colmap.git $PathToTargetRepository
git clone https://github.com/SBCV/ColmapForVisSatPatches.git $PathToPatchRepository
Modify the target repository as needed. Then, run:
$PathToRepositoryPatcher/create_patches.sh <PathToTargetRepository> <PathToPatches> <PathToCompatibleCommitFile> <overwrite_flag> <reset_index_changes_flag>
For instance:
$PathToRepositoryPatcher/create_patches.sh $PathToTargetRepository $PathToPatchRepository/patches $PathToPatchRepository/compatible_commit.txt 1 0
The script records the commit (SHA, date, subject) the patches were created
for in <PathToCompatibleCommitFile>.
Before you apply the patches, ensure that the main branch of target repository is active and up-to-date (e.g. HEAD should point to the latest commit). Reason: the apply_patches.sh script adds a new branch (based on the current state of the main branch) and checkouts the desired commit (see below).
To configure the main branch as described above on the command line (WARNING: reset --hard HEAD will PERMANENTLY DELETE modifications in the current branch) you might want to use:
cd $PathToTargetRepository
git reset --hard HEAD
git switch main
git pull
Finally, apply the patches with the following command. Possible values for the --tool option are git_apply reject, git_apply 3way, patch reject and patch merge. The <target_commit_hash> after --commit is optional.
$PathToRepositoryPatcher/apply_patches.sh --tool <tool> <tool_option> --patch_dp <patch_dp> --target_dp <target_dp> --commit <compatible_commit_hash> [<target_commit_hash>]
For instance:
$PathToRepositoryPatcher/apply_patches.sh --tool git_apply reject --patch_dp $PathToPatchRepository/patches --target_dp $PathToTargetRepository --commit $(cat $PathToPatchRepository/compatible_commit.txt | head -c 40)
Note: Do NOT run apply_patches.sh with sh $PathToRepositoryPatcher/apply_patches.sh - this will not produce the required results!
Add a <target_commit_hash> to the --commit option of the apply_patches.sh call.
$PathToRepositoryPatcher/apply_patches.sh --tool <tool> <tool_option> --patch_dp <patch_dp> --target_dp <target_dp> --commit <compatible_commit_hash> <target_commit_hash>
For instance, you can use HEAD to point to the latest commit.
$PathToRepositoryPatcher/apply_patches.sh --tool git_apply 3way --patch_dp $PathToPatchRepository/patches --target_dp $PathToTargetRepository --commit <compatible_commit_hash> HEAD
This will iterate over all commits from <compatible_commit_hash> to <target_commit_hash> and try to individually apply each patch. If the application of the patch (i.e. the merge) fails, the script depending on the value of <option> will do the following:
For the current conflicting_file.ext the script will create a rejection file conflicting_file.ext.rej. The rejection file contains all conflicting hunks that could not be merged.
Use Github or PatchViewer to view the corresponding patch file (e.g. src__base__cost_functions.h.patch) - it will highlight the required changes. Copy the desired changes (i.e. the green parts) to corresponding place in the source code in $PathToColmapLatest!
After merging the hunks, use
$PathToRepositoryPatcher/create_patches.sh $PathToTargetRepository $PathToPatchRepository/patches $PathToPatchRepository/compatible_commit.txt 1 1
to update the set of patches in $PathToPatchRepository/patches (overwriting previously outdated patches). Repeat this procedure until all conflicts are resolved.
Use the apply script with the git_apply --3way to write conflict markers into the affected files. For each conflicting hunk the script adds the code from the file to be patched between <<<<<<< ours and =======, and the code from the patch file between ======= and >>>>>>> theirs. IMPORTANT: because 3-way merging uses a common base file for merging, the content between ======= and >>>>>>> theirs MIGHT DIFFER from the actual hunk in the patch file (for example, this part might contain also context lines of the original hunk). With the conflict markers one can use the 3-way view of VSCode to merge the results (see below).
In VSCode open the file with the conflict markers, and click on the bottom right on Resolve in Merge Editor. In the Merge Editor the top left pane (incomming) represents the content of the patch file and the top right (current) represents the file that should be patched (i.e. the current commit of the vanilla target repository). The bottom pane shows the current state of the merged result. Click on the toolbar of the bottom pane on X Conflicts Remaining to jump to the next conflict (relative to the current selected line). After merging the hunks click on Complete Merge on the bottom right of the merged result in the bottom pane. This is essential to convert the 3way merge to an updated 2way merge. Important: non-matching hunk contexts ARE SHOWN AS SEPARATE conflicts. This is a huge advantage compared to the plain conflict markers and to the visualization in other tools such as meld.
After merging the hunks, use
$PathToRepositoryPatcher/create_patches.sh $PathToTargetRepository $PathToPatchRepository/patches $PathToPatchRepository/compatible_commit.txt 1 1
to update the set of patches in $PathToPatchRepository/patches overwriting previously generated (outdated) patches. Repeat this procedure until all conflicts are resolved.
git apply cannot follow renames: when a commit of the target repository RENAMES (or moves) a file targeted by a patch, applying the patch fails at that commit even though there is no semantic conflict. apply_patches.sh detects this situation and prints a hint naming the renamed files.
Migrate the patch set across such a commit with git's rename detection instead:
$PathToRepositoryPatcher/migrate_rename_commit.sh --patch_dp $PathToPatchRepository/patches --target_dp $PathToTargetRepository --commit <rename_commit_hash>
The script applies the patch set at the PARENT of the rename commit on a temporary branch, commits it, and merges that state INTO the rename commit - git's rename detection carries the patched changes over to the new file paths. Real conflicts (if any) are left in the tree for manual resolution (same workflow as the 3way case above). Afterwards, follow the printed next steps: git reset --mixed <rename_commit_hash>, regenerate the patch set with create_patches.sh, and delete patch files whose target path no longer exists.
Note: commits that SPLIT a file's content into multiple new files defeat rename detection as well - those require manual porting of the affected hunks.
git log --format="%H" --reverse <current_commit_sha>..HEAD | head -n 1
If you observe
warning: inexact rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 568 and retry the command.
run somethng like:
git config --global diff.renameLimit 1000