Jean-Cloud is a small association providing hosting services on second-hand hardware. It is currently launching the Shlagernetes project, a software that enables services to be distributed and managed across several second-hand servers. Git is used in certain cases to install a service on a server or update it.
The objective is to obtain the latest version (or a specific version) of a git repository, using as few resources as possible. By resources, we mean the data flow from the remote to the local folder, as well as the memory space occupied by the repository on the local server.
The created Git repository will not send any data to the remote. It has access to tags but not history. It can keep some local untracked files in addition to its Git clones. It includes submodules if present. It can either download the last main commit (default) or a commit from a certain reference, i.e. branch or tag.
Tests on various commands were carried out on a dummy repository. The test file is transportable and can be downloaded here. Note that to run locally, you need to authorize the protocol for local files: `git config --global protocol.file.allow always`. This is not the default configuration, as it may represent a security vulnerability.
The tests consist in analyzing the memory space taken up by the local repository using the bash command `du`, as well as analyzing the text produced by Git during cloning.
tags is used to fetch tags, and must be specified even if a tag is fetched by reference
depth=1 allows only the last commit to be considered
prune deletes references that are no longer accessible from the local remote folder
prune-tags not only deletes references in the local remote repository that are no longer accessible, but also deletes local tags that do not exist on the remote.
recursive applies the command to submodules of submodules etc.
force ignores local changes to submodules and automatically checks out the new version
depth=1 allows you to consider only the last submodule commit
remote updates from the original remote submodule
CAREFUL: order does matter here. Using this instruction first would make it ineffective because of the `--recurse-submodules` of the `git reset`. This option is yet kept to deal with the case of deletion of a submodule.
this command marks all isolated reflogs as expired immediately instead of 90 days later. This makes for a bigger git gc clean up. git rev-list allows you to check which objects are linked and will not be marked as expired.
if this command is omitted, files created without committing are retained.
This combination does not save any changes made to our repository, apart from the creation of non-committed files if git clean is omitted, which is the case in git_update.sh.
Shallow cloning means not cloning the entire repository history.
A partial clone means not cloning all the files and/or folders in the repository, according to a filter. Filters may concern Binary Large Objects (blobs) or trees. If the filter concerns age, then a partial clone can also be a shallow clone.
Partial clones can be created using the `git clone --filter command`.
During check-out or switch operations, objects initially ignored by the `--filter` can be imported. In our case, we only want to keep one precise commit, which will in any case be let through by `git clone --filter` which is therefore irrelevant.
Partial clones can also be created by `sparse-checking`. Some files and/or folders then do not appear at all in the local folder and are not affected by git porcelain (surface) operations. Nevertheless, the objects associated with these files and folders are still stored in the .git repository.
A surface clone can be created using the `depth=<number>` option, which specifies the number of commits to be kept. This option is available for both the clone and fetch commands.
LFS is a Git extension that lets you manipulate selected files (by name, expression or size) using a local cache. In practice, files are replaced by references in the Git repository and a local folder outside the repository is created to store the files. They are downloaded lazily, i.e. only when checked out. All older versions are stored on an online server.
This is a very interesting mechanism, which we will not use for the same reason as the `--filter` clone: we only want to keep one specific version of the files, which would in any case be downloaded by LFS.
The git filter-branch command is not recommended by the Git documentation. It has several security and performance flaws. It can be used to rewrite branch history using filters.
The Java repo-cleaner library works, but the Git documentation considers the Python filter-repo library to be faster and more secure. We do not wish to install either Python or Java, hence we will not dig any deeper into these two possibilities here.
We want to delete the entire history without filtering, so the git command fetch --depth=1 followed by a git checkout, reset or merge works for us.
## checkout ? merge ? reset ?
Once we have fetched the changes to our local remote/ folder, what is the best way to apply them to our index and working directory?
Let us compare 4 possibilities: `git merge -X`, `git merge -s`, `git reset --hard`, `git checkout -f -B`. The final results are identical, except for `git merge -X`.
However, since we are working in `--depth=1`, the two branches have no common ancestor, and the `--allow-unrelated-histories` option must be supplied. The absence of a common ancestor prevents Git from recognizing similarities within the same file. Any modification to a tracked file on ours, even on a new line, will thus cause a conflict and be overwritten. This command does, however, save newly created and committed files on ours.
Newly created uncommitted files are kept unless `git clean` is run.
[caution: the notions of theirs and ours are reversed here, as `git merge -s theirs` does not exist].
This command applies a ours strategy that gives prevalence to ours, whether there is a conflict or not. It will ignore all changes and file creations committed to theirs. It will also ignore uncommitted modifications. Uncommitted file creations are retained unless git clean is run. This is the same result as with the `git reset --hard` command.
As the `git merge -s theirs` option does not exist, we need to do a little manipulation:
This command is equivalent to `git merge -s ours` and `git reset --hard`, with the difference that you end up in detached HEAD state, which does nos cause any problem in our case since we do not want to push any changes from our repository.
Tests show that the most memory-efficient options are `git checkout -force -B`, `git merge -s ours` and `git --reset hard`, which all do the same thing. However, `git reset --hard` does not involve the creation of a temporary branch and does not end in detached HEAD state, hence it is the one we choose.
Submodules are initially cloned using `git clone --recurse-submodules --remote-submodules`.
They are updated using `git submodule update --init --recursive --force --depth=1 –remote`.
`git reset --hard` must be supplied with the `--recurse-submodules` option in order to delete submodules from the working directory.
The same rules apply to submodules as to the rest of the repository. In the .gitmodules file, it is possible to specify rules for importing submodules, such as a certain tag or branch. By removing `--remote-submodules` from `git clone` and `--remote` from `git submodule update`, submodules will be identical to the repository being cloned and no longer to the original submodule repository.
The script consists of twenty-nine tests (listed in the results below), based on three functions: generate_random_file, get_storage_used and get_bandwidth.
generate_random_file uses the bash command dd and /dev/random.
get_storage_used uses the bash command du.
get_bandwidth retrieves the output of Git commands and extracts the traffic displayed. This does not take submodule traffic into account.
The first five tests concern cloning.
The following tests involve updating the repository using different commands, with three cases for each command: after adding a file, after deleting a file, after adding then deleting a file.
