e28cc79c92
This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising. |
||
|---|---|---|
| .. | ||
| foreachref | ||
| pipeline | ||
| tests/repos | ||
| batch_reader.go | ||
| blame.go | ||
| blame_test.go | ||
| blob.go | ||
| blob_gogit.go | ||
| blob_nogogit.go | ||
| blob_test.go | ||
| command.go | ||
| command_race_test.go | ||
| command_test.go | ||
| commit.go | ||
| commit_convert_gogit.go | ||
| commit_info.go | ||
| commit_info_gogit.go | ||
| commit_info_nogogit.go | ||
| commit_info_test.go | ||
| commit_reader.go | ||
| commit_test.go | ||
| diff.go | ||
| diff_test.go | ||
| error.go | ||
| git.go | ||
| git_test.go | ||
| hook.go | ||
| last_commit_cache.go | ||
| last_commit_cache_gogit.go | ||
| last_commit_cache_nogogit.go | ||
| lfs.go | ||
| log_name_status.go | ||
| notes.go | ||
| notes_gogit.go | ||
| notes_nogogit.go | ||
| notes_test.go | ||
| parse_gogit.go | ||
| parse_gogit_test.go | ||
| parse_nogogit.go | ||
| parse_nogogit_test.go | ||
| README.md | ||
| ref.go | ||
| remote.go | ||
| repo.go | ||
| repo_archive.go | ||
| repo_attribute.go | ||
| repo_attribute_test.go | ||
| repo_base.go | ||
| repo_base_gogit.go | ||
| repo_base_nogogit.go | ||
| repo_blame.go | ||
| repo_blob.go | ||
| repo_blob_gogit.go | ||
| repo_blob_nogogit.go | ||
| repo_blob_test.go | ||
| repo_branch.go | ||
| repo_branch_gogit.go | ||
| repo_branch_nogogit.go | ||
| repo_branch_test.go | ||
| repo_commit.go | ||
| repo_commit_gogit.go | ||
| repo_commit_nogogit.go | ||
| repo_commit_test.go | ||
| repo_commitgraph.go | ||
| repo_commitgraph_gogit.go | ||
| repo_compare.go | ||
| repo_compare_test.go | ||
| repo_gpg.go | ||
| repo_hook.go | ||
| repo_index.go | ||
| repo_language_stats.go | ||
| repo_language_stats_gogit.go | ||
| repo_language_stats_nogogit.go | ||
| repo_language_stats_test.go | ||
| repo_object.go | ||
| repo_ref.go | ||
| repo_ref_gogit.go | ||
| repo_ref_nogogit.go | ||
| repo_ref_test.go | ||
| repo_stats.go | ||
| repo_stats_test.go | ||
| repo_tag.go | ||
| repo_tag_gogit.go | ||
| repo_tag_nogogit.go | ||
| repo_tag_test.go | ||
| repo_test.go | ||
| repo_tree.go | ||
| repo_tree_gogit.go | ||
| repo_tree_nogogit.go | ||
| sha1.go | ||
| sha1_gogit.go | ||
| sha1_nogogit.go | ||
| signature.go | ||
| signature_gogit.go | ||
| signature_nogogit.go | ||
| submodule.go | ||
| submodule_test.go | ||
| tag.go | ||
| tag_test.go | ||
| tree.go | ||
| tree_blob.go | ||
| tree_blob_gogit.go | ||
| tree_blob_nogogit.go | ||
| tree_entry.go | ||
| tree_entry_gogit.go | ||
| tree_entry_mode.go | ||
| tree_entry_nogogit.go | ||
| tree_entry_test.go | ||
| tree_gogit.go | ||
| tree_nogogit.go | ||
| utils.go | ||
| utils_test.go | ||
Git Module
This module is merged from https://github.com/go-gitea/git which is a Go module to access Git through shell commands. Now it's a part of gitea's main repository for easier pull request.