byron / gitoxide Goto Github PK
View Code? Open in Web Editor NEWAn idiomatic, lean, fast & safe pure Rust implementation of Git
License: Apache License 2.0
An idiomatic, lean, fast & safe pure Rust implementation of Git
License: Apache License 2.0
It'd be nice if static pages were supported because we want to rewrite our thing in Rust and for security reasons we recommend not running complex scripts and programs on the HTTP server, which means we recommend configuring all git repos with the so-called "dumb" protocol.
Our current thing currently just uses git directly and clones/fetches everything into a single cache repo, which means we can't parallelize it, so we're looking into alternatives for that, whether by using something like gitoxide or having multiple temporary caches and joining them together after fetching. However, having static pages support is a must.
Most of them seem to point at git-config. When seeing unwraps()
I recommend making them into '.expect(โฆ)in case it's more like an assertion, or to transform them into errors to be able to use
?` instead.
Everything unrelated to git-config
I will fix, please let me know @edward-shen if I should use this opportunity to take a first and more thorough look myself and fix them as I go along.
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:70:5
|
70 | / pub fn pop(&mut self) -> Option<(Key, Cow<'event, [u8]>)> {
71 | | let mut values = vec![];
72 | | // events are popped in reverse order
73 | | while let Some(e) = self.section.0.pop() {
... |
93 | | None
94 | | }
| |_____^
|
note: the lint level is defined here
--> git-config/src/lib.rs:3:9
|
3 | #![warn(clippy::pedantic, clippy::nursery)]
| ^^^^^^^^^^^^^^^^
= note: `#[warn(clippy::missing_panics_doc)]` implied by `#[warn(clippy::pedantic)]`
note: first possible panic found here
--> git-config/src/file.rs:82:55
|
82 | return Some((k, normalize_cow(values.pop().unwrap())));
| ^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:242:5
|
242 | / pub fn value(&self, key: &Key) -> Option<Cow<'event, [u8]>> {
243 | | let range = self.get_value_range_by_key(key);
244 | | if range.is_empty() {
245 | | return None;
... |
271 | | )))
272 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:253:22
|
253 | _ => unreachable!(),
| ^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
= note: this warning originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:288:5
|
288 | / pub fn values(&self, key: &Key) -> Vec<Cow<'event, [u8]>> {
289 | | let mut values = vec![];
290 | | let mut found_key = false;
291 | | let mut partial_value = None;
... |
316 | | values
317 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:309:21
|
309 | partial_value.as_mut().unwrap().extend(&**v);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:620:5
|
620 | / pub fn section<'lookup>(
621 | | &mut self,
622 | | section_name: &'lookup str,
623 | | subsection_name: Option<&'lookup str>,
... |
626 | | Ok(self.sections.get(section_ids.last().unwrap()).unwrap())
627 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:626:12
|
626 | Ok(self.sections.get(section_ids.last().unwrap()).unwrap())
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:635:5
|
635 | / pub fn section_mut<'lookup>(
636 | | &mut self,
637 | | section_name: &'lookup str,
638 | | subsection_name: Option<&'lookup str>,
... |
644 | | ))
645 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:643:13
|
643 | self.sections.get_mut(section_ids.last().unwrap()).unwrap(),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:762:5
|
762 | / pub fn remove_section<'lookup>(
763 | | &mut self,
764 | | section_name: &'lookup str,
765 | | subsection_name: impl Into<Option<&'lookup str>>,
... |
773 | | self.sections.remove(&id)
774 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:772:21
|
772 | .remove(self.section_order.iter().position(|v| *v == id).unwrap());
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:811:5
|
811 | / pub fn rename_section<'lookup>(
812 | | &mut self,
813 | | section_name: &'lookup str,
814 | | subsection_name: impl Into<Option<&'lookup str>>,
... |
824 | | Ok(())
825 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:819:18
|
819 | let id = id.last().unwrap();
| ^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:919:5
|
919 | / pub fn get(&self) -> Result<Vec<Cow<'_, [u8]>>, GitConfigError> {
920 | | let mut found_key = false;
921 | | let mut values = vec![];
922 | | let mut partial_value = None;
... |
955 | | Ok(values)
956 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:931:27
|
931 | for event in &self.section.get(section_id).unwrap().0[offset..offset + size] {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:998:5
|
998 | / pub fn set_value<'a: 'event>(&mut self, index: usize, input: Cow<'a, [u8]>) {
999 | | let EntryData {
1000 | | section_id,
1001 | | offset_index,
... |
1010 | | );
1011 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1006:13
|
1006 | self.section.get_mut(§ion_id).unwrap(),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1021:5
|
1021 | / pub fn set_values<'a: 'event>(&mut self, input: impl Iterator<Item = Cow<'a, [u8]>>) {
1022 | | for (
1023 | | EntryData {
1024 | | section_id,
... |
1038 | | }
1039 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1033:17
|
1033 | self.section.get_mut(section_id).unwrap(),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1051:5
|
1051 | / pub fn set_owned_values_all(&mut self, input: &[u8]) {
1052 | | for EntryData {
1053 | | section_id,
1054 | | offset_index,
... |
1065 | | }
1066 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1060:17
|
1060 | self.section.get_mut(section_id).unwrap(),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1074:5
|
1074 | / pub fn set_values_all<'a: 'event>(&mut self, input: &'a [u8]) {
1075 | | for EntryData {
1076 | | section_id,
1077 | | offset_index,
... |
1088 | | }
1089 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1083:17
|
1083 | self.section.get_mut(section_id).unwrap(),
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1114:5
|
1114 | / pub fn delete(&mut self, index: usize) {
1115 | | let EntryData {
1116 | | section_id,
1117 | | offset_index,
... |
1125 | | }
1126 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1121:13
|
1121 | self.section.get_mut(section_id).unwrap().0.drain(offset..offset + size);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1130:5
|
1130 | / pub fn delete_all(&mut self) {
1131 | | for EntryData {
1132 | | section_id,
1133 | | offset_index,
... |
1142 | | self.indices_and_sizes.clear();
1143 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1138:17
|
1138 | self.section.get_mut(section_id).unwrap().0.drain(offset..offset + size);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1195:5
|
1195 | / pub fn get_raw_value<'lookup>(
1196 | | &self,
1197 | | section_name: &'lookup str,
1198 | | subsection_name: Option<&'lookup str>,
... |
1215 | | Err(GitConfigError::KeyDoesNotExist)
1216 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1210:30
|
1210 | if let Some(v) = self.sections.get(section_id).unwrap().value(&key) {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1228:5
|
1228 | / pub fn get_raw_value_mut<'lookup>(
1229 | | &mut self,
1230 | | section_name: &'lookup str,
1231 | | subsection_name: Option<&'lookup str>,
... |
1272 | | Err(GitConfigError::KeyDoesNotExist)
1273 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1242:31
|
1242 | for (i, event) in self.sections.get(section_id).unwrap().0.iter().enumerate() {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1315:5
|
1315 | / pub fn get_raw_multi_value<'lookup>(
1316 | | &self,
1317 | | section_name: &'lookup str,
1318 | | subsection_name: Option<&'lookup str>,
... |
1337 | | }
1338 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1324:17
|
1324 | / self.sections
1325 | | .get(§ion_id)
1326 | | .unwrap()
| |_____________________________^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: docs for function which may panic missing `# Panics` section
--> git-config/src/file.rs:1395:5
|
1395 | / pub fn get_raw_multi_value_mut<'lookup>(
1396 | | &mut self,
1397 | | section_name: &'lookup str,
1398 | | subsection_name: Option<&'lookup str>,
... |
1448 | | }
1449 | | }
| |_____^
|
note: first possible panic found here
--> git-config/src/file.rs:1413:31
|
1413 | for (i, event) in self.sections.get(section_id).unwrap().0.iter().enumerate() {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#missing_panics_doc
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/file.rs:1773:1
|
1773 | impl<'a> Into<Vec<u8>> for GitConfig<'a> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(clippy::from_over_into)]` on by default
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/file.rs:1779:1
|
1779 | impl<'a> Into<Vec<u8>> for &GitConfig<'a> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:130:1
|
130 | impl Into<Vec<u8>> for Event<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:136:1
|
136 | impl Into<Vec<u8>> for &Event<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:350:1
|
350 | impl Into<Vec<u8>> for ParsedSectionHeader<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:356:1
|
356 | impl Into<Vec<u8>> for &ParsedSectionHeader<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:362:1
|
362 | impl<'a> Into<Event<'a>> for ParsedSectionHeader<'a> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:415:1
|
415 | impl Into<Vec<u8>> for ParsedComment<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/parser.rs:421:1
|
421 | impl Into<Vec<u8>> for &ParsedComment<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:224:1
|
224 | impl Into<Vec<u8>> for Value<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:230:1
|
230 | impl Into<Vec<u8>> for &Value<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:376:1
|
376 | impl Into<bool> for Boolean<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:385:1
|
385 | impl<'a, 'b: 'a> Into<&'a [u8]> for &'b Boolean<'a> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:394:1
|
394 | impl Into<Vec<u8>> for Boolean<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:400:1
|
400 | impl Into<Vec<u8>> for &Boolean<'_> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:492:1
|
492 | impl<'a, 'b: 'a> Into<&'a [u8]> for &'b TrueVariant<'a> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:617:1
|
617 | impl Into<Vec<u8>> for Integer {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:623:1
|
623 | impl Into<Vec<u8>> for &Integer {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:839:1
|
839 | impl Into<Vec<u8>> for Color {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
warning: an implementation of `From` is preferred since it gives you `Into<_>` for free where the reverse isn't true
--> git-config/src/values.rs:845:1
|
845 | impl Into<Vec<u8>> for &Color {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= help: consider to implement `From` instead
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#from_over_into
[...]
As a follow-up of #90
It's not in mainline yet, but definitely kind of on the way and available.
reftable.txt
in the git repositoryI thought I saw it show up once in the mainline git repository, but it's definitely not there anymore.
I just found this project today and have a question does gitoxide already have the improvements suggested and made in VFS for git?
In the git-url crate, implement parsing of all URL types that git supports, as documented here: https://www.git-scm.com/docs/git-clone#_git_urls
The implementation should be practical, and does require tests for each case.
nom
can be used, and if used, the parser should not support streaming.
Please feel free to add your author name into the crate list.
It's preferred to have a WIP PR and push all results into that one as I must reserve the right to pick up what's there in case development stalls to avoid blocking overall progress.
anyhow::Error
is smaller than Box<dyn Error>
; it pulls some tricks to store itself in one pointer rather than two. It's also more convenient to use; for instance, anyhow::Result<T>
is an alias for Result<T, anyhow::Error>
, and anyhow::Error
already implies Send
and Sync
so you don't have to specify them. And anyhow will make it easier to provide additional context on errors.
You might consider switching code that currently uses Box<dyn Error>
(or similar), and benchmarking to see if it has any performance impact. (Even if it doesn't provide a noticeable improvement, if it doesn't cause a regression it'd probably still be worth switching.)
I'm not really sure what can be done, maybe add a warning, but gio
collides with an executable from glib2
, I don't know for other linux distributions, but on Arch it's pretty much impossible to not install glib2
.
It would be great to have the ability to create commits that are signed, I thought that this issue might be a good place to start talking about how that workflow should be designed.
Here is a post that describes how the signing process works with git2-rs:
https://blog.hackeriet.no/signing-git-commits-in-rust/
Basically it creates a commit object in memory, lets the user program sign it however they want, and then have a function that writes the object together with the signature string to the git repository.
That design have a couple of pros:
But it also adds complexity for the user of the library, that need to do all the heavy lifting of interacting with gpg (or other). And have multiple different ways to create a commit.
One other point is that there now exists a pure-rust pgp implementation called https://sequoia-pgp.org/ that might be better to use than the gpgme library.
How would you like to design the library functions for creating signed commits?
I don't know how your release workflow works, but the CHANGELOG.md hasn't been updated since 0.4.1, so i have no idea what's new ;)
This is a fun research project to potentially drive feature development one day to allow replacing git2 with gitoxide.
In order to migrate, not all features would have to be present at first at the cost of compiling both gitoxide and git2 (Something which probably should be avoided).
worktree
and submodule
APIs are similar.git status
(requires index comparison with working tree and head to index)
git
, and it's about reducing the amount of pack files which we can already do pretty well I think, and loose objects could be included in the pack as well (even though undeltified)fetch.unpackLimit
to keep repositories clean (cargo
runs gc occasionally as well)git2::Cred::default()
)gitoxide
must build (and be tested on) 32 bit systemsgitoxide
in comparison to git2
git
binary for file://
protocol due to lack of native upload-pack
functionality. Remedy is planned with Rust Foundation grant (stretch goal)ssh
binary for ssh
connectionNote that replacing clone/fetch related features is tracked in #449 .
cd cargo && rg git2:: -l
Hi there,
I don't seem to be able to install gitoxide via cargo on macOS Catalina, despite the builds passing. I get the same errors when running the default (max) and max-termion feature:
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:10:26
|
10 | #[clap(long, short = "t")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:17:26
|
17 | #[clap(long, short = "v")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:33:17
|
33 | short = "f",
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:49:30
|
49 | #[clap(long, short = "p")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:55:30
|
55 | #[clap(long, short = "r")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:76:30
|
76 | #[clap(long, short = "p")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:96:21
|
96 | short = "i",
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:105:30
|
105 | #[clap(long, short = "p")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:131:21
|
131 | short = "c",
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:158:30
|
158 | #[clap(long, short = "s")]
| ^^^ expected `char`, found `&str`
error[E0308]: mismatched types
--> /Users/x/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-0.4.1/src/plumbing/pretty/options.rs:163:21
|
163 | short = "a",
| ^^^ expected `char`, found `&str`
error: aborting due to 11 previous errors
For more information about this error, try `rustc --explain E0308`.
error: could not compile `gitoxide`.
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `gitoxide v0.4.1`, intermediate artifacts can be found at `/var/folders/kt/mnhg_d0d261668l69cs_m2vh0000gn/T/cargo-installJ5NecG`
Let me know if you need any more info.
Given that gitoxide is allocating memory and working across many threads, it would be worthwhile to benchmark using alternate memory allocators.
It'd be trivial to test jemalloc
: just use the jemallocator
crate from the top-level gitoxide bin crate, following the instructions in its README.
The scudo
allocator has even more performance, as well as some resilience against some kinds of security-related bugs. See https://expertmiami.blogspot.com/2019/05/what-is-scudo-hardened-allocator_10.html for performance information. However, it isn't packaged for Rust yet, so it won't be quite as easy to try.
gitoxide/git-odb/src/traverse.rs
Line 66 in 405dd9d
It's too easy to get bitten by git pull
not working anymore as git cannot perform its operations anymore. This is because each fetch MAY create a new pack in libgit2. And even if not, and packs are exploded into loose objects, usually 10_000 of these warrant the creation of a new pack. It's just a matter of time until there are too many of them for the git-repository to memory map all of these for fast access.
Gitoxide should be better and either GC automatically once files cannot be opened anymore due to insufficient handles, or track its own usage to know when they are about to be tight and do an incremental GC to combine a few packs into one, fast.
Another avenue would be try to map only packs that actually have objects we are interested in, avoiding to map all by default, if that's even possible or viable.
Related to rust-lang/docs.rs#975 , and probably many more.
Currently a few portions of the API, like refs parsing, are crate-only. By now it's clear these are useful and after some cleanup should be made public to allow users of git-transport
alone to use them without relying on the existing fetch implementation.
It becomes clear that in order to be more useful for application developers, it's best to support async operation in the transport and protocol layers. This issue tracks the overall progress in making this conversion from blocking code to non-blocking/async code.
blocking
to figure out if we can have blocking::spawn(future)
to avoid clumsy code and overhead.&mut StreamingPeekableIter
into a child future, and when exhausted, it must be transformed back into the &mut _
that created it. That way, only a single mutable reference to said Iter is present at any time. Unfortunately the generated futures (using async
) don't support that as we would have to keep both the future and the parent that created it inside of our own struct. Instead of hiding this using pointers, one could implement the magical part by hand, a custom future, which happily dissolves into its mutable parent iter ref. That would be quite some work though.git-transport
via quinn
- how would that work? Maybe async http isn't even necessary?quinn
needs it).
surf
via hyper
seems to be locked to tokio 0.2 (even though async-std supports tokio 1.0 as well.maybe-async
.tokio
futures::io
related traits.As it's async by default which requires more dependencies than sync code it won't be used in crates to avoid pulling in unnecessary dependencies for builds that don't use it.
This also means definitive duplicates here and there for IO code that otherwise could be deduplicated with maybe_async.
However, tests can and should benefit from maybe_async as minimal dependencies are less important there. Thus we deduplicate tests with it to assure there is no drift in code. This rightfully assumes decent test coverage.
is_async
feature toggle?
quinn
and other custom transports.
quinn
streams support both tokio
and futures
AsyncReadWrite traits, so going with the more general trait at first should be good. Probably it's a good idea to have a redirection for traits so replacing them with the tokio version is easier should it ever be required.surf
based http backend with feature toggles to allow backend selection. This allows choosing the runtime.gixp clone
path - this represents the application and is a good example on how the API feels, alongside the blocking implementation.client::blocking
and start working with maybe_async to pull the git::Connection
into async land, step by step, also using maybe_async. This has a feature-toggle-able ripple effect and hides all other blocking implementations (as the trait is either blocking or async) until these are ported. Generally the git-transport
crate will only support one mode at a time.git-transport/protocol
Capabilities::from_lines_with_version_detection(โฆ)
blocking
crate exists.fetch(โฆ)
#[allow(dead_code)]
in async modegit-transport
git-packetline
Write unit test to reproduce.
Manual testing with this sample program it turned out that loose objects can be located, but not objects that are packed.
Here is how it can be invoked:
git rev-list --objects --no-object-names HEAD | cargo eval ./object-access.rs
I recommend installing cargo-eval
using cargo install --git https://github.com/reitermarkus/cargo-eval
to get the latest greatest.
I have a similar project and must have missed the above gitoxide aim when I visited in October or would have contacted you then. I'm interested in collaboration or to see if your project is close enough to my goal that I could contribute and use it as is, or adapt some part etc.
I happened here today looking again for Rust projects to help with my p2p Git Portal project (see below). The Git Portal proof-of-concept uses git-bug to add issues and comments seamlessly to any git repository. I'm no git expert but believe the approach is sound and it performs well. git-bug can already import issues from github and can push/pull its own data to git services as a git extra CLI (git bug), all without messing with your project's own git data. The only problem I've found with git-bug is that it is written in Go. It was great for a quick proof-of-concept but is much less suitable for building a product for various reasons, so I'm again looking for options using Rust which is how I arrived here today.
So I'm wondering:
Git Portal is a proof-of-concept Svelte + Go/Web Assembly app which can create and view git based issues alongside core git functionality such as browsing the worktree and listing commits. So functionally similar to github but in a test harness UI (someone is working with me on a more github like experience).
The Git Portal demo/poc is live here. First load can be slow because of the 12MB Go runtime, so be patient (instructions in the README). What you see there is a test harness which proves the concept. From that we'll improve the poc to a reasonable demo, but longer term I don't think it makes sense to continue with Go, and Rust is my first choice.
It is written in Go because I chose to build the poc using git-bug for issue support (and go-git for git functionality). It is compiled to Web Assembly and so runs in a browser from static storage without server side code. My aim is to run it from peer-to-peer storage on https://SafeNetwork.tech although it would be possible to use it as the basis of a decentralised app from any static or p2p storage.
Thanks for reading, I'll be interested to hear back although I realise code may be a higher priority and in the mean time I'll spend some time getting to know gitoxide.
Do what's needed to fetch as good as git does (on a bare repository, one without a working tree). This particularly includes proper ref handling as well as safety in the light of concurrent repository access.
gixp pack-receive
into gixp clone
creating an empty repository (for lack of index handling/checkout) and cloning the first pack.git-protocol
.git-repository
to greatly simplifying doing ref-listings and fetches?REF_LOG_ONLY
but they are deleted with the owning reference.git-lock
.std::mem::forget
to implement it correctly.ctrlc
usage with signal-hook (i.e. current atexit handler for interrupts)git-features::interrupt
into git-repository
as this kind of utility is for application usage only. There the git-tempfile
integration makes sense, too.with_mut
less cumbersome to use by assuming the interrupt handler will indeed abort.()
- unit errors per feature toggle.commit()
is recoverablegit-ref
filebuffer
Benchmark it and different alternatives like:
Be sure to put Sha256/32 bytes in there right away to have a more faithful prediction of the future and prevent optimizations that currently are definitely at play.
The outcome is to find the tradeoff between ergonomics and performance and apply the change to the codebase.
Also: Select one name for the type and use it everywhere unchanged, right now some crates use it as Digest
.
Lines 110 to 111 in ea0ecc2
It's known but apparently not super easy to fix given the way things currently work. This issue is verified to exist in verbose/not-quite mode, but is probably also present with the --progress TUI.
This issue was created to track progress on RUSTSEC-2020-0056
and RUSTSEC-2020-0071
, which are advisories caused by a dependency on time =0.2.22
(see time-rs/time#293 for more details).
The git-oxide project won't be affected by these advisories until local time support is added.
Requirements to close this issue:
prodash
dependency is bumped to a version not depending on time =0.2.22
.deny.toml
.Currently there is branching code in git_odb::compound::Db::locate(โฆ)
to workaround a borrowcheck shortcoming using the upcoming Polonius borrow checker.
gitoxide/git-odb/src/compound/locate.rs
Lines 32 to 42 in b317200
Without the workaround, the lookup costs are doubled.
Using Polonius solves the problem but comes at a cost.
tinyvec
.Here is @joshtriplett thoughts on how to resolve this:
When first initializing the Db and looking up all the alternates, rather than recursing, collect all of the alternates as a top-level Vec of
(Packs, Loose)
, so that alternates themselves never have alternates. That way, you don't have to recurse, and there's only one level of alternates. Then, in the loop over alternates, rather than recursively calling locate, just inline the same code that looks in packs and loose, but without the check for alternates.
I think the simplest way to do that would be to have a version ofat
(at_no_alternates
) that ignores alternates entirely, and then have the top-levelat
collect a deduplicated list of alternates and callat_no_alternates
on each one.
Also have a look at how this is currently done for looking up objects in packs. The gist is to get an index of where the object is found first, bypassing the need for borrowing an output buffer, and fill the buffer if an index was found.
gitoxide/git-odb/src/compound/locate.rs
Lines 28 to 30 in b317200
It should be possible to pull in a lower-level crate like git-odb without pulling in prodash, or similar high-level output crates.
(This is not in any way urgent, but it'd be nice to reduce the dependencies of applications building on git-odb and similar.)
When writing any tool it's obvious that we really, really want to be able to resolve refs to objects.
We need to add the concept of stores
right away as there are various types of them.
In typical repositories this requires the following:
files
and packed-refs
format.FETCH_HEAD
and MERGE_HEAD
Right now is not a good time to contribute as a few parts are missing:
I see this commit dc7908f
You don't plan to support shallow clones ?
Has to do with zlib-ng not building on docs.rs, see the logs:
I have a project that would benefit from being able to make reachability queries using only commit-graph files, so I'm interested in adding support for reading commit-graph files to this project.
The README mentions commit graphs would be part of git-repository
, but commit graphs feel more like git-odb
's pack indices to me. In particular, I think it could share the fan-out and oid-bisect code from git-odb/pack/index
. The reachability code could still be in git-repository
.
So:
As promised on Reddit, here's an outline of the parallel pack generation use case:
I'd love to test this out, and I'd be happy to do so on the biggest machines I can throw at it, though I'd also like it to work well in the 2-8 CPU case.
Originally posted by kim June 25, 2021
It turns out that I needed to define my own Transport
impl, because:
to_url
and is_stateful
methods are used, but morally they need to return different things than the default Connection
for my caseTo do that, I needed to make the capabilities parsing public. While that is probably always needed for custom transports, Iโm wondering if the interface could be generalised such that more use cases can just reuse a parametric Connection
. For example, the delegate pattern could be employed for the handshake, and extra parameters could be stored in Connection
.
Iโd be happy to propose a patch, but wanted to gauge first if thereโs interest, or maybe different plans already.
cargo install gitoxide
yields:
...
Compiling crosstermion v0.3.2
Compiling prodash v9.0.0
error[E0425]: cannot find function `select_all` in module `futures_util::stream`
--> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:162:48
|
162 | let mut events = futures_util::stream::select_all(vec![
| ^^^^^^^^^^ not found in `futures_util::stream`
error[E0599]: no method named `boxed` found for struct `futures_util::stream::Map<impl futures_core::Stream, [closure@/Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:163:44: 163:59]>` in the current scope
--> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:163:61
|
163 | ticker(duration_per_frame).map(|_| Event::Tick).boxed(),
| ^^^^^ method not found in `futures_util::stream::Map<impl futures_core::Stream, [closure@/Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:163:44: 163:59]>`
|
= help: items from traits can only be used if the trait is in scope
= note: the following traits are implemented but not in scope; perhaps add a `use` for one of them:
candidate #1: `use futures_lite::future::FutureExt;`
candidate #2: `use futures_lite::stream::StreamExt;`
candidate #3: `use futures_lite::FutureExt;`
candidate #4: `use futures_lite::StreamExt;`
error[E0599]: no method named `boxed` found for struct `futures_util::stream::Map<impl futures_core::Stream, fn(crosstermion::input::Key) -> render::tui::engine::Event {render::tui::engine::Event::Input}>` in the current scope
--> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:164:43
|
164 | key_receive.map(Event::Input).boxed(),
| ^^^^^ method not found in `futures_util::stream::Map<impl futures_core::Stream, fn(crosstermion::input::Key) -> render::tui::engine::Event {render::tui::engine::Event::Input}>`
|
= help: items from traits can only be used if the trait is in scope
= note: the following traits are implemented but not in scope; perhaps add a `use` for one of them:
candidate #1: `use futures_lite::future::FutureExt;`
candidate #2: `use futures_lite::stream::StreamExt;`
candidate #3: `use futures_lite::FutureExt;`
candidate #4: `use futures_lite::StreamExt;`
error[E0599]: no method named `boxed` found for type parameter `impl futures_core::Stream<Item = Event> + Send` in the current scope
--> /Users/sergibli/.cargo/registry/src/github.com-1ecc6299db9ec823/prodash-9.0.0/src/render/tui/engine.rs:165:20
|
165 | events.boxed(),
| ^^^^^ method not found in `impl futures_core::Stream<Item = Event> + Send`
|
= help: items from traits can only be used if the trait is in scope
= note: the following traits are implemented but not in scope; perhaps add a `use` for one of them:
candidate #1: `use futures_lite::future::FutureExt;`
candidate #2: `use futures_lite::stream::StreamExt;`
candidate #3: `use futures_lite::FutureExt;`
candidate #4: `use futures_lite::StreamExt;`
error: aborting due to 4 previous errors
Some errors have detailed explanations: E0425, E0599.
For more information about an error, try `rustc --explain E0425`.
error: could not compile `prodash`.
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `gitoxide v0.4.0`, intermediate artifacts can be found at `/var/folders/2h/z_3xs03s58sb4qc9c1x8nj1w0000gq/T/cargo-installZn5MqM`
Caused by:
build failed
OSX 10.15.6
cargo 1.46.0 (149022b1d 2020-07-17)
rustc 1.46.0 (04488afe3 2020-08-24)
Repro steps:
./gixp pack-create > /dev/null
The global handler (from git-features
) will respond correctly but the process will refuse to terminate until EOF
(ctrl-d) is provided.
I believe the "hanging" behavior is attributed to .lines()
call in gitoxide-core/pack/create.rs:L89
as that call will continue to read from the buffer until EOF.
Not sure what the best solution is here, since the global handler implies it will terminate but it never actually does, but on the other hand this isn't quite a user facing tool, so they might realize they just need to send an EOF to stop it.
gen-pack like plumbing command
Generate a pack using some selection of commits or possibly objects. Drives different kinds of iteration as well as ways of building a pack.
gixp pack-create
Easy*
hash_hasher
for single-threaded counting to see how it performs compared to the standard hash-set with hasher override.
prodash
will cause slowdown.ignore_replacements
. It's correct to ignore replacements during reachability traversal, i.e. packpack-index-from-data
.cargo build --release --no-default-features --features max,cache-efficiency-debug --bin gix && /usr/bin/time -lp ./target/release/gix -v free pack create -r tests/fixtures/repos/rust.git --statistics --thin -e tree-traversal --pack-cache-size-mb 200 --object-cache-size-mb 100 HEAD --nondeterministic-count
cargo build --release --no-default-features --features max,cache-efficiency-debug --bin gix && /usr/bin/time -lp ./target/release/gix -v free pack create -r tests/fixtures/repos/rust.git --statistics --thin -e tree-diff --pack-cache-size-mb 400 --object-cache-size-mb 100 < tests/fixtures/repos/rust.git/first-thousand.commits
echo HEAD | /usr/bin/time -lp git -C . pack-objects --all-progress --stdout --revs >/dev/null
gixp
subcommand to make the functionality available for stress testing and performance testingI have a system image, which contains potentially 10k to 100k blobs taking up a total of 100M-1G, many of them binary (such as executables and libraries), and I don't know whether the server has any of those blobs or not. I want to turn that into a standalone commit (typically with no parents), push that commit as a compressed pack while (easier) transferring no blobs or trees the server already has, (medium) delta-compressing blobs reasonably against each other, and (hard) delta-compressing any blobs or trees vs the most similar ones the server does have to make a thin-pack. If the server doesn't have anything useful I want to recognize that quickly and just push a reasonably compressed full pack. Metrics I care about: the server spending as little memory as possible incorporating the pack into its repository (to be immediately usable), the transfer being limited only by bandwidth on either fast (100Mbps) or slow (1Mbps) connections to the server, and getting decent compression and delta-compression to help with the slow-connection case.
The user is working in a git repository, such as the Linux kernel repository. As fast as possible, I want to figure out the changes from their most recent commit (ignoring the index), create a new commit with their current commit as a parent, and then push that commit to the server (along with whatever the server doesn't have). Same easy/medium/hard as above, same metrics as above.
stream pack entries (base objects only) from an iterator of input objects
write pack entries to a V2 pack
clone-like - the other side has no objects
fetch-like - the other side has some objects
Handle tag objects - their pointee must be added to the set as well.
Make it so that the total amount of objects to be written is known in advance
Statistics about re-used objects and repacked ones, maybe more. This is critical to eventually avoid repacking most.
--statistics
restore chunk-ordering to allow multi-threaded generators to yield the same pack all the time (chunks may be out of order)
figure out why the pack hash in parallel mode is still not reproducible
write object entries directly, without forcing to copy their compressed data into an output::Entry
re-use delta objects as well and write their offsets correctly.
gixp pack-create
git rev-list
) (AsIs traversal is default here)gitoxide/git-odb/src/linked/find.rs
Line 52 in 34b6a2e
gitoxide/git-odb/src/pack/data/init.rs
Lines 32 to 33 in ad04ad3
Edit: oops, opened this without content. hang on a second.
This is kind of a tracking issue for the currently broken windows build, as mentioned in the latest release.
I opened this because i needed a space to leave my findings, specifically this issue: actions/runner-images#1143. I never worked with cross before, but perhaps it's an option to run this workflow on a linux runner?
Supporting it adds quite some complication to our codebase, namely:
Provider
is not an iterator.069b312 copied the documentation from git into the project, but the overall project license is still listed as MIT. (And in any case, you don't want to have to keep updating them from git.git.)
Hey @Byron,
I met with @xla and @kim to discuss gitoxide
and our needs in radicle-link
.
We landed in a high-level priority list as follows:
Looking through the checklist from the README, we saw that one part of the fetching story is
complete but we would need pack-send
which is unchecked. As I mentioned above, the fetch exchange
would be top of our list. Maybe you could flesh out what's complete here, what's missing, and where
we could start looking in the code-base to understand this better.
To elaborate on our needs in this area, we need to implement our own transport
layer, so we'd be interested in what the gitoxide
API looks like here.
We will have a lot of refs, which we constantly read and update. Eventually, we will want a
"real" database, but if we keep using libgit2, we would have to teach it to use a custom backend. As
a middleground, if gitoxide supports "packed refs" (that's a single file containing all the refs),
this would already be an improvement (that's assuming libgit2 loads the refs lazily).
For reference handling, we need to be able to read references, perform commits, and compute merge
basing. This functionality is necessary for our handling of documents on the network.
We use the git config in a couple of places currently. One way is to track remotes and the other is
for configuring working-copies for fetching changes. Since we can rely on libgit here, it's lower
priority.
Because in v2, the server doesn't send all the refs unsolicited, but the client requests them
explicitly, and can filter by prefix. It is also, afaiu, that the "ref-in-want" feature would allow
us to request refs across namespaces, without jumping through hoops to make the server ref
advertisement match what we are likely going to fetch (we can compute all refs we want to fetch up
front).
Somewhat relatedly, libgit2 currently insists to only update (a subset of) the refs the server
advertised, at the exact oids the server advertised -- what we would want is a bit more control over
this:
Say we have refs/remotes/XYZ/rad/foo
, for which the remote peer advertises that its tip is
abcde
. We know, however, that XYZ signed a tip xyz09
which is an ancestor of abcde
. We'd like
to ask the remote end for xyz09
, and if that object is in the packfile, we'd update
refs/remotes/XYZ/rad/foo
locally to this. Currently, if the remote end is ahead, we would not
update the ref at all.
We're excited about the prospect of using gitoxide
and being able to contribute to it, however,
we've only got so much person-power at the moment. So this is a good first step in figuring out how
much work is involved. After that, we can work out how much time we delegate to helping out on the
project.
git index-pack is streaming a pack and creates an index from it. The difficulty arises from having to decompress every entry in the pack stream, which can be composed of many small objects. These are placed in some sort of index to accelerate the next stage that is all about resolving the deltas in order to produce a SHA1. Per pack entry, the SHA1, pack offset and CRC32 are written into the index file to complete the operation.
The indexing phase in inherently single-threaded with little potential for improvements, whereas the resolving phase is fully multithreaded and entirely lock free. The first phase could be improved by writing the pack file in parallel - right now it happens after reading it (the pack file is used later for lookup to not hold everything in memory). However, IO doesn't appear to be the bottleneck at all.
Compared to gitoxide
, git is considerably faster when creating the index, averaging 54MB/s of reading uncompressed bytes. gitoxide
clocks in at about 45MB/s 50MB/s, and slows down considerably during the end. Part of that slowdown might be attributed to this issue with resetting miniz_oxide's decompressor.
Luckily gitoxide
is way faster when resolving deltas, which already gives it a good first place in the race, with some room for more if it manages to get as fast as git when decompressing and indexing objects.
The picture below shows the fastest git run I could produce, probably with everything being properly cached:
Without cache, it seems to look different:
The fastest gitoxide
runs, which are pretty comparable in the amount of work done, as they also write out the pack and the index. The only difference is that they use the packfile directly instead of reading it from stdin, it's streamed nonetheless though, and merely an oversight.
Memory consumption of git hovers consistently around 650MB (for the kernel pack), and is lower higher than the 1.2GB 750MB 580MB that gitoxide
uses. However, gitoxide
can temporarily use more memory as it keeps intermediate decompressed objects per thread, whose maximum sizes depend on the amount of children and the base size. So I have seen this go up to 850MB for small fractions of time because of that.
Hi! I'm a developer that's been tasked to implement GitOxide into a WASM stack in an effort to clone a project as fast as possible in the browser.
As part of that work, I'm looking to introduce a lot of new functionality into the project to get clone
working. As an initial task, I'm hoping to get the git-config
package able to read and write config files. I've got a (very) WIP port of config parsing logic from isomorphic-git
over on my fork.
I won't bother asking for a code review right now, I'm brand new to Rust and have a colleague taking a look at the code right now to give me some pointers on how I can improve my work
That said, I am unsure of what direction we want to go in with regards to data organization. The current output is a pretty unmanageable tangled vector of Sections and Entries alike in a single struct
I've noticed that the current git-config
file has both Section
and Entry
types that look to contain all of the relevant properties, so I'll be refactoring my current code to use that data structure.
Outside of that, I was hoping you could give me some kind of direction when it comes to the different mod
s in the git-config/lib.rs
file:
https://github.com/Byron/gitoxide/blob/main/git-config/src/lib.rs#L57-L72
https://github.com/Byron/gitoxide/blob/main/git-config/src/lib.rs#L131-L217
I'm not entirely sure what borrowed
or spawned
is referring to (which I admit as a point-of-noobiness), and I'm not sure I track what Span
is doing here.
I also would love some insight as to what a Token
is referring to here:
https://github.com/Byron/gitoxide/blob/main/git-config/src/file.rs#L4-L23
As I'm not sure an instance that would have a Section
and only a single Entry
Looking forward to learning more and being able to meaningfully contribute upstream!
By running cargo install gitoxide
I get following crash while compiling.
โฏ cargo install gitoxide
Updating crates.io index
Installing gitoxide v0.4.3
Compiling libc v0.2.81
Compiling proc-macro2 v1.0.24
Compiling unicode-xid v0.2.1
Compiling syn v1.0.54
Compiling version_check v0.9.2
Compiling cfg-if v0.1.10
Compiling serde_derive v1.0.118
Compiling serde v1.0.118
Compiling autocfg v1.0.1
Compiling log v0.4.11
Compiling memchr v2.3.4
Compiling proc-macro-hack v0.5.19
Compiling bitflags v1.2.1
Compiling futures-core v0.3.8
Compiling scopeguard v1.1.0
Compiling crunchy v0.2.2
Compiling lazy_static v1.4.0
Compiling getrandom v0.2.0
Compiling tiny-keccak v2.0.2
Compiling cfg-if v1.0.0
Compiling unicode-width v0.1.8
Compiling futures-task v0.3.8
Compiling pin-utils v0.1.0
Compiling cc v1.0.66
Compiling pkg-config v0.3.19
Compiling cache-padded v1.1.1
Compiling quick-error v2.0.0
Compiling futures-io v0.3.8
Compiling fastrand v1.4.0
Compiling parking v2.0.0
Compiling typenum v1.12.0
Compiling pin-project-lite v0.1.11
Compiling unicode-segmentation v1.7.1
Compiling cassowary v0.3.0
Compiling const_fn v0.4.4
Compiling waker-fn v1.1.0
Compiling radium v0.5.3
Compiling futures-sink v0.3.8
Compiling event-listener v2.5.1
Compiling build_const v0.2.1
Compiling tinyvec_macros v0.1.0
Compiling once_cell v1.5.2
Compiling matches v0.1.8
Compiling vec-arena v1.0.0
Compiling getrandom v0.1.15
Compiling ansi_term v0.12.1
Compiling human_format v1.0.3
Compiling bytesize v1.0.1
Compiling nix v0.18.0
Compiling tap v1.0.0
Compiling maybe-uninit v2.0.0
Compiling funty v1.0.1
Compiling wyz v0.2.0
Compiling hex v0.4.2
Compiling humantime v2.0.1
Compiling sha1 v0.6.0
Compiling percent-encoding v2.1.0
Compiling itoa v0.4.6
Compiling curl v0.4.34
Compiling opaque-debug v0.3.0
Compiling ppv-lite86 v0.2.10
Compiling cpuid-bool v0.1.2
Compiling byteorder v1.3.4
Compiling bytes v0.5.6
Compiling same-file v1.0.6
Compiling home v0.5.3
Compiling ryu v1.0.5
Compiling adler v0.2.3
Compiling arrayvec v0.5.2
Compiling base64 v0.12.3
Compiling remove_dir_all v0.5.3
Compiling anyhow v1.0.35
Compiling serde_json v1.0.60
Compiling hashbrown v0.9.1
Compiling quick-error v1.2.3
Compiling termcolor v1.1.2
Compiling strsim v0.10.0
Compiling vec_map v0.8.2
Compiling os_str_bytes v2.4.0
Compiling lock_api v0.3.4
Compiling lock_api v0.4.2
Compiling instant v0.1.9
Compiling textwrap v0.12.1
Compiling concurrent-queue v1.2.2
Compiling git-repository v0.4.0
Compiling standback v0.2.13
Compiling time v0.2.23
Compiling generic-array v0.14.4
Compiling nom v6.0.1
Compiling proc-macro-error-attr v1.0.4
Compiling proc-macro-error v1.0.4
Compiling futures-channel v0.3.8
Compiling tinyvec v1.1.0
Compiling num-traits v0.2.14
Compiling crossbeam-utils v0.7.2
Compiling miniz_oxide v0.4.3
Compiling indexmap v1.6.1
Compiling unicode-bidi v0.3.4
Compiling crc v1.8.1
Compiling form_urlencoded v1.0.0
Compiling walkdir v2.3.1
Compiling humantime v1.3.0
Compiling uluru v1.0.0
Compiling heck v0.3.1
Compiling async-channel v1.5.1
Compiling libz-sys v1.1.2
Compiling curl-sys v0.4.39+curl-7.74.0
Compiling futures-lite v1.11.2
Compiling bitvec v0.19.4
Compiling quote v1.0.7
Compiling unicode-normalization v0.1.16
Compiling mio v0.7.6
Compiling signal-hook-registry v1.2.2
Compiling polling v2.0.2
Compiling num_cpus v1.13.0
Compiling nb-connect v1.0.2
Compiling socket2 v0.3.17
Compiling filebuffer v0.4.0
Compiling atty v0.2.14
Compiling crossbeam-channel v0.4.4
Compiling rand_core v0.5.1
Compiling async-io v1.3.1
Compiling signal-hook v0.1.16
Compiling env_logger v0.7.1
Compiling idna v0.2.0
Compiling btoi v0.4.2
Compiling const-random-macro v0.1.13
Compiling rand_chacha v0.2.2
Compiling digest v0.9.0
Compiling block-buffer v0.9.0
Compiling sha-1 v0.9.2
Compiling rand v0.7.3
Compiling const-random v0.1.13
Compiling url v2.2.0
Compiling ahash v0.3.8
Compiling dashmap v3.11.10
Compiling tempfile v3.1.0
Compiling ctrlc v3.1.7
Compiling pin-project-internal v1.0.2
Compiling time-macros-impl v0.1.1
Compiling thiserror-impl v1.0.22
Compiling clap_derive v3.0.0-beta.2
Compiling time-macros v0.1.1
Compiling thiserror v1.0.22
Compiling pin-project v1.0.2
Compiling futures-util v0.3.8
Compiling clap v3.0.0-beta.2
Compiling smallvec v1.5.1
Compiling bstr v0.2.14
Compiling parking_lot_core v0.7.2
Compiling parking_lot_core v0.8.1
Compiling parking_lot v0.10.2
Compiling parking_lot v0.11.1
Compiling git-ref v0.4.0
Compiling git-packetline v0.2.1
Compiling git-url v0.1.0
Compiling git-object v0.4.0
Compiling crossterm v0.17.7
Compiling tui v0.10.0
Compiling tui-react v0.10.1
Compiling crosstermion v0.3.2
Compiling prodash v10.0.2
Compiling git-features v0.6.0
Compiling git-features v0.7.0
Compiling git-odb v0.4.2
Compiling git-transport v0.2.1
Compiling git-protocol v0.1.1
Compiling gitoxide-core v0.4.1
error[E0433]: failed to resolve: could not find `DecodeEntryLRU` in `cache`
--> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/explode.rs:221:22
|
221 | pack::cache::DecodeEntryLRU::default,
| ^^^^^^^^^^^^^^ could not find `DecodeEntryLRU` in `cache`
error[E0433]: failed to resolve: could not find `DecodeEntryLRU` in `cache`
--> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:145:53
|
145 | EitherCache::Right(pack::cache::DecodeEntryLRU::default())
| ^^^^^^^^^^^^^^ could not find `DecodeEntryLRU` in `cache`
error[E0412]: cannot find type `DecodeEntryNoop` in module `pack::cache`
--> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:76:23
|
76 | Left(pack::cache::DecodeEntryNoop),
| ^^^^^^^^^^^^^^^ help: a trait with a similar name exists: `DecodeEntry`
|
::: /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/git-odb-0.4.2/src/pack/cache.rs:4:1
|
4 | pub trait DecodeEntry {
| --------------------- similarly named trait `DecodeEntry` defined here
error[E0412]: cannot find type `DecodeEntryLRU` in module `pack::cache`
--> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:77:24
|
77 | Right(pack::cache::DecodeEntryLRU),
| ^^^^^^^^^^^^^^ help: a trait with a similar name exists: `DecodeEntry`
|
::: /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/git-odb-0.4.2/src/pack/cache.rs:4:1
|
4 | pub trait DecodeEntry {
| --------------------- similarly named trait `DecodeEntry` defined here
error[E0425]: cannot find value `DecodeEntryNoop` in module `pack::cache`
--> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:143:52
|
143 | EitherCache::Left(pack::cache::DecodeEntryNoop)
| ^^^^^^^^^^^^^^^ not found in `pack::cache`
error[E0061]: this function takes 3 arguments but 4 arguments were supplied
--> /Users/patrickhaller/.cargo/registry/src/github.com-1ecc6299db9ec823/gitoxide-core-0.4.1/src/pack/verify.rs:149:17
|
149 | idx.verify_integrity(
| ^^^^^^^^^^^^^^^^ expected 3 arguments
150 | pack.as_ref().map(|p| (p, mode, algorithm.into())),
| --------------------------------------------------
151 | thread_limit,
| ------------
152 | progress,
| --------
153 | cache,
| ----- supplied 4 arguments
error: aborting due to 6 previous errors
Some errors have detailed explanations: E0061, E0412, E0425, E0433.
For more information about an error, try `rustc --explain E0061`.
error: could not compile `gitoxide-core`.
To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `gitoxide v0.4.3`, intermediate artifacts can be found at `/var/folders/pz/1bgtc0456j33k6w2w_ggxm5m0000gn/T/cargo-install7bD95K`
Caused by:
build failed
I am on macOS 11.1 (Beta).
I would be very interested in having the support for Git server operations receive-pack
and upload-pack
. I see this mentioned in the README but there's no timeline.
Tree parsing is defined as all_consuming(many1(parse_entry))
, which rejects the (valid) empty tree. I'm pretty sure that should be many0
.
Hey! I love the idea of the organize
command to clean up my messy ~/Code
folder.
I ran it this morning and have found that all of the moved directories now end in .git
. Here's an example of the directory with all my work repositories in it:
drwxr-xr-x - alafroscia 2 Oct 2020 AIExpedition.git
drwxr-xr-x - alafroscia 4 Dec 2019 boxes-interview-app.git
drwxr-xr-x - alafroscia 10 Jan 2020 builder.git
drwxr-xr-x - alafroscia 11 Feb 2020 cartridge_hooks.git
drwxr-xr-x - alafroscia 9 Mar 14:21 dependabot-clubhouse-actions.git
drwxr-xr-x - alafroscia 11 Jan 08:53 docker-images.git
drwxr-xr-x - alafroscia 20 May 2020 ember-bar-chart.git
drwxr-xr-x - alafroscia 5 Dec 2019 ember-cli-deploy-github-deployment-status.git
drwxr-xr-x - alafroscia 24 Jan 2020 ember-cli-deploy-gitlab-scm-data-generator.git
drwxr-xr-x - alafroscia 23 Oct 2019 ember-cli-mirage.git
drwxr-xr-x - alafroscia 2 Feb 17:09 ember-cli-sentry.git
drwxr-xr-x - alafroscia 1 Mar 13:57 ember-data-json-api-bulk-ext.git
drwxr-xr-x - alafroscia 4 Sep 2019 ember-examples.git
drwxr-xr-x - alafroscia 20 Sep 2019 ember-pop-over.git
drwxr-xr-x - alafroscia 25 Jan 2020 ember-radio-button.git
drwxr-xr-x - alafroscia 27 Apr 2020 eslint-plugin-no-wildcard-postmessage.git
drwxr-xr-x - alafroscia 11 Mar 16:02 fluid-docs-investigation.git
drwxr-xr-x - alafroscia 13 Nov 2019 fluid-tailwind-playground.git
drwxr-xr-x - alafroscia 18 Sep 2019 fluid-wc.git
drwxr-xr-x - alafroscia 8 May 13:22 front-end.git
drwxr-xr-x - alafroscia 8 Apr 2020 fullstack-interview-stats-table.git
drwxr-xr-x - alafroscia 11 Feb 2020 gitd
drwxr-xr-x - alafroscia 11 Feb 2020 gitd.git
drwxr-xr-x - alafroscia 6 May 16:34 lint-config.git
drwxr-xr-x - alafroscia 28 Apr 16:15 movableink.git
drwxr-xr-x - alafroscia 25 Feb 16:13 page-objects.git
drwxr-xr-x - alafroscia 29 Jun 2020 provisioning
drwxr-xr-x - alafroscia 6 Sep 2019 rfcs.git
drwxr-xr-x - alafroscia 28 Oct 2020 studio-app-tools.git
drwxr-xr-x - alafroscia 28 Oct 2020 studio-apps.git
drwxr-xr-x - alafroscia 29 Mar 15:14 studio-framework.git
drwxr-xr-x - alafroscia 28 Oct 2020 studio-package-proxy.git
drwxr-xr-x - alafroscia 20 Jan 11:53 studio-packaging-service.git
drwxr-xr-x - alafroscia 1 Apr 10:02 tailwind-config.git
drwxr-xr-x - alafroscia 14 Apr 15:18 template-lint-plugin.git
drwxr-xr-x - alafroscia 19 Nov 2020 terraform.git
Is this intentional? At least in my case, this was unexpected. I ran the dry run first but I guess didn't notice that .git
was going to be added to all of these.
There's a somewhat lesser known feature in git where you can use aliases for your remote URLs by putting something like
[url "ssh://[email protected]/"]
insteadOf = "gh:"
And then use git clone gh:Byron/gitoxide
instead of typing out the entire URL.
As of 0.9, gitoxide fails to parse those remotes:
~/scratch
โฏ git clone gh:Byron/gitoxide
Cloning into 'gitoxide'...
remote: Enumerating objects: 24757, done.
remote: Counting objects: 100% (5485/5485), done.
remote: Compressing objects: 100% (2069/2069), done.
remote: Total 24757 (delta 3133), reused 5447 (delta 3101), pack-reused 19272
Receiving objects: 100% (24757/24757), 5.50 MiB | 7.97 MiB/s, done.
Resolving deltas: 100% (14407/14407), done.
~/scratch
โฏ gix t organize
11:17:42 organize Error when handling directory "./gitoxide": Remote URLs must have host names: file://gh:Byron/gitoxide
Error: Failed to handle 1 repositories
I'm not sure if gitoxide can parse .gitconfig
files right now, so this might be tricky to handle correctly...
git is somewhat robust when killed in the middle. libgit2 seems somewhat less so. I'd love to see gitoxide be rock-solid on that front.
When doing things like clone, fetch, and similar, it's important to deal with network failure, ctrl-c, systems being powered off, and other sources of abrupt interruption. It's important that this never leave the repository in an inconsistent state (e.g. references to objects that don't exist, corrupted packs or indexes that git will choke on, and similar). This includes things like putting things in temporary locations and atomic-renaming them in place, or ensuring that objects are put in place before references to them.
Hi, I wish to contribute to a feature in this repo, and pack-send sounds straightforward. Should I try to work on this if it's already not taken?
git verify-pack
I remembered timings on a cold cache that indicated something around 5:50min for git to run a verify pack on the linux kernel pack. However, turns out that if the environment is a little more controlled, git is still considerably faster than us despite using an LRU cache and despite using multiple cores quite efficiently.
Git uses a streaming pack approach which is optimized to apply objects inversely. It works by
We work using a memory mapped file which is optimized for random access, but won't be very fast for this kind of workload.
Wait until we have implemented a streaming pack as well and try again, having the same algorithmical benefits possibly faired with more efficient memory handling.
Git for some reason limits the application to 3 threads, even though we do benefit from having more threads so could be faster just because of this.
The streaming (indexing) phase of reading a pack can be parallelised in case we have a pack on disk, and it should be easy to implement if the index datastructure itself is threadsafe (but might not be worth the complexity or memory overhead, let's see).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.