svn externals replacement

[andry81]

In svn you can reference a subdirectory in an external:

/ project-X
|
+---/ _externals
|   |
|   +---/ project-A
|       |
|       +---/ DirA1
|       +---/ DirA2
|       +---/ DirA3
|
+---/ DirX1
+---/ DirX2
+---/ DirX3
+---/ DirX4
+---/ DirX5

/ project-A
|
+---/ _externals
|   |
|   +---/ project-X
|       |
|       +---/ DirX3
|       +---/ DirX4
|       +---/ DirX5
|
+---/ DirA1
+---/ DirA2
+---/ DirA3
+---/ DirA4
+---/ DirA5

/project-X/_externals

svn-url-to-project-A project-A/DirA1
svn-url-to-project-A project-A/DirA2
svn-url-to-project-A project-A/DirA3

/project-A/_externals

svn-url-to-project-X project-X/DirX3
svn-url-to-project-X project-X/DirX4
svn-url-to-project-X project-X/DirX5

.gitlinks in project-X:

project-A _externals/project-X git-url-to-project-X master

.gitlinks in project-A:

project-X _externals/project-A git-url-to-project-A master

But how to checkout not all directories in each external (sparse checkout)?

[andry81]

Seems w/o sparse checkout support (https://git-scm.com/docs/git-sparse-checkout) or w/o a custom method of checkout w/o builtin recursion, there is currently no way to use recursed submodules (gil update with links creation). Without links creation there is another issues with gil pull.

#7 #8 #9 #10 #11

Conclusion

• With links creation (Administrative mode) git commands does fall into infinite recursion: error: cannot spawn git: Filename too long #9
• Without links creation (non Administrative mode) gil pull and gil status treats working copy as changed and git pull does not work w/o externals merge: error: you need to resolve your current index first #10
• Recursive submodules involves infinite unstaged changes: infinite unstaged changes because of recursive submodules #11

Side issues

• GitExtensions reports in the Commit dialog in the Stage area the staged but not commitable (can not be moved out of the stage) submodules.
• GitExtensions Repo Browser does not report of existing submodules and you just can not see submodules branch tip and so the submodules actual commit hashes:
  nested submodules detection w/o .gitmodules file gitextensions/gitextensions#10642
  nested submodules detection w/o .gitmodules file gitextensions/gitextensions#10644
• GitExtensions other issues:
  Unsupported file: GitExtUtils.ExternalOperationException: fatal: git cat-file ...: bad file gitextensions/gitextensions#10643 (comment)

But issues with the GitExtensions can be ignored.

[chronoxor]

Can you prepare a minimal public repository structure that reproduces all the issues? So I can check them on my side

[andry81]

Can you prepare a minimal public repository structure that reproduces all the issues? So I can check them on my side

First simple tests: #11 (comment)

[chronoxor]

Please try https://github.com/chronoxor/gil-test1-root

git clone https://github.com/chronoxor/gil-test1-root.git
cd gil-test1-root
gil update
gil status

[andry81]

Please try chronoxor/gil-test1-root

git clone https://github.com/chronoxor/gil-test1-root.git
cd gil-test1-root
gil update
gil status

That is a different project. In mine case A has B as a dependency, when B has A. There is no 3d one. And this is not recursive in SVN, because were taken sudirectories, not the externals. Now I want to port SVN into GIT.

And that is wrong, _externals must exist, there is no sense to ignore it.

A -> B

Externals of A must exist, when externals of B in A may not. The same for B -> A.

[chronoxor]

You can have tree-based projects dependency explicitly:

• A
• B->A
• C->A
  C->B->A

If you want to have cycles projects dependencies, you have to place them under some root project and put root .gitlinks in it:

• A->B
• B->A
• C->A->B->A->....
  C->B->A->B->....
• ROOT->A
  ROOT->B
  ROOT->C

That's how gil tool works now

[andry81]

• ROOT->A
  ROOT->B
  ROOT->C

You have used _externals/* in .gitignore in case of https://github.com/chronoxor/gil-test1-root.git. I didn't want to ignore it.

[chronoxor]

Linked externals should be ignored if you commit in the current repository with git commit command. However you could commit with gil commit command - as the result all changes in the current repository and all linked ones will be committed by checking dependencies in .gitlinks file.

[chronoxor]

The following commands are working for the whole linked structure:

• gil pull [params]
• gil push [params]
• gil commit [params]
• gil status [params]

[chronoxor]

My common workflow is to make changes in several linked projects, then go to the root and make global commit with

gil commit -a -m "Commit message"
gil push

[andry81]

My common workflow is to make changes in several linked projects, then go to the root and make global commit with

gil commit -a -m "Commit message"
gil push

I don't like this method of committing. I use GitExtensions to make a commit and push. This is a convenient way to commit anything in git. First of all you see all unstaged items in the GUI and can run commands on them like the reset directly on selected items in unstaged area. At second, you see the difference file for each item from the changeset. At third, you have access to the history of commit messages and can amend with last commit message restore. Plus all other options and advantages. With the gil commit I don't have all of this.

[andry81]

Linked externals should be ignored if you commit in the current repository with git commit command.

How you suppose to work like this? I have to maintain 2 working copies at the same time for each project? I have to copy all the local changes between 2 projects with and w/o externals before commit them. This is quite a ridiculous.

By the way, the sparse checkout automatically does ignore filtered paths (https://git-scm.com/docs/git-sparse-checkout):

This command is used to create sparse checkouts, which change the working tree from having all tracked files present to only having a subset of those files.
...
When in a sparse-checkout, other Git commands behave a bit differently. For example, switching branches will not update paths outside the sparse-checkout directories/patterns, and git commit -a will not record paths outside the sparse-checkout directories/patterns as deleted.
...
Since submodules may have unpushed changes or untracked files, removing them could result in data loss. Thus, changing sparse inclusion/exclusion rules will not cause an already checked out submodule to be removed from the working copy.

So if you would implement the sparse checkout then, it is better to convert .gitlinks into yaml structure like vcstool has, to be able to avoid submodules to be checked out by default.

Here is example of the sparse checkout in the vcstool:

dirk-thomas/vcstool#218
https://github.com/dirk-thomas/vcstool/pull/219/files

[chronoxor]

As your projects are linked with a symbolic link, you have only one place to update and commit (e.g. in A, or B->A which points to the same place as A).

[chronoxor]

I also use TortoiseGit sometimes to review & commit, but you have to do it separately for each project you change (e.g. A and B).

[andry81]

As your projects are linked with a symbolic link, you have only one place to update and commit (e.g. in A, or B->A which points to the same place as A).

I want a single place to commit from and I want to observe the dependencies at the same time. Better to back off the recursion at all so the links won't raise the problem.

I also use TortoiseGit sometimes to review & commit, but you have to do it separately for each project you change (e.g. A and B).

That was a bit bad idea because there is many git tools and IDEs which does call to git directly, so the script does interfere with them and interrupts the usage. I have to switch to GUI/tools and back to the script to make a simple commit or pull.
Of cause you can use the script once like to clone or to link, but deal with it each time you want to commit or pull is redundant. This just mine opinion.

PS: I've found another closest solution and updated the comment: #6 (comment)

[andry81]

Just for instance. I have tried to promote one of the GitExt side issues and this is what they said:

gitextensions/gitextensions#10642 (comment)

You have a funky state and I doubt is supported as valid git state.

So just because git is detecting nested repo and helping you out in summary call, it is not a submodule. The modules file is required to actually work git submodules.

[andry81]

Just for instance. Have sent questions into git folks mailing list:
https://lore.kernel.org/git/[email protected]/