git stdout: GitLab: The project you were looking for could not be found

[jedrekdomanski]

Important: GitHub issues are for feature requests or bug reports. The Capistrano team recommends you use Stack Overflow for general questions. For more details, please see our contribution policy.

---

Steps to reproduce

Deploy an application to a remote host, change app repo url in Git, update repo url in Capistrano config, try to deploy the app. It won't deploy. You'll get an error (see below). To resovle the problem you need to ssh onto the remote host and remove repo folder, then re-deploy. It looks like Capistrano relies on repo url from repo folder. I am posting this thread as a suggestion for improvement, because I think that it would be nice if we didn't have to remove repo folder manually, secondly, I struggled to figured it out for an hour or two so we could save other people's lives trying to solve similar problem.

Expected behavior

App should deploy

Actual behavior

git exit status: 128 (SSHKit::Runner::ExecuteError)
git stdout: GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.

System configuration

Please link to the output of cap <stage> doctor in a GitHub Gist.
https://gist.github.com/jedrekdomanski/64e7ee0f4bb57c9631cbdbb60e3ca792

Thanks for helping improve Capistrano!

[mattbrictson]

Thanks for the detailed report!

Capistrano should automatically update the remote git URL if it changes. This feature was added in #1826 . Do you have any idea why git remote set-url origin <url> is not working in your case? Is there evidence in the log output that the set-url is getting called?

[jedrekdomanski]

Let me give you a bit more details about my case. The code was pushed to GitLab but capistrano config was still referencing the old repo (this a takeover project). The code was deployed. Then, I changed the repo url because I realized that my predecessors forgot to update capistrano repo url which should be pointing to the GitLab repo, not their own gitlab instance. I updated the repo url in capistrano and tried to deploy but it errored out. I looked at repo folder on the remote host and the repo url was from before my update. I do not have the logs anymore, sorry. This interesting, because, as you mentioned, the change has been already implemented but something is still not working. The case described here #1826 is the same as mine so ...very interesting...

[mattbrictson]

I just tested this and it seems that capistrano is doing the right thing. I did a cap production deploy, then updated deploy.rb and changed the :git_url, then did another cap production deploy. It succeeded and after the fact I checked the repo folder on the remote host and repo/config contained the new URL.

I did see this warning in the git output after changing the URL:

warning: no common commits

But otherwise it seemed to work fine.

Perhaps in your case it had something to do with cached credentials? Was the previous URL an https one?

I'll leave this ticket open in case other people run into a similar problem and have additional information to report.

[jedrekdomanski]

No, definitely nothing related to cached credentials or https. Thanks for checking!

[greenclaw]

Hey,
I also encountered this problem in the past.
The original issue I've posted on stackoverflow: issue link.
As noted in issue, the problem solved by removing content of repo folder and scripts in /tmp folder.

[mattbrictson]

Full disclosure here: I am officially stumped on this one. Sorry I can't be of help!

I am particularly confused as to why deleting scripts in /tmp would have any effect. You can see that capistrano replaces the /tmp script every time it is run, so any existing script should not make a difference:

capistrano/lib/capistrano/scm/tasks/git.rake

Lines 8 to 10 in f8f081f

	execute :mkdir, "-p", File.dirname(fetch(:git_wrapper_path)).shellescape
	upload! StringIO.new("#!/bin/sh -e\nexec /usr/bin/ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no \"$@\"\n"), fetch(:git_wrapper_path)
	execute :chmod, "700", fetch(:git_wrapper_path).shellescape

Furthermore, the git:check task does not use the repo folder at all. As you can verify in the output, it is a git ls-remote command. It does not cd into the repo folder, nor does it reference the repo folder in any way.

capistrano/lib/capistrano/scm/git.rb

Line 38 in f8f081f

git :'ls-remote', git_repo_url, "HEAD"

I do not understand how deleting repo or /tmp would have any effect on this task.

Perhaps someone with more expertise in git or GitLab specifically can help me here?

[bfcoder]

For me, it doesn't seem to have anything to do with the remote. On my Mac machine, I can deploy without problems. But on my Linux (Ubuntu) machine, I am all of a sudden getting this error.

I did recently install ClamAV, so I thought maybe that was interfering. But turning it completely off, I still get this error. ¯_(ツ)_/¯

[bfcoder]

Hello I.T., Did you try turning it off and on again?

Just did, and now I am able to deploy from my Linux machine. There must have been a bad git ssh cache somehow on my machine.

[mattbrictson]

For me, it doesn't seem to have anything to do with the remote. On my Mac machine, I can deploy without problems. But on my Linux (Ubuntu) machine, I am all of a sudden getting this error.

Hmm, in that case this could be related to the private key you are using on your local machine for SSH. If that key is not associated with the correct GitLab account, I can see how that might give you a "The project you were looking for could not be found". Maybe?

[leehambley]

Definitely plausible @mattbrictson I was thinking the same, since if you have multiple keys loaded into your agent (whether you know it or not, check with $ ssh-add -l) then the likes of GitHub and GitLab will let in the first key that authenticates, but it might not be a key with access to the project you want.

FWIW they'll also only try a few keys, I think up to 3..5 before giving up. Working as a contractor I have to be super careful which keys are in my agent else I can run into this kind of thing (not necessarily just with Capsitrano)

[bfcoder]

My ssh key was loaded. I was able to run the git ls-remote --heads [email protected]:project/project.git manually just fine. And I only had 1 key loaded at the time, so it wasn't a problem with running through the first 5 keys keys loaded and then giving up.

[bfcoder]

I just replicated this.

Capistrano 3.6.1
Ubuntu 18.04
git 2.17.1

I had 2 ssh keys loaded, neither were the correct key for the server I was attempting to deploy to. Received:

#<Thread:0x0000562c62bdca10@/home/<username>/.asdf/installs/ruby/2.5.7/lib/ruby/gems/2.5.0/gems/sshkit-1.17.0/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
  1: from /home/<username>/.asdf/installs/ruby/2.5.7/lib/ruby/gems/2.5.0/gems/sshkit-1.17.0/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/home/<username>/.asdf/installs/ruby/2.5.7/lib/ruby/gems/2.5.0/gems/sshkit-1.17.0/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as ubuntu@<dns>.compute.amazonaws.com: git exit status: 128 (SSHKit::Runner::ExecuteError)
git stdout: Nothing written
git stderr: remote: 
remote: ========================================================================
remote: 
remote: The project you were looking for could not be found.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
(Backtrace restricted to imported tasks)
cap aborted!

After ssh-add the correct key to login to the machine, I still get this error. Which is odd since it isn't the key that is needed to authenticate with gitlab. That key was loaded(albeit it was loaded manually after cancelling the passphrase prompt, and then manually running ssh-add, earlier in the day when trying to git pull). I also got this error message if I did load the server key, but not the gitlab key, cancelled out of the passphrase prompt, manually ssh-add the gitlab key, then try to deploy. It holds onto that original denial.

If I reboot my machine (I'm sure there is a service I could restart too), ssh-add both my gitlab key, and my server key, I can then cap deploy just fine.

And maybe this is actually a problem with git and it is just manifesting here while using cap. I don't really know.

[leehambley]

Thanks for commenting back @bfcoder

This is indeed a problem with Git. You can modify the git+ssh wrapper script that Capistrano exports to expose the following env vars GIT_CURL_VERBOSE=1 GIT_SSH_COMMAND="ssh -v" GIT_TRACE=1 (or export them in your shell) and you can see exactly what Git is doing under the hood.

Git is written as a series of tiny tools that get piped together, so on the simplest level there's going to be a call to git ls-remote <the remote> which is expected to show what commit the remote is at, followed by some calls to something like git receive-pack <the remote> ....0xffff -- git receive pack is usually "wired" with a pipe to something that reads from SSH.

I'm recounting this a lot from memory, so your results may vary, a bit. What may be happening is that with the "wrong" keys loaded into the agent is that Git is using up some of it's retries using the wrong keys, and proceeding on with no "retries" left on the counter, so a single failure (even if you should be allowed to try n keys z times) is enough to make it bail out.

You might also experiment with GIT_SSH_COMMAND="ssh -vvv" rather than just one -v.

Regarding clearing/rebooting, an ssh-add -D ( Deletes all identities from the agent.) should be sufficient to get to a "clean" state.

When using Capistrano on the server-side, we write out a git wrapper script (which is used in place of which ssh, which allows us to pass options). It's not uncommon to do this in environments where you want to set SSH options to specify which identity file should be used, to skip, or pre-approve SSH host keys, to change logging levels, etc, etc.