Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error #01: Needed resources are used by other tasks. #1276

Closed
ishaulov opened this issue Apr 18, 2024 · 11 comments · Fixed by #1271
Closed

Error #01: Needed resources are used by other tasks. #1276

ishaulov opened this issue Apr 18, 2024 · 11 comments · Fixed by #1271

Comments

@ishaulov
Copy link

ishaulov commented Apr 18, 2024

Hi! Need help with publishing packages!

Packages are assembled on Jenkins, and then they are further uploaded and published to the repository using the aptly api:

curl -f -X POST -F file=@"libson-1.11.6248_amd64.deb" http://repos.local/deb/api/files/uploaded-files?_async=true -u repouser:repopassword 
curl -f -X POST http://repos.local/deb/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./debrepo?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-20.04?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./mint?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 
curl -f -X PUT http://repos.local/deb/api/publish/:./ubuntu-22.04??_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"gpgkey"}}' -u repouser:repopassword 

The problem is that there may be parallel builds, downloads, and publications at the same time, and I get the following error:
aptly — "Error #1: Needed resources are used by other tasks."
curl — (22) The requested URL returned error: 409 Conflict


{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"info","remote":"172.27.0.4","method":"POST","path":"/api/repos/debrepo/file/uploaded-files/libson-1.11.6248_amd64.deb?_async=true","protocol":"HTTP/1.0","code":"202","latency":"166.954µs","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00"}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./debrepo?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.423243ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-20.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.386ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./mint?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.258301ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

{"level":"info","time":"2024-04-18T09:15:42+05:00","message":"Executing task asynchronously"}

{"level":"warn","remote":"172.27.0.4","method":"PUT","path":"/api/publish/:./ubuntu-22.04?_async=true","protocol":"HTTP/1.0","code":"409","latency":"4.272133ms","agent":"curl/7.58.0","time":"2024-04-18T09:15:42+05:00","message":"Error #01: Needed resources are used by other tasks."}

Have I tried using this version:
#1125 (comment)
and using ?_async=true, but the error persists.

I tried to build aptly from the master branch, errors persist.
What else can I do, or maybe I'm doing something wrong?

@neolynx
Copy link
Member

neolynx commented Apr 18, 2024

Hi, thanks for reporting !

yes, this might be fixed with #1125, which is currently being rebased on master.

the ?_async=true will make the API calls return a task_id. the caller should wait for the task to complete, before issuing a next api call. this can be done with GET /tasks/:id/wait
maybe you could try this ?

@ishaulov
Copy link
Author

Thanks for the reply!

Yes, I can. But let me clarify: do I understand correctly that I need to write a shell handler that will process each curl request, take the ID and put it into GET /tasks/:ID/wait ?

@neolynx
Copy link
Member

neolynx commented Apr 18, 2024

yes, something like this:
curl http://repos.local/api/tasks/1/wait

@ishaulov
Copy link
Author

ishaulov commented May 21, 2024

Hello, @neolynx!

I apologize for the delay in replying, it took a while to check everything out.

I've added /tasks/:ID/wait in my shell script, but I still get this error when building in parallel on Jenkins:

[GIN] 2024/05/21 - 13:41:39 | 200 |   83.949261ms |      172.19.0.2 | GET      "/api/tasks/9/wait"
2024/05/21 13:41:39 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:39 | 202 |    2.403509ms |      172.19.0.2 | PUT      "/api/publish/:./ubuntu-20.04?_async=true"
[GIN] 2024/05/21 - 13:41:39 | 200 |   87.470772ms |      172.19.0.2 | GET      "/api/tasks/10/wait"
[GIN] 2024/05/21 - 13:41:47 | 200 |  310.719834ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 202 |      115.25µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6316_amd64.deb?_async=true"
[GIN] 2024/05/21 - 13:41:47 | 200 |  307.632802ms |      172.19.0.2 | POST     "/api/files/uploaded-files?_async=true"
2024/05/21 13:41:47 Executing task asynchronously
[GIN] 2024/05/21 - 13:41:47 | 409 |     116.276µs |      172.19.0.2 | POST     "/api/repos/libs/file/uploaded-files/libs-generator.87.6317_amd64.deb?_async=true"
Error #01: Needed resources are used by other tasks.

Aptly version 1.5.0+ds1-1+b4 .

This is stdout of shell script with set -x:

+ echo 'Moving deb package to repository...'
Moving deb package to repository...
++ curl -f -X POST 'http://repos.local/deb/api/repos/libs/file/uploaded-files//libs-generator.87.6317_amd64.deb?_async=true' -u repo-api:aGF#EQLr
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    53    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 409

And do I understand correctly that I can't get ID and STATE on this request?

+ echo 'Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb'
Uploading deb package: libsdeb///libs-generator.87.6317_amd64.deb
++ curl -s -f -X POST -F file=@libsdeb///libs-generator.87.6317_amd64.deb 'http://repos.local/deb/api/files/uploaded-files?_async=true'
+ upload_response='["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ '[' 0 -ne 0 ']'
+ echo '["uploaded-files//libs-generator.87.6317_amd64.deb"]'
+ json_pp
[
   "uploaded-files//libs-generator.87.6317_amd64.deb"
]

And is it correct that on GET /tasks/:ID/wait I should get STATE=2? Is there any actual documentation on this function somewhere?

@neolynx
Copy link
Member

neolynx commented Jun 5, 2024

Hi !

I looked in the problem and I think aptly does not support parallel operation, i.e. different clients sending requests at the same time. For this reason in the molior project there is a queue before all aptly operations, making in work with parallel request.
The background API also does not solve the problem, is is merely allowing to do other things while waiting for aptly to complete, the requests still must be sent sequentially.

There are ideas to introduce a mutex to wait until the resources are available, but I am not sure this is safe and requests will be executed in order (imagine creating, deleting and recreating same repo. those should be executed in order as they arrive). So probably aptly should queue requests and process them in order, which whould be a bigger implementation.

In the meanwhile, I there a way to wait for all jenkins tasks to complete, and then do all aptly operations in one go, not using async ?

@neolynx
Copy link
Member

neolynx commented Jun 8, 2024

I implemented a queue in #1271, this should allow to publish from different clients instead of returning 409.

Something like this should work for all commands which support async:

task_id=`curl -f -X PUT http://localhost:3142/api/publish/bookworm_12.5-archs-copy_repos_test_1.0-copy2/stable?_async=true -H 'Content-Type: application/json' --data '{"Signing":{"Batch":true,"GpgKey":"reposign@molior", "PassphraseFile": "/var/lib/aptly/gpg-pass"}}'  | jq .ID`

curl http://repos.local/api/tasks/$task_id/wait

Could you give this a try?

Regarding your questions:

  • I will check if the file upload API also support async
  • yes, STATE=2 is SUCCEEDED (see task/task.go:39)
  • all of this should be documented indeed, if you would like to help with this, PRs to https://github.com/aptly-dev/www.aptly.info are very welcome :)

The question now is, how to deal with non async requests. Maybe they should go through the queue as well and additionally just wait for the task to finish before returning 200 ?

This was referenced Jun 8, 2024
@neolynx
Copy link
Member

neolynx commented Jun 8, 2024

The normal non-async requests are not also queued in #1271, and also file uploads should work concurrently now.

Could you please give it a try?

@ishaulov
Copy link
Author

ishaulov commented Sep 14, 2024

@neolynx Hi! Sorry for the wait and thank you so much for fix! Builds have been falling less frequently.

I checked parallel builds with aptly version: 1.5.0+207+g372ce3c4. Error Needed resources are used by other tasks is no longer present

I keep getting an error when doing parallel builds on Jenkins with new error:
Error #01: open /var/www/html/devrepo/deb/upload/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb: no such file or directory

Maybe, I must open new issue?

Jenkins log from the broken job:

13:29:29 + echo '++ curl -f -X POST -F file=@/home/deb//lib_fstyle_4.4.3.7368_amd64.deb '\''http://192.168.100.150/deb/api/files/uploaded-files?_async=true'\'' -u api:pass
13:29:29 ++ json_pp
13:29:29   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
13:29:29                                  Dload  Upload   Total   Spent    Left  Speed
13:29:29 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 11  813M    0     0   11 92.6M      0  95.9M  0:00:08 --:--:--  0:00:08 95.9M
 23  813M    0     0   23  192M      0  97.8M  0:00:08  0:00:01  0:00:07 97.8M
 35  813M    0     0   35  290M      0  98.0M  0:00:08  0:00:02  0:00:06 98.0M
 47  813M    0     0   47  389M      0  98.1M  0:00:08  0:00:03  0:00:05 98.1M
 58  813M    0     0   58  478M      0  96.2M  0:00:08  0:00:04  0:00:04 96.2M
 71  813M    0     0   71  578M      0  96.9M  0:00:08  0:00:05  0:00:03 97.1M
 83  813M    0     0   83  679M      0  97.5M  0:00:08  0:00:06  0:00:02 97.4M
 95  813M    0     0   95  780M      0  98.0M  0:00:08  0:00:07  0:00:01 98.0M
100  813M    0     0  100  813M      0  87.5M  0:00:09  0:00:09 --:--:-- 79.6M
100  813M    0     0  100  813M      0  79.2M  0:00:10  0:00:10 --:--:-- 63.2M
13:29:29 curl: (22) The requested URL returned error: 500 Internal Server Error

aptly json log:

    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/files/uploaded-files?_async=true",
      "protocol": "HTTP/1.0",
      "code": "200",
      "latency": "26.75636ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/repos/devrepo/file/uploaded-files/lib_masterstyle_4.4.3.7369_amd64.deb?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "232.443µs",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./devrepo?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "5.076542ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-20.04?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.331976ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-22?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.326821ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-24?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.303508ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./astra-ce-2.12?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.404118ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "error",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/files/uploaded-files?_async=true",
      "protocol": "HTTP/1.0",
      "code": "500",
      "latency": "1.978734261s",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Error #01: open /var/www/html/devrepo/deb/upload/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb: no such file or directory"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "POST",
      "path": "/api/repos/devrepo/file/uploaded-files/lib_fstyle_4.4.3.7368_amd64.deb?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "229.132µs",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./devrepo?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "21.189441ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-20.04?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "4.111851ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-22?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "3.787162ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./ubuntu-24?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "3.901205ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    },
    {
      "level": "debug",
      "time": "2024-09-14T13:32:04+05:00",
      "message": "Executing task asynchronously"
    },
    {
      "level": "info",
      "remote": "192.168.100.113",
      "method": "PUT",
      "path": "/api/publish/:./astra-ce-2.12?_async=true",
      "protocol": "HTTP/1.0",
      "code": "202",
      "latency": "3.703295ms",
      "agent": "curl/7.58.0",
      "time": "2024-09-14T13:32:04+05:00"
    }

And I want to clarify: when I query curl
curl -f -X POST -F file=@/home/deb/lib_mastertest_amd64.deb 'http://192.168.100.150/deb/api/files/uploaded-files?_async=true -u api:pass

I get json

[
    "uploaded-files/lib_mastertest_amd64.deb"
]

Do I understand correctly that this query does not need to reverse ?_async=true, since I can't get the state?

@neolynx
Copy link
Member

neolynx commented Sep 14, 2024

if you are using parallel uploads / add files, make sure a unique upload directory is used:

folder=`mktemp -u tmp.XXXXXXXXXXXXXXX`
curl -f -X POST -F file=@/home/deb/lib_mastertest_amd64.deb 'http://192.168.100.150/deb/api/files/$folder -u api:pass
...

if the same folder is used from multiple sources, you might get the no such file or directory error message because when adding the uploaded files to a repo, the upload folder is deleted.

the background operation via ?_async=true is only implemented for a few API calls which take a long time, in order to avoid API timeouts. so for uploading a file, the call will block anyway until the file is uploaded.

for the subsequent publishing of the repo, you could use the async operating and poll the corresponsing task until successful (State: 2).

@neolynx neolynx reopened this Sep 14, 2024
@ishaulov
Copy link
Author

@neolynx thanks! 😃

The doc says that the directory will be created if it doesn't exist, and a successfully downloaded package will be deleted. So the directory will be deleted too.... Got it. I'll check it out right now.

That is, in parallel builds, when one has already uploaded the package and is publishing, the file and directory are deleted. At the same time a file from another build was added to this directory...

@ishaulov
Copy link
Author

@neolynx it's worked! Your support and advice have been incredible helpful 😃

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants
@neolynx @ishaulov and others