Adopting Scoop-like manifests (for manifest auto updates)

[camjac251]

I should've switched to this sooner. Managing plugins and scripts in AVS is a pain if there's not updated info available on the wiki or no info found for those hidden 64-bit ports.

I've been using a regular package manager on Windows for a little while and it has some great features that might benefit AVSRepo.
The app is called scoop and uses buckets and app manifests to install apps. Maybe buckets aren't needed with AVSRepo (unless you wanted to allow the creators of plugins to manage their own buckets, like StainlessS, Groucho, and real.finder)
Buckets can be separate GitHub repositories ran by different users, and then added into scoop to crawl and build a local cache of a repo of all apps. It doesn't have dependencies as everything is isolated/portable so that would be different to AVSrepo.

The extra thing that if I understand correctly that's cool about Scoop's manifest system is its autoupdate function. This can be used with AppVeyor and the powershell scripts, to check the version available for apps and update the manifest automatically to the newer version with a GitHub commit. And there's various methods they use to check the versions.

This might not work as well for AVSRepo if say the majority of plugins are available as mediafire/sendspace/doom9 forum downloads if the link is paywalled/registrationwalled or dead.
What if AppVeyor could be used to create all of the manifests for downloading to a server as a mirror to try if the default url fails.
And then the mirror could be built on a CDN that would host all versions available of scripts/plugins for easy downloads (I'd help fund that) and having the manifests pull from there. Unless that's too much work of course 😄

The simplified idea would be just autoupdating the manifests using similar methods to scoop from GitHub releases/Direct Downloads/and various file hosting sites.

Here's some links on the process with scoop
lukesampson/scoop/wiki/App-Manifests
lukesampson/scoop/wiki/App-Manifest-Autoupdate

[theChaosCoder]

I will have a look at it.
But it sounds very similar to what avsrepo already does. With vsrupdate I can also auto update all plugins (currently only github repos with releaases are supported).
And like you said, it won't solve the problem with mediafire links.

Vapoursynth does not really have this problem as most plugin / scripts are already hosted on github.

[camjac251]

Oh I didn't realize that vsrepo auto updates. That's cool.

Here's a few more examples to various manifests of apps, check out the autoupdate on these.
https://github.com/lukesampson/scoop-extras/blob/master/bucket/ffqueue.json
https://github.com/lukesampson/scoop-extras/blob/master/bucket/madvr.json
https://github.com/engnut/scoop-bucket/blob/master/bucket/tsmuxer.json
https://github.com/ScoopInstaller/Main/blob/master/bucket/eac3to.json
https://github.com/lukesampson/scoop-extras/blob/master/bucket/musicbee.json

MusicBee one seems to work with mediainfo, just need to pull the version somewhere else or maybe combo it with another regex. Maybe pulling from Doom9 forums for versions.

eac3to for example doesn't use any hash checking, instead eac3to -v which is cool for exes but won't work for dlls. But it could be very flexible if it's all in Powershell I think.

I'll try to find other various methods that people came up with for checking and downloading. I'll see if there's a forum download method, different website configs, file hosting sites.

[theChaosCoder]

It seems scoop does have support for dependencies https://github.com/lukesampson/scoop/wiki/Dependencies

[camjac251]

That makes thing easier. There are some things that would need to be removed if this could work. Scoop uses symlinks to link a current folder to the version installed. That wouldn't work well with AVS. And also managing different CPU instruction DLLs or AVS2.5 AVS2.6 AVS+ specific DLLs. And then you have modded versions of scripts of the original to support AVS+, when maybe the original works better in AVS 2.6

[theChaosCoder]

Managing different cpu instructions + avs+, avs2.6, etc. could be also done with avsrepo. Just create a new json file with a different name xyz_xp.json and filter by the xp version instead of the current one. Dependencies will also work.

But who will do all the work?

[camjac251]

I planned on testing scoop soon to adopt it for this. Hopefully it won't be too difficult.

[theChaosCoder]

XP user don't have powershell5 installed :D
Minimum support is Windows 7 SP1+

[camjac251]

Maybe the whole powershell idea might be too much work even if it was supported for everyone. I wish I knew python.

Would it be too much work to adopt specifically the checkver/autoupdate functions? Storing an autoupdate field in the plugin manifests with a url and regex for filename-version and checksum verification with last version checksum stored in the manifest.
https://github.com/lukesampson/scoop/blob/472cd58a8cdd4e8b0adceb01c4d2b200640d3711/lib/core.ps1#L896

https://github.com/lukesampson/scoop/blob/master/bin/checkver.ps1

[theChaosCoder]

I think avsrepo would only need a function to check simple html websites/links. Maybe we could use scrapy for that: https://scrapy.org/

Another downside with scoop is that it hashes the whole zip archive. Vsrepo hashes single files and with that we can detect the "installed" plugin version very easily, even if the user replaced the file.

Line 922 with the mediafire url looks very interessting...

[camjac251]

Scrapy does look interesting.

I tried to find any vbulletin attachment scraping on GitHub, nothing really for scrapy for attachments but I did find these. The first 3 seemed the most relevant. Some might be duplicates, I didn't look too closely. They use regex
https://github.com/cove9988/forexfactorythread/blob/1d17f1e03316757b8802779761aa9748cb5941e1/getforexfactorythread.py
https://github.com/N3X15/Watchdog/blob/e01d76d565d3ee5dd907389fc92e3f1e13fcd422/watchdog/repo/amattachment.py
https://github.com/TundraFizz/Image-Retriever/blob/27a478ff66da561bbc04ad7b0d1780dd1f5e698a/image-retriever.py
https://github.com/sveinse/vbgrab/blob/11b2a41ce8eebc3b4c699578b5825a1655ec3ee6/vbgrab.py
https://github.com/Tarnacop/crypto-firmware-binaries/blob/39188807d06b70843125cbce8b40677108e7c01c/data-collection-script/digitalkaos_module.py
https://github.com/seanchen/plonexp/blob/6c704f73c47babe31ccf69c5961a354384a7d2ae/sandbox/iscorpio.sandbox/iscorpio/sandbox/readwrite.py
https://github.com/VovaBobyr/PythonLearning/blob/c3c5ef3cccbd37273518a409c0624041a17f9b9f/PicLoad/PicLoad1Click.py

[camjac251]

I forgot to include this one. Might be https://github.com/ArchiveTeam/pokecommunity-grab/blob/a248f3cd2aade668b5239e56c0f1ba07c61c4b6c/pipeline.py

I hope these are useful. If not :/

[theChaosCoder]

I think I start slow by using this working mediafire downloader https://github.com/gilcu3/mediafire-dl
It seems Groucho is also hosting his plugins there now.

But I need to mirror the files otherwise there will be often 404 not found errors when a new version comes out.

[camjac251]

You might not need to mirror them. With each version released, what if the checksum of the files it will keep are saved in the manifest and compared to the new downloads made? A mirror would be great to have since it could always be reliable and fast but will it take a lot to make?

[theChaosCoder]

The Mediafire file link will change, that is why I need a mirror.
I mean the old file will be deleted and a new one uploaded. Downloads will fail until the manifest is updated.

[camjac251]

Oh I see what you mean. Would you cache everything, even current versions? Since hosts go down quite often. I can help setup a storage bucket with a CDN endpoint when the time comes so it's fast for everyone if you want (thinking backblaze b2 and cloudflare (or bunnycdn).

[theChaosCoder]

I would cache the whole mediafire folder (new and old files), so I don't have to implement mediafire stuff on the client side. The files are between 2kb and 200kb, I don't think we need a fancy cdn.