New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rename archivebox oneshot
-> archivebox snapshot
& archivebox extract
, expose atomic subtasks for future job queue system
#1289
Comments
pirate
changed the title
Rename oneshot command to "snapshot" and add "extract" command to break down take and make them runnable independently of a full "add" job
Rename oneshot command to "snapshot" and add "extract" command to break down tasks and make them runnable independently of a full "add" job
Dec 15, 2023
pirate
changed the title
Rename oneshot command to "snapshot" and add "extract" command to break down tasks and make them runnable independently of a full "add" job
Rename Dec 18, 2023
oneshot
command to snapshot
, add new extract
command to expose add
subtasks for future job queue system
pirate
changed the title
Rename
Rename Dec 18, 2023
oneshot
command to snapshot
, add new extract
command to expose add
subtasks for future job queue systemarchivebox oneshot
-> archivebox snapshot
, add new archivebox extract
command to break down add
subtasks for future job queue system
pirate
changed the title
Rename
Rename Dec 18, 2023
archivebox oneshot
-> archivebox snapshot
, add new archivebox extract
command to break down add
subtasks for future job queue systemarchivebox oneshot
-> archivebox snapshot
& archivebox extract
, expose atomic subtasks for future job queue system
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I want each thing that ArchiveBox does internally to be a subcommand exposed directly to users or callable by the Huey job queue system.
It will also have the side effect of making the process tree nicely represent what ArchiveBox is doing internally, and allow users to kill stuck subtasks independently without stopping the entire "add" import.
The old
oneshot
will be renamed and joined by a new command to run a single extractor method:archivebox snapshot
Can be run to snapshot an individual URL into the current directory (runs all extractors by default).
This works the same way as
oneshot
does now, and I'll aliasoneshot
to the new command so we don't break backwards compatibility.archivebox extract
This runs an individual extractor method and outputs into the current directory.
After the refactor,
archivebox add
will work by internally enqueuing a job that runsarchivebox snapshot ...
for each imported URL.The snapshot job then in turn enqueues a job for each extractor needed on that URL.
Each extractor job then runs
archivebox extract --method=...
internally to write the output into the final archive directory.The text was updated successfully, but these errors were encountered: