Sample Python usage?

[gerroon]

Hi

I want to write a simple Python code to submit to my remote server. Does anyone have a basic working code? The API pages do not seem to have any real example beside this, but this one does not show to connect to the server for real.

import os
DATA_DIR = '~/archivebox/data'
os.chdir(DATA_DIR)

# you must import and setup django first to establish a DB connection
from archivebox.config import setup_django
setup_django()

# then you can import all the main functions
from archivebox.main import add, remove, server

add('https://example.com', index_only=True, out_dir=DATA_DIR)
remove(...)
server(...)
...

[pirate]

There is no remote Python API currently, only a local one.

You could write a small flask server like this to handle remote requests and pass them to the local ArchiveBox API as you see fit:

server.py:

import os
from flask import Flask, request

app = Flask(__name__)

os.chdir('/path/to/archivebox/data')
from archivebox.config import setup_django
setup_django()

from archivebox.main import add

@app.route('/add', methods=['POST'])
def add_endpoint():
    return add(**request.get_json())

if __name__ == '__main__':
    app.run(debug=True)

client.py:

import requests

# URL of the Flask server's /add endpoint
url = 'http://127.0.0.1:5000/add'

print(requests.post(url, json={
    'urls': 'https://example.com\nhttps://example.com/2nd/url',
    'tag': 'tag1,tag2,tag3',
    'depth': 1,
    'extract': 'singlefile,screenshot,dom,readability,media',
}).text)

[gerroon]

Thank you, that could probably work for me. Do you happen to have a sample for the Rest API? I could not find an example for it. I can't run local commands since my server is remote and I need to process it locally and pass it to the server. The rest API might make it easier since I wont need to run extra stuff on the server but if not I will take your Python example.

[gerroon]

Btw from archivebox.config import setup_django throws AttributeError: module 'os' has no attribute 'getuid' on Windows. As far as I understand this is because Windows does not have getuid

[pirate]

Windows is not really supported at all outside Docker.

A full REST API is still a work-in-progress and is not released yet (follow here for progress #496). Right now you can reuse the endpoints used by the Admin UI, which is what the ArchiveBox chrome extension does. (use browser devtools to see how the browser makes a request to the /add page for example)

[gerroon]

For the record "extract" does not seem to exist but I found out that it is actually "extractors"

I also have a big issue with the client. It tried to grab 324 URLs when I passed slightly modified code as a test :(

print(
    requests.post(
        url,
        json={
            "urls": "https://stackoverflow.com/questions/55116381/modulenotfounderror-no-module-named-flask#55116933",
            "tag": "tag1,tag2,tag3",
            "depth": 1,
            "extractors": "singlefile,screenshot,dom,readability,media",
        },
    ).text
)

[pirate]

Ah yeah sorry it's called --extract as a CLI flag but extractors in the Python API.

Also 324 URLs is expected with that code, as you have depth=1. (which means it will recursively archive all the URLs one hop out from the first URL, useful if you're passing a URL to an RSS feed or page containing many outlinks you want to save)

Just set depth=0 if you only want to archive the URL provided without crawling one hop out.