AJAX

AJAX is Infy's most complex append mode and it comes in two very different versions, to make things even more complex. However, this is the append mode you'll want to use when you're dealing with Complex Websites. There's a lot going on under the hood with this mode, so it really warrants its own section.

Before reading this section, it's really recommended to first read and understand the following sections:

• How to tell you're on an AJAX site - You'll definitely want to know if you're on an AJAX site (or SPA) to begin with
• Append Element Mode - AJAX uses the same concept of there being a "Page Element"
• Paths - You'll need to construct the correct paths to the Click Element and Page Element necessary for AJAX

There's (at least) three different ways to handle AJAX sites:

1. AJAX Iframe - Mirror the page (e.g. to an iframe or offscreen document) and then import the elements
2. AJAX Native - Disable the page from scrolling and removing elements each time the button is clicked
3. AJAX Request - Make the same AJAX request the page is making to the server and then append the response (elements)

To discuss AJAX, we'll use Pixiv as an example website.

Here's an Example URL to follow along with:
https://www.pixiv.net/tags/オリジナル/illustrations?mode=safe

Click Element

AJAX is only compatible with the "Click Element" action; it's assumed we are clicking a button and having the page append the elements itself. There are no links (though they may look like it or be provided by the site as a fallback), so do not use the more conventional "Next Link" action for this append mode (you'll get an error warning you to select the right action in case you forget).

You'll need to provide a path to the Click Element as you would normally for the next link. You have multiple options such as Auto Detect, the Element Picker, and you can manually enter or edit in a path yourself. Note that the Click Element is always highlighted on the page each time the path is edited, so you can visually see if it's picking the right one or not before you click ACCEPT (you may need to scroll the window so that Infy's UI Window isn't covering it up).

AJAX Iframe

This is the default AJAX mode because it's the most stable and safest. This appends a hidden iframe on the page that continually clicks the next button and imports the elements to the page. It also scrolls the iframe very carefully to ensure that all images/media have lazily loaded before being imported to the page. You can observe what's really going on by going to Options > Extra and checking the Enable Debug Mode checkbox, then visiting the Example URL. You'll notice there's another page on the page you're on (this is the iframe) and it's doing a lot of strange things like scrolling itself continuously.

AJAX Iframe Example (Pixiv)

{
  "action": "click",
  "append": "ajax",
  "clickElement": "//nav/button[@aria-current='true']/following-sibling::a[not(@hidden)]",
  "loadElement": "//section//div/ul//figure",
  "pageElement": "//section/div/ul | //section/div/div/ul | //section/div/div/div/ul",
  "url": "^https://www\\.pixiv\\.net/"
}

Tip: You can copy the example above and add it to your Saves to test it out.

Load Element Path

This path is optional in most cases. However, if you notice that some of the images didn't load, you'll want to try using it. The Load Element Path is the path to the elements that haven't yet loaded. Inspect the page and try to see if there are any differences between the images that haven't loaded and the ones that have loaded. For example, you may notice that the images that haven't loaded have an attribute like lazy-load or their parent container is a different tag altogether.

On Pixiv, we can observe that a figure tag exists if the image hasn't yet loaded, so we can write our Load Element Path as follows (XPath):

//section//div/ul//figure

This will force Infy to continue scrolling the AJAX Iframe until there are no more figure tags before appending the next page's images.

AJAX Native

This is Infy's most advanced append mode, but it's so experimental, it's still in proof of concept status. It actually manipulates the website itself into providing infinite scrolling - natively.

To better understand how this append mode works, we need to quickly go over how AJAX sites work.

Typically, the workflow goes like this:

1. The user clicks on the Next button
2. The site then removes the current elements on the page and replaces them with the next page's elements
3. The site then scrolls the user back to the top

Native uses a unique approach by disabling the site from doing 2 and 3; specifically, it tries to prevent the site from removing any elements and from scrolling you back to the top (while still allowing the site to append the next page's elements each time the next button is clicked).

AJAX Native Example (Pixiv)

{
  "action": "click",
  "append": "ajax",
  "ajaxMode": "native",
  "clickElement": "//nav/button[@aria-current='true']/following-sibling::a[not(@hidden)]",
  "pageElement": "html[@lang='ja']//ul[@class='sc-l7cibp-1 krFoBL']/*|html[not(@lang='ja')]//div[@class='sc-l7cibp-0 juyBTC']/*",
  "removeElement": "//li[@class='sc-l7cibp-2 gpVAva'][not(.//div[@type='illust'])]|//div[@class='sc-1rgyha8-0 leTtkh']|(//*[@class='sc-l7cibp-3 gCRmsl'])[position()<last()]",
  "url": "^https://www\\.pixiv\\.net/"
}

Tip: You can copy the example above and add it to your Saves to test it out.

Note: AJAX Native best works with the Japanese version of Pixiv. You can click the language setting on the bottom of the page or by changing your language settings (assuming you have a Pixiv account and are logged into it).

Remove Element Path

This path is optional in most cases. However, you may notice some extra nodes that the website appends. These placeholder nodes, or what I like to call "ghost" nodes, can be removed by Infy. Normally, what the site is doing is appending these placeholder nodes temporarily while the real nodes are being fetched, and then removes them afterwards. Because Infy disables the site from removing elements, the nodes will remain on the page. You can therefore provide a path to the nodes. Just inspect them using your browser's DevTools and look for anything unique about them that can allow you to write a path targeting just them.

On Pixiv using the AJAX Native append mode, we can see some ghost nodes being appended by the site.

(To be continued as time allows)

AJAX Request

Important: This AJAX mode is not currently implemented in Infy Scroll (but might be in the future). This is only documented here for theoretical purposes so you can skip this section for now.

There is a theoretical third way to handle AJAX sites. Using your browser's DevTools, under the Network tab, you can see the specific XHR/Fetch request that is being made to the server each time the button is clicked. Assuming the URL follows a simple pattern, like https://ajax.example.com/p=N, it's possible for Infy to make the same AJAX request directly and then append the response to the page.

Sites can choose to respond with HTML or JSON (usually the latter). This won't work with JSON type responses or responses that require custom script handling unless a custom script was also added into Infy for that specific site.

AJAX Request Example (Pixiv)

For example, using the same Pixiv example we've been following, and by inspecting our browser's DevTools, we can see that this is the specific AJAX request (URL) that is being made to fetch Page 2's elements:

https://www.pixiv.net/ajax/search/illustrations/オリジナル?word=オリジナル&order=date_d&mode=safe&p=2&s_mode=s_tag_full&type=illust_and_ugoira&lang=ja

And we can see that Pixiv's server responds with a JSON object that looks something like this:

{
  "error": false,
  "body": {
    "illust": {
      "data": ["..."]
    }
  }
}