Using data from Collections in Global Data (with a side of async)

[PersonMeetup]

I feel this might be a more Javascript specific issue, but I'm bringing it up here because it runs up against Eleventy's order of operations. What I'm trying to do is have .addGlobalData wait for a variable to be defined before it returns with data initially generated from .addCollections. I've tried playing with promises/async, but data is consistently only available to templates after a refresh (assuming --serve has been set).

Stepping back, here's a run down of what I'm trying to do and why:

• I've added image extensions as a template language, something which Eleventy is absolutely NOT built for as it breaks the files when coping them over to the dist folder. That's fine to me, because I'm more interested in utilizing those files as data (which I can use to copy them over manually).
• I'm using Directory Data Files to filter out images outside of specially marked directories, whose contents and data I plan on utilizing inside templates. Currently, I have an .addCollections function that groups all the marked images together. That collection goes through a filter to clean the data for templates, but the ideal would be to work with global data instead. That way I don't have to utilize methods like pagination's before callback.
• All this is intended to be packaged into a plugin, as I want others to be able to easily add the functionality I'm working on to their sites!

What follows is a modified snippet of the code that I'm working with, focused on what I believe are the functions that are most relevant to this problem. If you need any further information, please do let me know! Thanks for taking the time to read this incredibly cursed question!

function importComics(collection) {...}  // Returns array
var comicData;
let imageFormats = ["jpg", "png", "gif"];

module.exports = function (eleventyConfig) {
  eleventyConfig.addTemplateFormats(imageFormats);

  eleventyConfig.addExtension(imageFormats, {
    outputFileExtension: "*",
    compile: async (inputContent, inputPath) => {
      return async () => {
        return;
      };
    },
  });

  eleventyConfig.addGlobalData(
    "eleventyComputed.eleventyExcludeFromCollections",
    () => {
      return (data) => {
        if (
          imageFormats.includes(path.extname(data.page.inputPath)) &&
          !path
            .normalize(image.page.filePathStem)
            .split(path.sep)
            .includes(data.comicRoot)
        ) {
          return true;
        }
        return data.eleventyExcludeFromCollections;
      };
    }
  );

  eleventyConfig.addCollection("comics", (collectionApi) => {
    // Instead of returning, we send the data to the Rarebit class for processing
    let collection = collectionApi.getAll().filter((item) => {
      return "comicRoot" in item.data;
    });
    comicData = importComics(collection);
    
    return collection;
  });
  
  eleventyConfig.addGlobalData("comics", () => {
    return new Promise((resolve) => {
      while (rarebitData === []) {  // Attempted infinite loop, can't seem to get JS to wait
        setInterval(() => {}, 2500)
      }
      console.log(comicData);
      resolve(comicData);
    });
  });
};

[PersonMeetup]

Turns out I missed a section of the collections documentation:

addCollection callbacks can return any arbitrary object type and it’ll be available as data in the template. Arrays, strings, objects—have fun with it.

That solved the initial problem I was running up against! That said, I still need a solution to the original question as my data structure is running into issues seemingly related to #461. For reference, here's some sample JSON that I'm trying to use with pagination.

{
  comicTitle: {
    pages: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object]
    ],
    comicThumb: './comics/thumb.png',
    length: 14
  }
}

My intent is to use a specified pages object with pagination, but calling such data (collections.comics.comicTitle.pages) raises the following error:

[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] Could not find pagination data (./comic.html), went looking for: collections.comics.comicTitle.pages (via Error)

A possible solution has been discussed in #332, but given the format and use case of my data I feel it would involve solving my initial problem first.

[pdehaan]

I think you're hitting https://www.11ty.dev/docs/pagination/#paging-an-object where you're paging an object and it's resolving keys (default) or values.

// eleventy.config.js (snippet)
  eleventyConfig.addCollection("comics3", async () => {
    return {
        pages: [
          "One", "Two", "Three", "Four", "Five",
          "Six", "Seven", "Eight", "Nine", "Ten",
        ],
        comicThumb: './comics/thumb.png',
        length: 14,
    }
  });

// index.11tydata.js
module.exports = () => {
  return {
    pagination: {
      data: "collections.comics3",
      size: 3,
      resolve: "keys",
      before(data) {
        // console.log({ "this": this });
        console.log({ data }); // { data: [ 'pages', 'comicThumb', 'length' ] }
        return data.comicTitle.pages;
      }
    }
  }
};

OUTPUT

Because 11ty uses resolve: "keys" by default, the paginated data will only include the paginated object's keys:

{ data: [ 'pages', 'comicThumb', 'length' ] }

If we change to resolve: "values", the paginated data will include only the paginated object's values:

{
  data: [
    [
      'One',   'Two',
      'Three', 'Four',
      'Five',  'Six',
      'Seven', 'Eight',
      'Nine',  'Ten'
    ],
    './comics/thumb.png',
    14
  ]
}

[PersonMeetup]

Thanks for the response, @pdehaan! While setting resolve: "values" sadly didn't help (the problem isn't how the data is read, but is instead pointing pagination to where the data exists in the cascade), your demo code did make me to look at how pagination takes in data. That allowed me to figure out that pagination seemingly reads collections data differently from global data.

// eleventy.config.js (snippet)
eleventyConfig.addCollection("comics", async () => {
  return {
    comicTitle: {
      pages: [
        { folder: '1', thumb: './comics/thumb1.png' },
        { folder: '2', thumb: './comics/thumb2.png' },
        { folder: '3', thumb: './comics/thumb3.png' },
      ],
      comicThumb: './comics/thumb.png',
      length: 3,
    },
    graphicNovelTitle: {
      pages: [
        { folder: '1', thumb: './graphics/thumb1.png' },
        { folder: '2', thumb: './graphics/thumb2.png' },
        { folder: '3', thumb: './graphics/thumb3.png' },
        { folder: '4', thumb: './graphics/thumb4.png' },
      ],
      comicThumb: './graphics/thumb.png',
      length: 4,
    }
  }
});

// index.11tydata.js
module.exports = () => {
  return {
    pagination: {
      data: "collections.comics",
      size: 1,
      resolve: "values",
      before(data) {
        console.log({ data });
        return data;
      }
    }
  }
};

Quickly running the preceding code, we can confirm that pagination is able to read the collections.comic object.

{
  data: [
    { pages: [Array], comicThumb: './comics/thumb.png', length: 3 },
    { pages: [Array], comicThumb: './graphics/thumb.png', length: 4 }
  ]
}

When we change the data attribute for pagination to "collections.comics.comicTitle", however, we get the same error I was encountering before. (This time I've included the stack trace, just in case.)

[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] Could not find pagination data (./index.md), went looking for: collections.comics.comicTitle (via Error)
[11ty]
[11ty] Original error stack trace: Error: Could not find pagination data (./index.md), went looking for: collections.comics.comicTitle
[11ty]     at Pagination._get (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\Plugins\Pagination.js:141:13)
[11ty]     at Pagination.setData (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\Plugins\Pagination.js:82:29)
[11ty]     at new Pagination (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\Plugins\Pagination.j[11ty]     at Template.getTemplates (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\Template.js:703:21)
[11ty]     at TemplateMap.initDependencyMap (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\TemplateMap.js:414:41)
[11ty]     at async TemplateMap.cache (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\TemplateMap.js:458:5)
[11ty]     at async TemplateWriter._createTemplateMap (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\TemplateWriter.js:330:5)
[11ty]     at async TemplateWriter.generateTemplates (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\TemplateWriter.js:360:5)
[11ty]     at async TemplateWriter.write (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\TemplateWriter.js:407:23)
[11ty]     at async Eleventy.executeBuild (E:\My Documents\Programming\eleventy-pagination\node_modules\@11ty\eleventy\src\Eleventy.js:1191:13)

Now, if we were to instead add comics as global data (.addGlobalData()) and edit the data input to pagination accordingly (remove collections. since we're no longer calling a collections object), we would build just fine.

{ data: [ [ [Object], [Object], [Object] ], './comics/thumb.png', 3 ] }

The real kicker to me is that if I were to specify further and point to the pages array within comics.comicTitle (so the full data path would be comics.comicTitle.pages), everything still works!

{
  data: [
    { folder: '1', thumb: './comics/thumb1.png' },
    { folder: '2', thumb: './comics/thumb2.png' },
    { folder: '3', thumb: './comics/thumb3.png' }
  ]
}

Since my goal is to build pages from the above array, being able to specify sub-objects as pagination inputs is important. As is the case, it seems sub-objects within collections objects can't be inputs.