11ty / eleventy-fetch Goto Github PK

I was attempting to use/create an Asset Cache for a response from the Google API (using theiir Node library). I deleted my .cache directory to test if it was working and starting getting the following:

> ENOENT: no such file or directory, open '.cache/eleventy-cache-assets-google_calender_events.json'
[0] 
[0] `Error` was thrown:
[0]     Error: ENOENT: no such file or directory, open '.cache/eleventy-cache-assets-google_calender_events.json'

If I create an empty .cache directory it seems to work fine.

The culprit seems to be the AssetCache.save method. Should this check for the .cache directory and create it if it doesn't exist?

Note that I followed the tutorial for the plugin to get this error. It assumes the .cache directory already exists.

This caught me off guard. I was using dryRun to test some requests, and at some point I noticed that they were coming back with answers suspiciously quick. Turned out I had run the request without dryRun at some point and therefor a cached item had become available.

I am not sure what is the more expected behaviour here. When someone sets dryRun to true to disable writing new caches, should it still read old caches anyway?

I do think the effect of dryRun can be made a little more clear in the documentation regardless. I might file a PR for that later, just wanted to get a discussion started here.

I've been happily using eleventy-fetch for months in an Astro + Sanity + Netlify environment. Recently trying to move to Cloudflare but when I try to build the site I get this error:

 Could not resolve "fs"
  File:
    node_modules/@11ty/eleventy-fetch/src/AssetCache.js:1:19
  Code:
    > 1 | const fs = require("fs");
        |                   ^
      2 | const fsp = fs.promises; // Node 10+
      3 | const path = require("path");
      4 | const flatCache = require("flat-cache");
  Stacktrace:
Error: Build failed with 29 errors:
node_modules/@11ty/eleventy-fetch/src/AssetCache.js:1:19: ERROR: Could not resolve "fs"
node_modules/@11ty/eleventy-fetch/src/AssetCache.js:3:21: ERROR: Could not resolve "path"
node_modules/@11ty/eleventy-fetch/src/AssetCache.js:5:31: ERROR: Could not resolve "crypto"
node_modules/@11ty/eleventy-fetch/src/RemoteAssetCache.js:1:19: ERROR: Could not resolve "fs"
node_modules/flat-cache/src/cache.js:1:19: ERROR: Could not resolve "path"

I must note that the problem occurs only when using an SSR route (and the cloudflare adapter), when statically building routes, elventy fetch works normally of course.

If I do not import eleventy-fetch, there's no error... I understand this is probably an upstream problem (@astrojs/cloudflare) etc... but if this is something easily fixable from my package.json file, I'd rather ask here first :)

Thanks a ton for the maintainer's hard work on this precious module!

The presence of node-fetch 2.6.1 in eleventy-cache-assets is triggering GitHub's Dependabot alerts regarding CVE-2022-0235. Apparently nothing earlier than 3.1.1 is considered safe.

Per the request at #5 it would be good to make the client adapter-able under the hood, especially important for Deno/native Node fetch moving forward.

For example, the GitHub GraphQL API requires a POST request. It would be nice to reuse the existing network functionality rather than having to manually do the request (and store it in the cache).

Since the GitHub API has the same URL even as the query changes, the existing URL-based keying mechanism does not work if you want to do more than one GitHub API request on a website. Would it be possible to (optionally?) add support for serializing the body of the request (including headers and payload) in some stable manner and adding that to the hash key (maybe by doing a sha256 of it or similar?)

We've been getting 403 Rate Limit Exceeded on our builds, and this would be a great new feature.

I’m following the docs here:

https://www.11ty.dev/docs/plugins/serverless/#re-use-build-time-cache-from-the-fetch-plugin

So, here's a test case to describe my issue. Here are two URLs:

/build/
/serverless/

Hit the serverless endpoint, and with every hit you'll see the log that it's fetching:

[11ty/eleventy-fetch] Fetching: https://dummyjson.com/products/1

https://github.com/ryangittings/11ty-serverless-fetch-issue

I’d expect the build cache to be reused and the URL not to be hit.

First, thank you for building eleventy-fetch! It has been great for the GET + cache pattern that's so common in static site building.

However, dealing with error responses continues to be painful since eleventy-fetch only returns a prose message:

Could that line be changed to the following?

throw new Error(`Bad response for ${this.displayUrl} (${response.status}): ${response.statusText}`, { cause: response })

That would then provide the whole failed response object for further processing. It would help write much more defensive code when 429's, 401's, etc. are found while collecting data.

Thanks!
🎩

Howdy! I think it would be really nice if I could pass in a function that fetches my data from the target URL, rather than passing in the URL directly. For my use case specifically, I want to pull in data from the GitHub API at build time using their Octokit helper library. It's definitely possible to make requests by hand by assembling a request URL, but it's not as ergonomic (plus, Octokit allows you to aggregate paginated calls without needing to do this yourself). Is this something this plugin could accommodate? Maybe it would just need to accept a user-supplied key for caching since it's no longer getting a URL directly.

Is your feature request related to a problem? Please describe.

I was encountering an issue while trying to set up my 11ty project. When I attempted to run npm start, it takes an extended amount of time to initiate the dev environment, approximately 5 minutes. Upon inspecting the console, I observe a few 'ETIMEDOUT' errors. This is ok, is due to my API being currently offline and expected to remain so for a while.

So my problem was this huge wait time.

Describe the solution you'd like

I would like to be able to provide a timeout for EleventyFetch and maybe even being able to set one on a global level to reduce the build time in this scenarios.

Describe alternatives you've considered

I wrapped my EleventyFetch calls with a Promise.race like this:

const timeoutPromise = new Promise((_, reject) => {
    setTimeout(() => {
      reject(new Error("Timeout: API took to long"));
    }, 2500);
  });

apiCall = await Promise.race([apiCall, timeoutPromise]);

Additional context

I posted this as a problem into the 11ty discord server (Title: EleventyFetch problem, very long await for API response) and a few users told me that it would be good to have something like this built into EleventyFetch, and proposed me to open a feature request, so here I am. Also I reviewed the others feature request to see if there was already one or similar, but I didn't spotted it. I am writing this into 11ty-Project and not 11ty-Fetch because this feature form was formatted to receive feature requests and the 11ty-fetch form was not, so it seemed more appropriate to write it here.

If something like this already existed and I have not been able to see it, I am sorry for wasting your time, I tried to look for it and I only found this which was not exactly the same.

Thank you so much for the attention and for 11ty.

How can you use this plugin to cache JSON from an API that requires paging i.e. several calls to gather all of the data before committing it to cache?

Thank you for this library 👍. I use RemoteAssetCache alongside with one of my 11ty projects in development mode to cache few requests to a wordpress backend.

In addition to caching response.body, is there a way to also cache response.headers.
I am currently querying the wordpress backend via REST API and wordpress returns the total posts via a header x-wp-totalpages which I then use to make additional api requests until I get the entire list of posts and 11ty can build it out.

How about an extra option like: options.getResponseValue or options.beforeSave be exposed in order to modify the response before storing it in the cache?

I'm using eleventy-cache-assets to fetch repository JSON data from the Github API using the URL https://api.github.com/repos/tannerdolby/eleventy-photo-gallery.

Utilizing a global data file _data/gitPhotoGallery.js, everything works great to Cache the data from api but how can I Cache multiple repositories data? I'm currently creating a new data file for each new repository I want to get api data for. (Adding stars and some other fields to the projects section of my site, which is about 4 repositories needing api requests to cache the json result)

With my current setup, I'm creating a separate global data file to cache the API result from github for a new repo. If _data/gitPhotoGallery.js links to the request for api.github...tannerdolby/eleventy-photo-gallery then I would create a new data file to fetch JSON data about my website's repo api.github...tannerdolby/tannerdolby.com in _data/gitTannerDolby.js.

const Cache = require("@11ty/eleventy-cache-assets");

module.exports = () => {
    // without try/catch block for length purposes
    let json = Cache("https://api.github.com/repos/tannerdolby/eleventy-photo-gallery", {
        duration: "1d",
        type: "json"
    };
    return {
        repoName: json.name,
        githubHandle: json.owner.login,
        homepage: json.homepage,
        stargazers: json.stargazers_count,
        watchers: json.watchers_count
    }
};

I understand the usage is const Cache: (source: any, opts: any) => Promise<any> and since it returns a single Promise it only makes sense that a single file ie github.js is only capable of caching one repository worth of JSON data from the github API.

Can I cache multiple assets and return a single Promise or do I need to create separate global data files for each new repository I want to make an API call for? From my understanding of Promise, I think I might be asking a silly question but wanted to understand better. Any info would be greatly appreciated! @zachleat

Hi,

I'm using this plugin to cache API responses and I've run into a problem where, occasionally, a URL will resolve to the wrong resource from our database.

The problem seems to occur here, where shorthash is used to generate the cache ID.
https://github.com/11ty/eleventy-cache-assets/blob/4ff6669de4fecba7b74cb46decb91e890c89693d/src/RemoteAssetCache.js#L12

Logging a couple of the problem URLs from out API, here's the URL, shorthash(URL) and the resolved resource ID. The second subject ASC0000q71 resolves as subject ASC0000qu3.

https://api.zooniverse.org/projects/illustratedlife/talk/subjects/ASC0000qu3, bc88de75, ASC0000qu3
https://api.zooniverse.org/projects/illustratedlife/talk/subjects/ASC0000q71, bc88de75, ASC0000qu3

I am trying to use this plugin to get Pinboard data via their API:

module.exports = async function() {
  let json = await Cache("https://${USERNAME}:${PASSWORD}@api.pinboard.in/v1/posts/all?format=json&tag=read", {
      duration: "1d",
      directory: ".cache",
      type: "json"
  });
  return json;
};

However I consistently get this error: "Caching an already local asset is not yet supported"

> Caching an already local asset is not yet supported.

`Error` was thrown:
    Error: Caching an already local asset is not yet supported.
        at save (/Users/filippo/playground/11ty/node_modules/@11ty/eleventy-cache-assets/eleventy-cache-assets.js:26:25)
        at queue.add (/Users/filippo/playground/11ty/node_modules/@11ty/eleventy-cache-assets/eleventy-cache-assets.js:44:25)
        at run (/Users/filippo/playground/11ty/node_modules/p-queue/dist/index.js:250:104)
        at PQueue._tryToStartAnother (/Users/filippo/playground/11ty/node_modules/p-queue/dist/index.js:198:38)
        at Promise (/Users/filippo/playground/11ty/node_modules/p-queue/dist/index.js:264:18)
        at new Promise (<anonymous>)
        at PQueue.add (/Users/filippo/playground/11ty/node_modules/p-queue/dist/index.js:245:16)
        at queueSave (/Users/filippo/playground/11ty/node_modules/@11ty/eleventy-cache-assets/eleventy-cache-assets.js:44:15)
        at module.exports (/Users/filippo/playground/11ty/_data/pinboard.js:10:22)
        at TemplateData.getDataValue (/Users/filippo/playground/11ty/node_modules/@11ty/eleventy/src/TemplateData.js:385:29)

I am a bit clueless to be honest, and I have run out of ideas on things to check that could be causing the issue. I feel there might be something obvious that I am missing.

I am trying to pass an api key in the headers but am getting back 403.

I can use node-fetch with the code below but am wondering if it is possible to accomplish this with eleventy-fetch

const fetch = require('node-fetch');
const url =
"https://example.com";
const apiKey = "xxxxxxxxxx";

fetch(url, {
    method: 'GET',
    headers: {
        'x-api-key': apiKey
    },
    mode: 'cors',
    cache: 'default',
})
.then(response => response.json())
.then(json => console.log(JSON.parse(JSON.parse(json).data)))
.catch(err => console.log(err));

This is the current code that I have using eleventy-fetch:

const EleventyFetch = require("@11ty/eleventy-fetch");
const url =
"https://example.com";
const apiKey = "xxxxxxxxxx";

async function getTechData() {

  return EleventyFetch(url, {
    method: 'GET',
    headers: {
        'x-api-key': apiKey
    },
    duration: "1d", // save for 1 day
    type: "json"    // we’ll parse JSON for you
  });
  const techData = response
  return techData;
};

module.exports = getTechData;

Thanks!

I am building a site with Lit web components and 11ty, and I would like make a request with eleventy-fetch to feed data into my component. I've tried a few approaches to this, but I am unable to import the cached data in my component. The errors all seem to revolve around lack of default imports/exports in eleventy-fetch. Can anyone help me with this issue?

11ty JS data file:

// data.js
const EleventyFetch = require('@11ty/eleventy-fetch');

module.exports = async function () {
  let json = await EleventyFetch(apiEndpointUrl, { type: 'json' });
  return json;
};

// also it seems
// `import EleventyFetch from '@11ty/eleventy-fetch';`
// and
// `export default async function() { ...
// are not valid?

web component file:

// I am unable to access the fetched data in my web component file
import data from '../_data/data.js'; // Error: 'default' is not exported by src/_data/data.js

// trying to perform the fetch within this file does not seem to work either
import EleventyFetch from '@11ty/eleventy-fetch'; // Error: 'EleventyFetch' is not exported by node_modules/@11ty/eleventy-fetch/eleventy-fetch.js

I was trying to cache some semi-expensive network calls to npm APIs and noticed I get errors if I set the cache key to something with a "/" in it (like a scoped package name). It looks like eleventy-cache-assets/AssetCache is treating the backslash as a file path delimiter when saving the file, which them blows up since there presumably isn't an ".cache/eleventy-cache-assets-@11ty/" folder.

// src/_data/test.js
const { AssetCache } = require("@11ty/eleventy-cache-assets");

module.exports = async function() {
  // This seems to be the issue here (the "/" specifically).
  const asset = new AssetCache("@11ty/eleventy");
  if (asset.isCacheValid("1h")) {
    return asset.getCachedValue();
  }

  const value = "Johnny Cache";

  await asset.save(value, "json");
  return value;
};

OUTPUT

 npm run build

> [email protected] build /private/tmp/11ty-cache-asset-slug
> eleventy

> ENOENT: no such file or directory, open '/private/tmp/11ty-cache-asset-slug/.cache/eleventy-cache-assets-@11ty/eleventy.json'

`Error` was thrown:
    Error: ENOENT: no such file or directory, open '/private/tmp/11ty-cache-asset-slug/.cache/eleventy-cache-assets-@11ty/eleventy.json'
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `eleventy`
npm ERR! Exit status 1

Workaround

I had to slugify the key name:

-  const asset = new AssetCache("@11ty/eleventy");
+  const slugFn = require("@11ty/eleventy/src/Filters/Slug");
+  const name = slugFn("@11ty/eleventy");
+  const asset = new AssetCache(name);

Now eleventy-cache-assets will save the filename as "./.cache/[email protected]" instead of "./.cache/eleventy-cache-assets-@11ty/eleventy.json"

Can this plugin be used with Axios together? Is there any example I can set this up properly?

When manually storing data in the cache (example from site, if you don't call asset.isCacheValid() and aren't using any other Eleventy-Fetch code, running in debug mode won't show any output at all from the Eleventy Fetch plugin.

This makes it hard to know if your asset cache is working and if not, why.

Things that I'd love to have included in the verbose/debug output:

• writing to a particular cache (asset.save called)
• retrieved cached value (asset.getCachedValue() called)
• file path that caches are being stored in

Thanks for this plugin, I use it -a LOT-. One thing I loved about eleventy-cache-assets is that the filename represented the cache name, so if I needed to edit a particular file to mock some data, it would be easy to locate it. I often do things like change properties on my object responses to test different states, copy and paste a bunch of items to show how pagination would look, etc. Will there ever be an option to generate human readable names for the cache files?

I didn't notice that this package changed its name until today 😅. Maybe its a good idea to mark it as deprecated and link to the new package:

https://docs.npmjs.com/cli/v8/commands/npm-deprecate

Look at this API! https://twitter.com/philhawksworth/status/1257284661874692097

reading @swaroopsm issue about caching headers I realize I'm trying to achieve the exact same thing: I need to know the total page (x-wp-totalpages) from the response header. Except I can't find how I'm suppose to do it.
When I try to read the header it returns an error :
WordPress API call failed: TypeError: Cannot read properties of undefined (reading 'get')

My codes reads as follows:

const wordpressAPIpages = 'https://example.com/wp-json/wp/v2/posts?orderby=date&order=desc&_fields=id&page=1';
async function wpPostPages() {
  try {
    const res = await EleventyFetch(wordpressAPIpages, { duration: "2d", type: "json" });
    return res.headers.get('x-wp-totalpages') || 0;
  }
  catch(err) {
    console.log(`WordPress API call failed: ${err}`);
    return 0;
  }
}

I tried to read things like this. But it did not help. Is it normal I don't have my header coming back? I need guidance here 🤔

There’s some confusion about the primary use of this plugin. It’s a wrapper around fetch to add local caching. Hopefully the name change clears this up and improves discoverability.

The plugin writes out the URL it caches, and some of these URLs contain sensitive data. (API keys lol. Run it on Travis CI and everyone in the world can get them from the public logs.)

It would be cool if the constructor could accept a flag like silent: true or name: 'Secret API request' so you can either completely hide or rename such requests in logs.