Internet Archive Fetch Handler library

A custom library for handling API requests.

Demo

Live demo — try a plain fetch, a CSRF-protected request, and the retrier against real responses.

Installation

npm install @internetarchive/fetch-handler

Sample Usage

import { FetchHandler } from '@internetarchive/fetch-handler';

@property({ type: Object }) data: any = null;
@property({ type: String }) error: string = '';
@property({ type: Boolean }) loading: boolean = false;

private fetchHandler: FetchHandler;

this.fetchHandler = new FetchHandler({
  apiBaseUrl: 'https://archive.org',
});


async fetchData() {
  this.loading = true;
  this.error = '';
  try {
    const result =
      await this.fetchHandler.fetchApiPathResponse('/metadata/goody');
    this.data = result;
  } catch (error) {
    this.error = `Error fetching data: ${error}`;
  } finally {
    this.loading = false;
  }
}

See the demo directory for a more detailed example.

Query Params

Pass queryParams instead of building the query string yourself. Values are stringified and URL-encoded, and undefined/null entries are dropped, so optional params can go straight in without a conditional:

// GET https://archive.org/services/content-flags/?identifier=goody
await fetchHandler.fetchApiPathResponse('/services/content-flags/', {
  queryParams: { identifier: 'goody', mediatype: undefined },
});

Params of the same name already on the URL are replaced. To repeat a key, pass a URLSearchParams:

const queryParams = new URLSearchParams();
queryParams.append('flag', 'spam');
queryParams.append('flag', 'violence');

// GET https://archive.org/services/content-flags/?flag=spam&flag=violence
await fetchHandler.fetchApiPathResponse('/services/content-flags/', {
  queryParams,
});

Configuring Retry

You can customize how you'd like FetchHandler to retry a request, both globally and per-request.

Basic Usage

There are two included configurations, DefaultRetryConfiguration and NoRetryConfiguration, which can be accessed via FetchRetryConfig.default and FetchRetryConfig.noRetry. .default (the default) will retry twice with exponential backoff delay and .noRetry will not retry.

Example:

import { FetchHandler, FetchRetrier, FetchRetryConfig } from '@internetarchive/fetch-handler';

// setting default: retry twice with exponential backoff delay
const fetchRetrier = new FetchRetrier({
  retryConfig: FetchRetryConfig.default
})

const fetchHandler = new FetchHandler({
  fetchRetrier
})

// use default retry configuration
await fetchHander.fetch('https://foo.com/api')

// specify retry config per-request: will not retry this request
await fetchHandler.fetch('https://foo.com/api', {
  retryConfig: FetchRetryConfig.noRetry
})

Custom Retry Configuration

Fetch retrying can be configured using a RetryConfiguring object. This interface has two methods:

interface RetryConfiguring {
  shouldRetry(
    response: Response | null,
    retryNumber: number,
    error?: unknown,
  ): boolean;

  retryDelay(
    retryNumber: number,
    response?: Response | null,
  ): Milliseconds | null;
}

class MyCustomRetryConfig implements RetryConfiguring {
  shouldRetry(
    response: Response | null,
    retryNumber: number
  ): boolean {
    if (response === null) return false;
    if (retryNumber > 1) return false;
    return response.status !== 404;
  }

  retryDelay(retryNumber: number): Milliseconds | null {
    return 1000;
  }
}

const customRetryConfig = new MyCustomRetryConfig();
// use your custom retry config on a request or globally
await fetchHandler.fetch('https://foo.com/api', {
  retryConfig: customRetryConfig
})

CSRF Tokens

FetchHandler can attach an X-CSRF-Token header to a request. Provide a getCsrfToken function when constructing the handler, then opt individual requests in with includeCsrfToken: true:

const fetchHandler = new FetchHandler({
  apiBaseUrl: 'https://archive.org',
  getCsrfToken: async () => {
    const { token } = await fetch('/services/csrf-token').then(r => r.json());
    return token;
  },
});

// sends the X-CSRF-Token header — no need to include it in the body
await fetchHandler.fetchApiPathResponse('/favorites/add', {
  method: 'POST',
  body: JSON.stringify({ identifier: 'goody' }),
  includeCsrfToken: true,
});

This is opt-in and off by default: not every backend endpoint's CORS policy allow-lists that header yet, and getting it wrong fails loudly (a blocked CORS preflight) rather than just missing a security control, so each caller should only pass includeCsrfToken: true once its target endpoint is known to accept the header. A request without it (or with getCsrfToken unset entirely) behaves exactly as before.

If a caller already sets an X-CSRF-Token header explicitly, it's left as-is.

Local Demo with web-dev-server

Add 127.0.0.1 local.archive.org to your /etc/hosts file

npm run start

NOTE The first time you visit the site, the browser will say the site is insecure because it's using a self-signed certificate. Accept the certificate and visit the site and the browser will then accept the certificate in the future. It may always say the site is insecure, but for the purposes of development, it's secure enough.

To run a local development server that serves the basic demo located in demo/index.html

Testing with Web Test Runner

To run the suite of Web Test Runner tests, run

npm run test

To run the tests in watch mode (for <abbr title="test driven development">TDD</abbr>, for example), run

npm run test:watch

Linting with ESLint, Prettier, and Types

To scan the project for linting errors, run

npm run lint

To automatically fix many linting errors, run

npm run format