vxern / wiktionary-scraper Goto Github PK

🇬🇧 An extensible, robust and lightweight (45kB) Wiktionary.org scraper to fetch detailed information about words in various languages.

Home Page: https://npmjs.com/package/wiktionary-scraper

License: MIT License

TypeScript 100.00%

definitions dictionary english etymology javascript language parser scraper typescript wiktionary

wiktionary-scraper's Introduction

A lightweight scraper to fetch information about words in various languages from Wiktionary.

Table of contents
Usage
Completeness

Usage

To start using the scraper, first install it using the following command:

npm install wiktionary-scraper

The simplest way of using the scraper is as follows:

import * as Wiktionary from "wiktionary-scraper";

const results = await Wiktionary.get("word");

You can change the language of the target word by setting the lemmaLanguage:

import * as Wiktionary from "wiktionary-scraper";

const results = await Wiktionary.get('o', {
  lemmaLanguage: "Romanian",
});

You can specify if redirects should be followed by setting followRedirects to true:

import * as Wiktionary from "wiktionary-scraper";

// Redirects to and returns results for "Germany".
const results = await Wiktionary.get('germany', {
  followRedirects: true,
});

By default, the User-Agent header used in requests is filled in using a default value mentioning wiktionary-scraper.

To remove it, set userAgent to undefined.

If you want to change it, specify userAgent:

import * as Wiktionary from "wiktionary-scraper";

const results = await Wiktionary.get('word', {
  userAgent: "Your App (https://example.com)",
});

You can also parse HTML of the website directly, bypassing the fetch step.

ℹ️ Notice that, as opposed to get(), parse() is synchronous:

import * as Wiktionary from "wiktionary-scraper";

const results = Wiktionary.parse(html);

Completeness

This library currently only supports the English version of Wiktionary.

Features

Parses both single- and multiple-etymology entries.
Recognises standard, non-standard and some explicitly disallowed parts of speech, as defined here. In total, there are 60+ recognised parts of speech, which should cover the vast majority of definitions.
- Note, however, that it is very possible that the library will fail to recognise certain niche, non-standard parts of speech. Should you come across any, please post an issue.