macbre / nodemw Goto Github PK
View Code? Open in Web Editor NEWMediaWiki API and WikiData client written in Node.js
Home Page: https://www.npmjs.com/package/nodemw
License: BSD 2-Clause "Simplified" License
MediaWiki API and WikiData client written in Node.js
Home Page: https://www.npmjs.com/package/nodemw
License: BSD 2-Clause "Simplified" License
Enter via config's dryRun
entry set to true or DRY_RUN
env variable set.
Do not perform any "write" operations: edits, moves, deletes, uploads when in dry run mode.
I am writing Selenium tests for mediawiki/core and I would like to speed up some tests by using the API instead of the web interface.
Am I missing it, or is there no built-in way of creating users?
Is making direct API calls the only way to go?
Note that this only happens when other parameters are provided. The summary-only parameters seem to work fine for some reason. I'm using iojs v2.1.0.
Triggering code (very simple):
async.each(Object.keys(nameToUrls), function(file, cb) {
var from = nameToUrls[file];
bot.uploadByUrl(file, from, { text: 'some text',
comment: 'autoupload file',
ignorewarnings: false }, function(err, res) {
if (err) {
logger.error('Unable to upload ' + file, err);
return cb();
}
logger.info('Uploaded ' + file + ' from ' + from);
cb();
});
}, function() {
logger.info('Uploaded ' + lang);
});
Error:
error: Unable to upload <file>
Error: Request to API failed: Error: Argument error, options.body.
at Request._callback (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:162:14)
at self.callback (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:354:22)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at Request.init (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:609:12)
at new Request (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:265:8)
at request (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/index.js:56:10)
at Object.module.exports.doRequest (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:157:3)
at Object.<anonymous> (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:236:15)
at /home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:59:4
_http_outgoing.js:429
throw new TypeError('first argument must be a string or Buffer');
^
TypeError: first argument must be a string or Buffer
at ClientRequest.OutgoingMessage.write (_http_outgoing.js:429:11)
at Request.write (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:1510:25)
at /home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:703:18
at Array.forEach (native)
at end (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:702:21)
at Immediate._onImmediate (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:734:7)
at processImmediate [as _immediateCallback] (timers.js:368:17)
Hello macbre,
I'd like to use the prepend and append (section="new") edit, but don't know how is the best way to do it.
Maybe the edit function could be enhanced or other functions could be created....
https://en.wikipedia.org/wiki/Special:ApiSandbox#action=edit
Best
Every now and then I get one of these when I make an edit request:
Socket error: socket hang up (ECONNRESET)
Error: socket hang up
at createHangUpError (http.js:1263:15)
at Socket.socketOnEnd [as onend] (http.js:1351:23)
at TCP.onread (net.js:417:26)
I tried catching it by adding the following to doRequest():
req.on('socket', function(socket) {
socket.on('error', function(e) {
console.log("SOCKET ERROR:",e);
});
});
but to no avail. Any idea how to catch these? They happen very rarely--once every few hundred requests.
getImages()
https://github.com/macbre/nodemw/blob/devel/lib/bot.js#L757
next.aifrom is undefined, so (next && next.aifrom) is false
next appears to be an object like
{ aicontinue: '007-VARGR-Mooney.jpg', continue: '-||' }
The error handling seems a bit shaky. When doing an ask
query (Semantic MediaWiki) the info variable is undefined
even though data.query.results
has results. This causes the response to fall through all the ifs all they way town to the else case even though there is actually no error. And then crashes on api.js:206 because data.error
is undefined
.
I think this part of the code should only run if there actually is an error i.e. typeof data.error !== 'undefined'
.
This is my params
object for client.api.call
:
{
action: 'ask',
query: '[[Category:SIP Phone]]|?User'
};
and this is what the data
variable contains:
{ query:
{ printrequests: [ [Object], [Object] ],
results: { 'SIP Phone 60': [Object], 'SIP Phone 61': [Object] },
serializer: 'SMW\\Serializers\\QueryResultSerializer',
version: 0.5,
meta: { hash: 'd82a813f78da82e1d9b94acaee6e37c5', count: 2, offset: 0 } } }
and this is the complete error:
Debug: internal, implementation, error
TypeError: Cannot read property 'info' of undefined
at Request._callback (/home/max/projects/xml-phonebook/node_modules/nodemw/lib/api.js:208:63)
at Request.self.callback (/home/max/projects/xml-phonebook/node_modules/nodemw/node_modules/request/request.js:199:22)
at Request.emit (events.js:98:17)
at Request.<anonymous> (/home/max/projects/xml-phonebook/node_modules/nodemw/node_modules/request/request.js:1160:14)
at Request.emit (events.js:117:20)
at IncomingMessage.<anonymous> (/home/max/projects/xml-phonebook/node_modules/nodemw/node_modules/request/request.js:1111:12)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:944:16
at process._tickDomainCallback (node.js:486:13)
TL;DR: It tries to report an error even though everything went fine.
What's the best way to catch api errors? For example, I want to catch "Page doesn't exist" errors when I try to delete pages that don't exist, but surrounding bot.delete() calls in a try/catch doesn't seem to work.
Callbacks should be passed an Error
as the first argument and data as the second.
Like so:
client.logIn(function (err, data) {
if (err) { return console.log('login failed', err); }
console.log('login successful', data);
});
Instead of:
client.logIn(function (data) {
console.log('maybe login successful', data);
});
error: Request to API failed: %s code=ECONNRESET
TypeError: Cannot read property 'statusCode' of undefined
at Request._callback (/home/macbre/github/pyrabot/node_modules/nodemw/lib/api.js:166:16)
at self.callback (/home/macbre/github/pyrabot/node_modules/nodemw/node_modules/request/request.js:129:22)
Docs: http://www.wikia.com/api/v1
// @see http://poznan.wikia.com/api/v1/Mercury/WikiVariables
client.wikia.getWikiVariables(function(err, vars) {
console.log(vars.theme['color-links']); // #006cb0
console.log(vars.wikiCategories); // ["travel"]
});
// @see http://poznan.wikia.com/api/v1/User/Details?ids=119245&size=50
client.wikia.getUser(userId, function(err, function(err, user) {
console.log(user.avatar);
console.log(user.numberofedits);
});
// @see http://poznan.wikia.com/api/v1/User/Details?ids=1,119245&size=50
// Maximum size of id list is 100.
client.wikia.getUsers([user1Id, user2Id], function(err, function(err, users) {
console.log(users[0].avatar);
console.log(users[0].numberofedits);
console.log(users[1].avatar);
console.log(users[1].numberofedits);
});
support for https config would be great. I've implemented it locally, but it's a minor change.
Use Mocha
error: Request to API failed: %s Error: read ECONNRESET
at errnoException (net.js:904:11)
at TCP.onread (net.js:558:19)
TypeError: Cannot read property 'pages' of undefined
at /home/macbre/github/pyrabot/node_modules/nodemw/lib/bot.js:387:33
at Request._callback (/home/macbre/github/pyrabot/node_modules/nodemw/lib/api.js:162:5)
at self.callback (/home/macbre/github/pyrabot/node_modules/nodemw/node_modules/request/request.js:354:22)
at Request.emit (events.js:95:17)
at Request.onRequestError (/home/macbre/github/pyrabot/node_modules/nodemw/node_modules/request/request.js:1011:8)
at ClientRequest.emit (events.js:95:17)
at Socket.socketErrorListener (http.js:1547:9)
at Socket.emit (events.js:95:17)
at net.js:440:14
at process._tickCallback (node.js:419:13)
DEBUG=1 node script.js
should run script.js
in debug mode
I'd like to implement support for Semantic Mediawiki's ask.
http://semantic-mediawiki.org/wiki/Ask_API
however, while the action is ask, it doesn't populate data[ask] as api.js expects. it uses data[results].
is there a good way to resolve this?
thanks!
such method as "token" doesn't exist in the code.
It would be usefull to add the new-topic
for the "new" Flow extension for talk pages.
Documentation: https://www.mediawiki.org/w/api.php?action=help&modules=flow%2Bnew-topic
npm WARN package.json [email protected] specifying dependencies as array is deprecated
npm WARN package.json [email protected] specifying devDependencies as array is deprecated
I am using script depending both on nodemw and disconnect (by @bartve [https://github.com/bartve/disconnect]).
Method "edit" always ends up returning following error in callback: "Edit failed: null"
What about more one boolean parameter in getArticle: function(title, redirect, callback) {
?
if (redirect) params.redirects=1
I notice under some conditions (when logging in, over http) a POST request includes the full location which causes problems.
POST http://hostname.org:80/mw/api.php HTTP/1.1" 403 540 "-" "nodemw v0.3.5 (node.js v0.8.21)"
this seems to throw mediawiki off; if I remove http://hostname.org:80 it's ok. is this a bug?
// props defaults to [ 'statistics' ]
client.getSiteInfo(props, function(err, info) {
console.log(info.statistics.edits);
});
In non-english wiki .getPagesInCategory() doesn't work, since the prefix can vary for each wiki.
How do I parse the output returned from getArticle? It doenst seems like json .
{{about||the technology company|Apple Inc.|other uses}}
{{redirect|Apple tree}}
{{pp-semi-indef}}
{{pp-move-indef}}
{{Use dmy dates|date=January 2015}}
{{good article}}
{{taxobox
|name = Apple
|image = Red Apple.jpg
|image_caption = Fruit
|image2 = File:Malus domestica a1.jpg
|image2_caption = Flowers
|regnum = [[Plant]]ae
|unranked_divisio = [[Angiosperms]]
|unranked_classis = [[Eudicots]]
|unranked_ordo = [[Rosids]]
|ordo = [[Rosales]]
|familia = [[Rosaceae]]
|genus = ''[[Malus]]''
|species = '''''M. domestica'''''
|binomial = ''Malus domestica''
|binomial_authority = [[Moritz Balthasar Borkhausen|Borkh.]], 1803
|synonyms = ''Malus communis'' <small>Desf.</small><br/>
''Malus pumila'' <small>auct.</small><ref>{{GRIN | name = ''Malus pumila'' auct. | id = 410495 | accessdate = 4 January 2012}}</ref><br/>
''Pyrus malus'' <small>L.</small><ref>{{GRIN | name = ''Pyrus malus'' L. | id = 30530 | accessdate = 29 January 2012}}</ref>
|}}
The '''apple tree''' (''Malus domestica'') is a [[deciduous]] [[tree]] in the [[Rosaceae|rose family]] best known for its sweet, [[pome|pomaceous]] [[fruit]], the '''apple'''. It is [[agriculture|cultivated]] worldwide as a [[fruit tree]], and is the most widely grown species in the [[genus]] ''[[Malus]]. ''The tree originated in [[Central Asia]], where its wild ancestor, ''[[Malus sieversii]]'', is still found today. Apples have been grown for thousands of years in Asia and Europe, and were brought to North America by European colonists. Apples have [[religion|religious]] and [[mythology|mythological]] significance in many cultures, including [[Norse mythology|Norse]], [[Greek mythology|Greek]] and European Christian traditions.
debug
option is set to either true
or a debug level ("info", "debug")debug
option should be false
by defaultReason: It's currently impossible to not have an app print stuff to the console.
Both of these promise rejections can happen at the same time, which throws an error in deferred.js. (Aside: I've read that in some promise implementations, the second rejection is treated as a noop.)
Not sure what the best way to solve this is. It's nice to get the info for both errors, but it's not nice for the server to crash. Maybe could test each case separately, without rejecting, then do one combined rejection after both if statements?
Looks like it crashes if you don't specify an edit callback. Should probably be changed to callback && callback()
like the other helper functions have it?
info: getToken: Action 'edit' is not allowed for the current user
Hello, is it possible to get revisions content through bot.getArticleCategories(title, callback)
?
https://github.com/caolan/async#queue
concurrency
option)wondering if you might know why some simple code throws this error on 0.10.32, yet it works fine on 0.10.30?
thanks!
Would be nice to be on npm or at least have a package.json.
When making changes to existing content (e.g. not appending or prepending text), it is important that bots don't accidentally overwrite edits by other users.
The way bots should do this to, when fetching the existing content, pass the revision timestamp to the edit module. This way, if another edit has been made since then, the edit will be rejected. At this point the bot can either try again, or skip the item for the time being.
mwbot provides a getArticle method, but it doesn't expose any meta data besides the page content.
Please provide an easy way for developers to use mwbot to make edits in a way that doesn't cause human edits to be overwritten by default. It should perhaps be an option to ignore conflicts, but by default it probably should not ignore conflicts.
Ideas:
// getPage(string name) -> API query revisions, rvprop=content|timestamp
client.getPage(name, function (err, data) {
// data.title
// data.content
// data.timestamp
var newContent = change(data.content);
// Method 1: edit( pageName, content, summary, params, callback )
client.edit(data.title, newContent, '', { basetimestamp: data.timestamp, fn(err,data) });
// Method 2: edit( string|Object pageData, content, summary, calllback )
client.edit(data, newContent, '', function (err, data) { });
});
The second method is probably easiest and encourages developers to use it without hardcoding details of parameters.
Hello,
I'm trying to use the getImagesFromArticle
call on the bot and I'm not sure what I'm doing wrong, but it doesn't seem to be working the way it is supposed to. This is how I'm calling it:
client.getImagesFromArticle(bodyParsed[i].comName, function (err, data) {
if (err) { console.error(err) }
console.log('images: ');
console.log(data);
})
Here is where the problem occurs:
Any suggestions are much appreciated! Thanks!
This happens if mediawiki crashes (PHP error) to nodemw:
Maybe this is something you should check and return as an err callback instead.
Thanks for introducing this, btw!
TypeError: Cannot read property 'result' of undefined
at [...]\node_modules\nodemw\lib\bot.js:299:22
Do you mind adding a automatic continuation mechanism for getPagesInCategory? Because right now, it will stop after 500 entries. Any "cmlimit" value over 500 has no effect either (due to wikimedia).
I tried something fast (without fully reading through all the code).
This does the job for me:
I replaced:
this.api.call({
action: 'query',
list: 'categorymembers',
cmtitle: category,
cmlimit: 5000
}, function(err, data) {
callback(err, data && data.categorymembers || []);
});
with:
this.getAll(
{
action: 'query',
list: 'categorymembers',
cmtitle: category,
cmlimit: 5000,
cmtype: 'page',
rawcontinue: true
},
'categorymembers',
callback
);
inside "getPagesInCategory".
No idea if it is complete BS though.
letype
Hello!
I'm using the api.call method for editing pages, because I needed to adjust some more advanced parameters. The wiki of course needs the token in order to do this, but I can't send it through the parameters:
code="mustposttoken" info="The token parameter was found in the query string, but must be in the POST body"
If I don't set the token parameter I'll get The token parameter must be set
Best,
Simon
Hello,
I often encounter token errors (Error: Invalid token) and after doing some changes to my code, I realised that the error was made because of the concurrency of the requests.
I guess that a token has to be used directly to successfully edit a page. If a second token is asked in the meantime, the edit will fail with the aforementioned error.
Best regards,
Val'
Is there a way to define a bot's user agent as per Mediawiki's API guidelines?
Wikimedia is using a new continuation mechanism. Is it probably possible to support that?
For more info:
https://www.mediawiki.org/wiki/API:Raw_query_continue
Right now, "this.getAll(...)" wont continue at all after 500 entries. Except if you pass in "rawcontinue: true" as property.
client.getArticleCategories(title, function(err, categories) {
// ...
});
const url = 'https://www.youtube.com/watch?v=24X9FpeSASY'; // or https://vimeo.com/27986705
client.wikia.uploadVideo(fileName, url, (err, res) => {
// ...
});
action=addmediapermanent
title
provider
videoId
token
https://github.com/Wikia/app/blob/dev/extensions/wikia/VideoHandlers/apiwrappers/YoutubeApiWrapper.class.php
provider: youtube
id: axu2ju-HwTg
https://github.com/Wikia/app/blob/dev/extensions/wikia/VideoHandlers/apiwrappers/VimeoApiWrapper.class.php
provider: vimeo
id: 27986705
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.