macbre / nodemw Goto Github PK

Enter via config's dryRun entry set to true or DRY_RUN env variable set.

Do not perform any "write" operations: edits, moves, deletes, uploads when in dry run mode.

I am writing Selenium tests for mediawiki/core and I would like to speed up some tests by using the API instead of the web interface.

Am I missing it, or is there no built-in way of creating users?

Is making direct API calls the only way to go?

Note that this only happens when other parameters are provided. The summary-only parameters seem to work fine for some reason. I'm using iojs v2.1.0.

Triggering code (very simple):

        async.each(Object.keys(nameToUrls), function(file, cb) {
            var from = nameToUrls[file];
            bot.uploadByUrl(file, from, { text: 'some text',
                comment: 'autoupload file',
                ignorewarnings: false }, function(err, res) {
                if (err) {
                    logger.error('Unable to upload ' + file, err);
                    return cb();
                }

                logger.info('Uploaded ' + file + ' from ' + from);
                cb();
            });
        }, function() {
            logger.info('Uploaded ' + lang);
        });

Error:

error:   Unable to upload <file>
Error: Request to API failed:     Error: Argument error, options.body.
    at Request._callback (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:162:14)
    at self.callback (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:354:22)
    at emitOne (events.js:77:13)
    at Request.emit (events.js:169:7)
    at Request.init (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:609:12)
    at new Request (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:265:8)
    at request (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/index.js:56:10)
    at Object.module.exports.doRequest (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:157:3)
    at Object.<anonymous> (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:236:15)
    at /home/tux/IdeaProjects/archivesbot/node_modules/nodemw/lib/api.js:59:4
_http_outgoing.js:429
        throw new TypeError('first argument must be a string or Buffer');
              ^
TypeError: first argument must be a string or Buffer
    at ClientRequest.OutgoingMessage.write (_http_outgoing.js:429:11)
    at Request.write (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:1510:25)
    at /home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:703:18
    at Array.forEach (native)
    at end (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:702:21)
    at Immediate._onImmediate (/home/tux/IdeaProjects/archivesbot/node_modules/nodemw/node_modules/request/request.js:734:7)
    at processImmediate [as _immediateCallback] (timers.js:368:17)

Hello macbre,

I'd like to use the prepend and append (section="new") edit, but don't know how is the best way to do it.

Maybe the edit function could be enhanced or other functions could be created....

https://en.wikipedia.org/wiki/Special:ApiSandbox#action=edit

Best

Every now and then I get one of these when I make an edit request:

Socket error: socket hang up (ECONNRESET)
Error: socket hang up
    at createHangUpError (http.js:1263:15)
    at Socket.socketOnEnd [as onend] (http.js:1351:23)
    at TCP.onread (net.js:417:26)

I tried catching it by adding the following to doRequest():

req.on('socket', function(socket) {
    socket.on('error', function(e) {
        console.log("SOCKET ERROR:",e);
    });
});

but to no avail. Any idea how to catch these? They happen very rarely--once every few hundred requests.

getImages()

https://github.com/macbre/nodemw/blob/devel/lib/bot.js#L757

next.aifrom is undefined, so (next && next.aifrom) is false

next appears to be an object like

{ aicontinue: '007-VARGR-Mooney.jpg', continue: '-||' }

The error handling seems a bit shaky. When doing an ask query (Semantic MediaWiki) the info variable is undefined even though data.query.results has results. This causes the response to fall through all the ifs all they way town to the else case even though there is actually no error. And then crashes on api.js:206 because data.error is undefined.

I think this part of the code should only run if there actually is an error i.e. typeof data.error !== 'undefined'.

This is my params object for client.api.call:

{
  action: 'ask',
  query: '[[Category:SIP Phone]]|?User'
};

and this is what the data variable contains:

{ query: 
   { printrequests: [ [Object], [Object] ],
     results: { 'SIP Phone 60': [Object], 'SIP Phone 61': [Object] },
     serializer: 'SMW\\Serializers\\QueryResultSerializer',
     version: 0.5,
     meta: { hash: 'd82a813f78da82e1d9b94acaee6e37c5', count: 2, offset: 0 } } }

and this is the complete error:

Debug: internal, implementation, error 
    TypeError: Cannot read property 'info' of undefined
    at Request._callback (/home/max/projects/xml-phonebook/node_modules/nodemw/lib/api.js:208:63)
    at Request.self.callback (/home/max/projects/xml-phonebook/node_modules/nodemw/node_modules/request/request.js:199:22)
    at Request.emit (events.js:98:17)
    at Request.<anonymous> (/home/max/projects/xml-phonebook/node_modules/nodemw/node_modules/request/request.js:1160:14)
    at Request.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/home/max/projects/xml-phonebook/node_modules/nodemw/node_modules/request/request.js:1111:12)
    at IncomingMessage.emit (events.js:117:20)
    at _stream_readable.js:944:16
    at process._tickDomainCallback (node.js:486:13)

TL;DR: It tries to report an error even though everything went fine.

What's the best way to catch api errors? For example, I want to catch "Page doesn't exist" errors when I try to delete pages that don't exist, but surrounding bot.delete() calls in a try/catch doesn't seem to work.

Callbacks should be passed an Error as the first argument and data as the second.

Like so:

client.logIn(function (err, data) {
  if (err) { return console.log('login failed', err); }

  console.log('login successful', data);
});

Instead of:

client.logIn(function (data) {
  console.log('maybe login successful', data);
});

Docs: http://www.wikia.com/api/v1

// @see http://poznan.wikia.com/api/v1/Mercury/WikiVariables
client.wikia.getWikiVariables(function(err, vars) {
  console.log(vars.theme['color-links']); // #006cb0
  console.log(vars.wikiCategories); // ["travel"]
});

// @see http://poznan.wikia.com/api/v1/User/Details?ids=119245&size=50
client.wikia.getUser(userId, function(err, function(err, user) {
  console.log(user.avatar);
  console.log(user.numberofedits);
});

// @see http://poznan.wikia.com/api/v1/User/Details?ids=1,119245&size=50
// Maximum size of id list is 100.
client.wikia.getUsers([user1Id, user2Id], function(err, function(err, users) {
  console.log(users[0].avatar);
  console.log(users[0].numberofedits);

  console.log(users[1].avatar);
  console.log(users[1].numberofedits);
});

https://www.mediawiki.org/wiki/API:Purge

support for https config would be great. I've implemented it locally, but it's a minor change.

Use Mocha

https://www.youtube.com/watch?v=o_QbyP6q0AQ

http://stackoverflow.com/questions/5830387/how-to-find-all-youtube-video-ids-in-a-string-using-a-regex/5831191#5831191

DEBUG=1 node script.js should run script.js in debug mode

I'd like to implement support for Semantic Mediawiki's ask.

http://semantic-mediawiki.org/wiki/Ask_API

however, while the action is ask, it doesn't populate data[ask] as api.js expects. it uses data[results].

is there a good way to resolve this?

thanks!

such method as "token" doesn't exist in the code.

It would be usefull to add the new-topic for the "new" Flow extension for talk pages.

Documentation: https://www.mediawiki.org/w/api.php?action=help&modules=flow%2Bnew-topic

I am using script depending both on nodemw and disconnect (by @bartve [https://github.com/bartve/disconnect]).

Method "edit" always ends up returning following error in callback: "Edit failed: null"

What about more one boolean parameter in getArticle: function(title, redirect, callback) {?

if (redirect) params.redirects=1

API:Query#Resolving_redirects

I notice under some conditions (when logging in, over http) a POST request includes the full location which causes problems.

POST http://hostname.org:80/mw/api.php HTTP/1.1" 403 540 "-" "nodemw v0.3.5 (node.js v0.8.21)"

this seems to throw mediawiki off; if I remove http://hostname.org:80 it's ok. is this a bug?

In non-english wiki .getPagesInCategory() doesn't work, since the prefix can vary for each wiki.

How do I parse the output returned from getArticle? It doenst seems like json .

{{about||the technology company|Apple Inc.|other uses}}
{{redirect|Apple tree}}
{{pp-semi-indef}}
{{pp-move-indef}}
{{Use dmy dates|date=January 2015}}
{{good article}}
{{taxobox
|name = Apple
|image = Red Apple.jpg
|image_caption = Fruit
|image2 = File:Malus domestica a1.jpg
|image2_caption = Flowers
|regnum = [[Plant]]ae
|unranked_divisio = [[Angiosperms]]
|unranked_classis = [[Eudicots]]
|unranked_ordo = [[Rosids]]
|ordo = [[Rosales]]
|familia = [[Rosaceae]]
|genus = ''[[Malus]]''
|species = '''''M. domestica'''''
|binomial = ''Malus domestica''
|binomial_authority = [[Moritz Balthasar Borkhausen|Borkh.]], 1803
|synonyms = ''Malus communis'' <small>Desf.</small><br/>
''Malus pumila'' <small>auct.</small><ref>{{GRIN | name = ''Malus pumila'' auct. | id = 410495 | accessdate = 4 January 2012}}</ref><br/>
''Pyrus malus'' <small>L.</small><ref>{{GRIN | name = ''Pyrus malus'' L. | id = 30530 | accessdate = 29 January 2012}}</ref>
|}}
The '''apple tree''' (''Malus domestica'') is a [[deciduous]] [[tree]] in the [[Rosaceae|rose family]] best known for its sweet, [[pome|pomaceous]] [[fruit]], the '''apple'''. It is [[agriculture|cultivated]] worldwide as a [[fruit tree]], and is the most widely grown species in the [[genus]] ''[[Malus]]. ''The tree originated in [[Central Asia]], where its wild ancestor, ''[[Malus sieversii]]'', is still found today. Apples have been grown for thousands of years in Asia and Europe, and were brought to North America by European colonists. Apples have [[religion|religious]] and [[mythology|mythological]] significance in many cultures, including [[Norse mythology|Norse]], [[Greek mythology|Greek]] and European Christian traditions.

• The library should be completely silent unless the debug option is set to either true or a debug level ("info", "debug")
• The debug option should be false by default

Reason: It's currently impossible to not have an app print stuff to the console.

https://www.npmjs.com/package/diff

Both of these promise rejections can happen at the same time, which throws an error in deferred.js. (Aside: I've read that in some promise implementations, the second rejection is treated as a noop.)

Not sure what the best way to solve this is. It's nice to get the info for both errors, but it's not nice for the server to crash. Maybe could test each case separately, without rejecting, then do one combined rejection after both if statements?

Looks like it crashes if you don't specify an edit callback. Should probably be changed to callback && callback() like the other helper functions have it?

Hello, is it possible to get revisions content through bot.getArticleCategories(title, callback)?

https://github.com/caolan/async#queue

• three parallel workers are run by default (can be customized via concurrency option)
• order in which response arrive is no longer guaranteed - kind of obvious, but worth mentioning anyway ;)

https://github.com/mikeal/request

wondering if you might know why some simple code throws this error on 0.10.32, yet it works fine on 0.10.30?

thanks!

Help page

• Getting item ID
• Getting claims for a given item
• WikiData viewer

Would be nice to be on npm or at least have a package.json.

When making changes to existing content (e.g. not appending or prepending text), it is important that bots don't accidentally overwrite edits by other users.

The way bots should do this to, when fetching the existing content, pass the revision timestamp to the edit module. This way, if another edit has been made since then, the edit will be rejected. At this point the bot can either try again, or skip the item for the time being.

mwbot provides a getArticle method, but it doesn't expose any meta data besides the page content.

Please provide an easy way for developers to use mwbot to make edits in a way that doesn't cause human edits to be overwritten by default. It should perhaps be an option to ignore conflicts, but by default it probably should not ignore conflicts.

Ideas:

// getPage(string name) -> API query revisions, rvprop=content|timestamp
client.getPage(name, function (err, data) {
  // data.title
  // data.content
  // data.timestamp

  var newContent = change(data.content);

  // Method 1: edit( pageName, content, summary, params, callback )
  client.edit(data.title, newContent, '', { basetimestamp: data.timestamp, fn(err,data) });

  // Method 2: edit( string|Object pageData, content, summary, calllback )
  client.edit(data, newContent, '', function (err, data) { });
});

The second method is probably easiest and encourages developers to use it without hardcoding details of parameters.

Hello,

I'm trying to use the getImagesFromArticle call on the bot and I'm not sure what I'm doing wrong, but it doesn't seem to be working the way it is supposed to. This is how I'm calling it:

client.getImagesFromArticle(bodyParsed[i].comName, function (err, data) {
    if (err) { console.error(err) }
    console.log('images: ');
    console.log(data);
})

Here is where the problem occurs:

Any suggestions are much appreciated! Thanks!

This happens if mediawiki crashes (PHP error) to nodemw:

Maybe this is something you should check and return as an err callback instead.
Thanks for introducing this, btw!

TypeError: Cannot read property 'result' of undefined
at [...]\node_modules\nodemw\lib\bot.js:299:22

Do you mind adding a automatic continuation mechanism for getPagesInCategory? Because right now, it will stop after 500 entries. Any "cmlimit" value over 500 has no effect either (due to wikimedia).

I tried something fast (without fully reading through all the code).

This does the job for me:

I replaced:

this.api.call({
    action: 'query',
    list: 'categorymembers',
    cmtitle: category,
    cmlimit: 5000
}, function(err, data) {
    callback(err, data && data.categorymembers || []);
});

with:

this.getAll(
    {
        action: 'query',
        list: 'categorymembers',
        cmtitle: category,
        cmlimit: 5000,
        cmtype: 'page',
        rawcontinue: true
    },
    'categorymembers',
    callback
);

inside "getPagesInCategory".

No idea if it is complete BS though.

• create csv file from Special::Log entries
• allow filtering by letype

@PiotrGackowski thanks events log

Hello!

I'm using the api.call method for editing pages, because I needed to adjust some more advanced parameters. The wiki of course needs the token in order to do this, but I can't send it through the parameters:

code="mustposttoken" info="The token parameter was found in the query string, but must be in the POST body"

If I don't set the token parameter I'll get The token parameter must be set

Best,
Simon

• http://about.travis-ci.org/docs/user/getting-started/
• http://about.travis-ci.org/docs/user/languages/javascript-with-nodejs/

Hello,

I often encounter token errors (Error: Invalid token) and after doing some changes to my code, I realised that the error was made because of the concurrency of the requests.

I guess that a token has to be used directly to successfully edit a page. If a second token is asked in the meantime, the edit will fail with the aforementioned error.

Best regards,
Val'

Is there a way to define a bot's user agent as per Mediawiki's API guidelines?

Wikimedia is using a new continuation mechanism. Is it probably possible to support that?

For more info:
https://www.mediawiki.org/wiki/API:Raw_query_continue

Right now, "this.getAll(...)" wont continue at all after 500 entries. Except if you pass in "rawcontinue: true" as property.

Example

const url = 'https://www.youtube.com/watch?v=24X9FpeSASY'; // or https://vimeo.com/27986705

client.wikia.uploadVideo(fileName, url, (err, res) => {
  // ...
});

API request parameters

action=addmediapermanent

• title
• provider
• videoId
• token

Supported videos

YouTube

https://github.com/Wikia/app/blob/dev/extensions/wikia/VideoHandlers/apiwrappers/YoutubeApiWrapper.class.php

provider: youtube
id: axu2ju-HwTg

Vimeo

https://github.com/Wikia/app/blob/dev/extensions/wikia/VideoHandlers/apiwrappers/VimeoApiWrapper.class.php

provider: vimeo
id: 27986705