semanticmediawiki / semanticmediawiki

🔗 Semantic MediaWiki turns MediaWiki into a knowledge management platform with query and export capabilities

Semantic MediaWiki

Semantic MediaWiki (a.k.a. SMW) is a free, open-source extension to MediaWiki – the wiki software that powers Wikipedia – that lets you store and query data within the wiki's pages.

Semantic MediaWiki is also a full-fledged framework, in conjunction with many spinoff extensions, that can turn a wiki into a powerful and flexible knowledge management system. All data created within SMW can easily be published via the Semantic Web, allowing other systems to use this data seamlessly.

For a better understanding of how Semantic MediaWiki works, have a look at deployed in 5 min and the Sesame, Fuseki triplestore video, or browse the wiki for a more comprehensive introduction.

Requirements

Semantic MediaWiki requires MediaWiki and its dependencies, such as PHP.

Supported MediaWiki, PHP and database versions depend on the version of Semantic MediaWiki. See the compatibility matrix for details.

Installation

The recommended way to install Semantic MediaWiki is by using Composer. See the detailed installation guide as well as the information on compatibility.

Documentation

Most of the documentation can be found on the Semantic MediaWiki wiki. A small core of documentation also comes bundled with the software itself. This documentation is minimalistic and less explanatory than what can be found on the SMW wiki. It is however always kept up to date, and applies to the version of the code it comes bundled with. The most important files are linked below.

• User documentation
• Technical documentation
• Hacking Semantic MediaWiki

Support

Primary support channels:

• User mailing list - for user questions
• SMW chat room - for questions and developer discussions
• Issue tracker - for bug reports

Contributing

Many people have contributed to SMW. A list of people who have made contributions in the past can be found here or on the wiki for Semantic MediaWiki. The overview on how to contribute provides information on the different ways available to do so.

If you want to contribute work to the project please subscribe to the developers mailing list and have a look at the contribution guidelines.

• File an issue
• Submit a pull request
• Ask a question on the mailing list

Tests

This extension provides unit and integration tests and are normally run by a continuous integration platform but can also be executed locally using the shortcut command composer phpunit from the extension base directory. A more comprehensive introduction can be found under the test section.

License

GNU General Public License, version 2 or later. The COPYING file explains SMW's copyright and license.

semanticmediawiki's Issues

New well designed result printer mechanism

Right now the query results are tightly bound to the store, making it nearly impossible to test logic in consumers (ie result printers).

• Original tread on the dev list
• Some conceptual hacking

This item needs to be tackled before we can make big changes affecting result printers, such as adoption of the new data values libraries.

Clean up issue tracking

We have not been doing that well of a job of keeping on top of all the reported SMW issues on bugzilla. There are many issues there that are not prioritized in a meaningful way and are not categorized in any way. Furthermore a good chunk are very old issues that no one has looked at for a long time, and are perhaps no longer relevant.

We have now been trying our the issue tracking here on GitHub, which is working well, and seems to have many advantages over bugzilla. I suggest we stick with it, and drop our support for bugzilla. Meaning that we point people here rather than to bugzilla, and move over old bugs of interest on an as-needed basis.

This means we need to update the documentation explaining where and how to report bugs.

Fix Travis relbuild

The relbuild branch lags behind the actual commit which can cause instability (as seen #113) where the MW 1.19 commit only failed after the merge because the original relbuild commit was one step behind the pull commit and therefore didn't caught the issue within the pull commit.

This causes #112 to fail even though it contains the 1.19 fix.

Make use of DataValues library

SMW is dependent on the DataValues library as of version 1.9 since this version is dependent on Validator 1.0 which uses DataValues. DataValues is a fresh implementation of the DataItems in SMW and contains ValueParsers and ValueFormatters that can replace the SMWDataValue code. Switching out the old SMW implementation for the new one in the DataValues library will be some work though brings important advantages:

• Fresh implementation taking care of design issues with current code
• Full test coverage of the new code
• The new code is SMW independent so can be shared with other projects (ie Wikidata)

Depends on #125

Fix subproperty-subcategory-mixup for "nb"

This is done by merging branch I18n into master as soon as it is 2.0-alpha. After that the release notes have to be expanded accordingly.

See this commit

Make use of Ask library

Replace the implementation of the Ask language in SMW with the new one found in the Ask library.

Usage of this library brings us the following advantages:

• Fresh implementation taking care of design issues with current code
• Full test coverage of the new code
• Technical documentation
• Object serialization and deserialization support that can be used in web APIs
• The new code is SMW independent so can be shared with other projects (ie Wikidata)

Depends on #126

File does not exist: resources/resources

I just had a look at my webserver' log file. Every hit throws the following error:

[error] [client 91.64.131.8] File does not exist: /.../extensions/SemanticMediaWiki/resources/resources referer: ...

I am on SMW 1.9.1 (9a9e33d) with MW 1.22.2.

Exception from line 316MagicWord.php / Exception from line 31Validator.php

I use MW 1.22.2.
I have no acces to shell.

I use the Semantic MediaWiki 1.9.0.2 (+dependencies) package http://sourceforge.net/projects/semediawiki/
Install method : via [SITENAME]/mw-config/

Test 1 (sin Validator extension):

[8bc5593f] /mw-config/ Exception from line 316 of [SITENAME]/www/includes/MagicWord.php: Error: invalid magic word 'listerrors'

Backtrace:
#0 [SITENAME]/www/includes/MagicWord.php(239): MagicWord->load(string)
#1 [SITENAME]/www/includes/parser/Parser.php(4982): MagicWord::get(string)
#2 [SITENAME]/www/extensions/SemanticMediaWiki/extensions/Validator/src/legacy/ParserHook.php(208): Parser->setFunctionHook(string, array, integer)
#3 [SITENAME]/www/extensions/SemanticMediaWiki/extensions/Validator/src/legacy/ListErrors.php(37): ParserHook->init(Parser)
#4 [internal function]: ValidatorListErrors::staticInit(Parser)
#5 [SITENAME]/www/includes/Hooks.php(199): call_user_func_array(string, array)
#6 [SITENAME]/www/includes/GlobalFunctions.php(3877): Hooks::run(string, array)
#7 [SITENAME]/www/includes/parser/Parser.php(275): wfRunHooks(string, array)
#8 [SITENAME]/www/includes/parser/Parser.php(287): Parser->firstCallInit()
#9 [SITENAME]/www/includes/parser/Parser.php(4823): Parser->clearState()
#10 [SITENAME]/www/includes/parser/Parser.php(367): Parser->startParse(Title, ParserOptions, integer, boolean)
#11 [SITENAME]/www/includes/installer/Installer.php(608): Parser->parse(string, Title, ParserOptions, boolean)
#12 [SITENAME]/www/includes/installer/WebInstaller.php(662): Installer->parse(string, boolean)
#13 [SITENAME]/www/includes/installer/WebInstallerPage.php(359): WebInstaller->getInfoBox(string)
#14 [SITENAME]/www/includes/installer/WebInstallerPage.php(347): WebInstaller_ExistingWiki->showKeyForm()
#15 [SITENAME]/www/includes/installer/WebInstaller.php(275): WebInstaller_ExistingWiki->execute()
#16 [SITENAME]/www/mw-config/index.php(61): WebInstaller->execute(array)
#17 [SITENAME]/www/mw-config/index.php(29): wfInstallerMain()
#18 {main}

Test 2 (with Validator):

Unexpected non-MediaWiki exception encountered, of type "Exception"
[259a6a25] /mw-config/ Exception from line 31 of [SITENAME]/www/extensions/Validator/Validator.php: Validator depends on the ParamProcessor library.
Backtrace:
#0 [SITENAME]/www/LocalSettings.php(258): require_once()
#1 [SITENAME]/www/includes/installer/Installer.php(528): require(string)
#2 [SITENAME]/www/includes/installer/WebInstallerPage.php(284): Installer::getExistingLocalSettings()
#3 [SITENAME]/www/includes/installer/WebInstaller.php(275): WebInstaller_ExistingWiki->execute()
#4 [SITENAME]/www/mw-config/index.php(61): WebInstaller->execute(array)
#5 [SITENAME]/www/mw-config/index.php(29): wfInstallerMain()
#6 {main}

Thanks

Ask API supporting count, min, max, average, etc.

I filed this issue 46458 in Bugzilla about trying to count results in the Semantic MediaWiki Ask API. I'll admit at the time I was still learning more and more about Semantic MediaWiki and I think I now get the answer that @mwjames provided better now. It was also awesome that @mwjames added SMWInfo fields to get the data I was looking for at the time.

I would like to throw out though that the non-visual, data-centric formats being accessible in the API would be a really nice win. While working on WikiApiary I routinely want to ask for the count of certain things in Semantic MediaWiki but you cannot. Ideally something like:

https://wikiapiary.com/w/api.php?action=ask&query=[[Category:Website]][[Is%20active::True]]|format=count&format=json

would return the count. Other non-display formats like min, max, average, sum, earliest, latest could all be very useful via API. Obviously things like table and list make no sense, which is the reason format isn't supported in the API. To some extent the challenge here is mixing data actions (count, min, max) with visual output (table, list).

Is this something that is being considered at some point?

FWIW: Calling the URL example given above with the format in it still generates an exception.

PHP message: PHP Catchable fatal error: Argument 1 passed to SMW\Api\Query::addQueryResult() must be an instance of SMWQueryResult, string given, called in /.../SemanticMediaWiki/includes/api/Ask.php on line 38 and defined in /.../SemanticMediaWiki/includes/api/Query.php on line 67

Use language code "nb" instead of "no" for Norwegian (Bokmål variant)

So far SMW is using MW's fallback functionality with regard of the translations of datatypes, special properties, namespaces, etc. User messages are already directly using language code "nb". I18n should be made more consistent and direct.

Issue addressed by SMW community member Even Thorbergsen and opened to have a reference for the RELEASE-NOTES.

PHP Strict Standards: Declaration of SMWParamFormat::formatValue()

Spotted in #140 (comment)

Code coverage and project risks

With 1.9.0.2 we cover a total of 41.12% lines but some domains still suck big times:

• articlepages 0.00%
• dataitems 54.26%
• datavalues 10.07%
• export 1.63%
• query 56.05%
• queryprinters 33.11%
• sparql 0.00%
• specials 31.40%
• storage 34.57% (coverage is only achieved because of the integration tests otherwise the Store fails completely on the notion of unit testing)
• parserhooks 60.38% (mostly because of legacy code in DeclareParserFunction, DocumentationParserFunction, InfoParserFunction)

Top Project Risks

• SMWTimeValue (23655)
• SMWSparqlStoreQueryEngine (17030)
• SMWExporter (15464)
• SMWSQLStore3QueryEngine (13304)
• SMWExportController (7482)
• SMWSQLHelpers (7140)
• SMWURIValue (5700)
• SMWSQLStore3 (4755)
• SMWPageSchemas (4290)
• SMWSQLStore3SetupHandlers (4160)

Analysis is based on PHP_CodeCoverage 1.2.7 using PHP 5.4.7 and PHPUnit 3.7.13 at Fri Jan 17 6:36:41 UTC 2014.

Borke SMWListResultPrinter compatibility

https://semantic-mediawiki.org/wiki/Thread:Semantic-mediawiki.org:Community_portal/Demos_not_working

Fatal error: Class 'SMWListResultPrinter' not found in /srv/semantic-mediawiki_org/www/core/extensions/SemanticResultFormats/formats/widget/SRF_ListWidget.php on line 14

Travis running tests parallel

Composer and Symphony running tests parallel on Travis using something like:

before_script:
    - sudo apt-get install parallel

script:
    - ls -d src/Symfony/*/* | parallel --gnu --keep-order 'echo "Running {} tests"; phpunit --exclude-group tty,benchmark {};' || exit 1
    - echo "Running tests requiring tty"; phpunit --group tty

https://github.com/symfony/symfony/blob/master/.travis.yml

composer/composer@38917c2#diff-50f49bafa5c7a00b65ee37ec8d0490ff

Might be useful for SMW as well.

smw-paramdesc-table-transpose missing

The smw-paramdesc-table-transpose message key is used on Special:Ask, yet not in the message file.

Ping @mwjames

Create SMW 1.9 tarball

And put it on SourceForge

Make tests user-language agnostic

Currently tests expect $wgLanguageCode = "en" otherwise tests will fail with:

$wgLanguageCode = "ja"

There were 12 failures:

1) SMW\Test\PredefinedPropertyAnnotatorTest::testAddSpecialPropertiesOnMockObserver with data set #4 (array(SMW\DIWikiPage, array(array('_LEDT')), array()), array(1, '_LEDT', array('User:Lula')))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for '_LEDT' as '_wpg' with (User:Lula)
Failed asserting that an array contains '利用者:Lula'.

3) SMW\Test\AskParserFunctionTest::testInstantiatedQueryData with data set #1 (array('[[Modification date::+]] [[Category:Foo bar]] [[Has title::!Foo bar]]', '?Modification date', '?Has title', 'format=list'), array(4, array('_ASKST', '_ASKSI', '_ASKDE', '_ASKFO'), array('list', 4, 1, '[[Modification date::+]] [[Category:Foo bar]] [[Has title::!Foo bar]]')), array(false))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for '_ASKST' as '_cod' with (list, 4, 1, [[Modification date::+]] [[Category:Foo bar]] [[Has title::!Foo bar]])
Failed asserting that an array contains '[[Modification date::+]] [[カテゴリ:Foo bar]] [[Has title::!Foo bar]]'.

4) SMW\Test\AskParserFunctionTest::testInstantiatedQueryData with data set #2 (array('[[Modification date::+]][[Category:Foo]]', '?Modification date', '?Has title', 'format=list'), array(4, array('_ASKST', '_ASKSI', '_ASKDE', '_ASKFO'), array('list', 2, 1, '[[Modification date::+]] [[Category:Foo]]')), array(false))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for '_ASKST' as '_cod' with (list, 2, 1, [[Modification date::+]] [[Category:Foo]])
Failed asserting that an array contains '[[Modification date::+]] [[カテゴリ:Foo]]'.

7) SMW\Test\SubobjectParserFunctionTest::testInstantiatedPropertyValues with data set #3 (array('Foo bar', 'Foo=Help:Bar'), array(false, 'Foo_bar', 1, 'Foo', 'Help:Bar'))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for 'Foo' as '_wpg' with (Help:Bar)
Failed asserting that 'Help:Bar' contains "ヘルプ:Bar".

8) SMW\Test\EncodingIntegrationTest::testBaseTemplateToolboxURLEncoding with data set #0 (array(array(array(true), true), Mock_SkinTemplate_ecb3d13e), 'Special:Browse/2013-2F11-2F05')
Asserts that process() returns an encoded URL
Failed asserting that '/mw-core-github/index.php/%E7%89%B9%E5%88%A5:%E9%96%B2%E8%A6%A7/2013-2F11-2F05' contains "Special:Browse/2013-2F11-2F05".

11) SMW\Test\ProfileAnnotatorQueryProcessorIntegrationTest::testCreateProfile with data set #1 (array('', '[[Modification date::+]][[Category:Foo]]', '?Modification date', '?Has title', 'format=list'), array(4, array('_ASKST', '_ASKSI', '_ASKDE', '_ASKFO'), array('list', 2, 1, '[[Modification date::+]] [[Category:Foo]]')))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for '_ASKST' as '_cod' with (list, 2, 1, [[Modification date::+]] [[Category:Foo]])
Failed asserting that an array contains '[[Modification date::+]] [[カテゴリ:Foo]]'.

$wgLanguageCode = "de"

There were 34 failures:

1) SMW\Test\DataValueFactoryTest::testNewTypeIdValue with data set #19 ('_num', 9001, '9,001', 'SMWNumberValue')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'9,001'
+'9.001'

7) SMW\Test\RecurringEventsTest::testGetDates with data set #0 (array('property=Has birthday', 'start=01 Feb 1970', 'has title=Birthday', 'unit=month', 'period=12', 'limit=3'), array(0, array('1 February 1970', '1 February 1971 00:00:00', '1 February 1972 00:00:00', '1 February 1973 00:00:00'), 'Has birthday', array(array('Birthday'))))
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => '1 February 1970'
-    1 => '1 February 1971 00:00:00'
-    2 => '1 February 1972 00:00:00'
-    3 => '1 February 1973 00:00:00'
+    0 => '1 Februar 1970'
+    1 => '1 Februar 1971 00:00:00'
+    2 => '1 Februar 1972 00:00:00'
+    3 => '1 Februar 1973 00:00:00'
 )

13) SMW\Test\PredefinedPropertyAnnotatorTest::testAddSpecialPropertiesOnMockObserver with data set #4 (array(SMW\DIWikiPage, array(array('_LEDT')), array()), array(1, '_LEDT', array('User:Lula')))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for '_LEDT' as '_wpg' with (User:Lula)
Failed asserting that an array contains 'Benutzer:Lula'.

16) SMW\Test\AskParserFunctionTest::testInstantiatedQueryData with data set #0 (array('[[Modification date::+]]', '?Modification date', 'format=list'), array(4, array('_ASKST', '_ASKSI', '_ASKDE', '_ASKFO'), array('list', 1, 1, '[[Modification date::+]]')), array(false))
SMW\Test\SemanticDataValidator::assertContainsPropertyValues for '_ASKST' as '_cod' with (list, 1, 1, [[Modification date::+]])
Failed asserting that an array contains '[[Zuletzt geändert::+]]'.

21) SMW\Test\ConceptParserFunctionTest::testParse with data set #0 (array('[[Modification date::+]]'), array(true, 1, '[[Modification date::+]]', '', 1, 1))
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'[[Modification date::+]]'
+'[[Zuletzt geändert::+]]'

SemanticMediaWiki namespaces

[1] currently only contains en, de, zh-cn mapping [2] in order for MWNamespace::getCanonicalName() to provide the correct results.

1. https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/languages/SemanticMediaWiki.namespaces.php
2. https://bugzilla.wikimedia.org/show_bug.cgi?id=34383

Database error when moving a page that has subpages

When moving a page that has subpages a database error occurs:

PHP message: PHP Notice: Uncommitted DB writes (transaction from DatabaseBase::query (SMWSql3SmwIds::getDatabaseIdAndSort)). in /srv/www/mediawiki/public_html/w/includes/db/Database.php on line 3944

This results in the subpages not moving. Semantic MediaWiki and SESP are the only things that have changed recently in my setup.

I’m on 95ab7ce.

SMWRecordValue in #set throws an exception with reference to "SMW_DI_Container.php ..."

This is a weird one. When running any of the following MediaWiki maintenance scripts I receive this error:

"Exception from line 76 of /var/www/html/w/extensions/SemanticMediaWiki/includes/dataitems/SMW_DI_Container.php: Trying to get the subject of a container data item that has not been given any. This container can only be used as a search pattern."

I'm running 1.23wmf10, SMW 1.9.0.2

A few examples of the error with backtraces:

http://pastebin.com/GQZV8bdk

http://pastebin.com/6SrC8beP

http://pastebin.com/r2ZApqfg

MediaWiki installation "A database error has occurred" when SMW is loaded (via Composer) before initialization

When repopulating a test installation the following issue appeared [1] which is due to SMW being auto-loaded via Composer.

Replicate the issue

• Install MW, run composer install/update, run update.php (should work without any issue as SMW is loaded after the MW install is completed)
• Now delete LocalSettings.php and delete the DB (MW and SMW tables)
• Initiate a restart of the MW installation (Composer will load SMW before the MW DB install is completed) and issue [1] appears.

I can't circumvent the issue unless I uninstall SMW since I'm unable to set SMW as "not to be loaded" via Composer.

[1]

Setting up database... done
Creating tables... done
Populating default interwiki table... done
Initializing statistics... done
Generating secret keys... done
Creating administrator user account... done
Creating main page with default content... failed

Could not insert main page: A database error has occurred. Did you forget to run maintenance/update.php after upgrading?
See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script 
Query: SELECT smw_id,smw_sortkey FROM smw_object_ids WHERE smw_title = 'Main_Page' AND smw_namespace = '0' AND smw_iw = '' 
AND smw_subobject = '' LIMIT 1 Function: SMWSql3SmwIds::getDatabaseIdAndSort Error: 1 no such table:
 smw_object_ids

RFC: Use of vendor or component specific libraries

SMW 1.9 and Composer have enabled to use vendor specific libraries without the need to expose its visibly to end-users.

Rational

Providing vendor libraries enables to test and extend libraries independently without the need to change SMW-core (unless the interface definition changed), it forces the separation of MW specific implementations (decrease the likelihood of breakage in case MW decides to alter functionality), and decreases complexity by distributing functionality.

Of lesser importance is the possibility for third-party extension developers to make use of these libraries as well.

Potential components

• SMW\Serializers (not sure about this one as this needs SMW\DI or SMW\DataModel as dependency)

List of potential classes

• HtmlFormBuilder

PageAnnotationImportSystemTest fails on DBTYPE=sqlite MW=1.19.0 PHP 5.5.0

The PageAnnotationImportSystemTest consistently fails on DBTYPE=sqlite MW=1.19.0 PHP 5.5.0 with "The property id to add must be a positive integer" string(19) "No pages to import." [1]

Right now my only guess is that the combination DBTYPE=sqlite/PHP 5.5.0 is causing the issue because DBTYPE=sqlite; MW=master 5.4 just works fine.

[1] https://s3.amazonaws.com/archive.travis-ci.org/jobs/13693194/log.txt

Ask query for URL types should not be case sensitive

URL types are currently queried case sensitive. URL's (I would think) should not be case sensitive. See example of issue.

Resource path regression

#89 (comment)

Change as done here might be applicable: wmde/DataValuesJavaScript@900b971

Fix RDF export of subobjects

https://gerrit.wikimedia.org/r/#/c/70047/

Bug: 43708
Bug: 44700
Bug: 48361
Bug: 48708

Note: this is submitted against 1.8.x, needs rebase to master

Improve UpdateJob

Store update

Currently UpdateJob calls updateStore explicitly.

$parserData = $this->withContext()->getDependencyBuilder()->newObject( 'ParserData', array(
    'Title'        => $this->getTitle(),
    'ParserOutput' => $parserOutput
) );

$parserData->disableUpdateJobs()->updateStore();

Instead use LinksUpdate which runs updateStore() but additionally does an incremental update on pagelinks, imagelinks, externallinks, and templatelinks tables.

$linksUpdate = new \LinksUpdate( $this->getTitle(), $parserOutput, false );
$linksUpdate->doUpdate();

This would increase update accuracy and would allow to resolve Bug 20352 (currently Special:WhatLinksHere isn't updated which shows outdated links to pages no longer reside in a query or for deleted subjects).

When the LinksUpdatecall is invoked the LinksUpdateConstructed hook is executed.

Duplicates

It needs to be tested if setting $this->removeDuplicates = true reduces possible duplicates for the same Title object.

Reference

https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/tests/phpunit/integration/MwLinksUpdateWithSQLStoreDBIntegrationTest.php

SMW_PageSchemas.php escaping

This file appears to have some code that does not do proper escaping.

Spotted in https://gerrit.wikimedia.org/r/#/c/110632/1/includes/SMW_PageSchemas.php

Adopt Semantic Versioning convention

Request to consider adopting the Semantic Versioning method for version numbers of Semantic MediaWiki. Reasons include:

• Easier to read, clearer. Well documented definitions.
• MediaWiki extensions do a terrible job in general of using version schemes. Semantic MediaWiki could lead some standardization.
• Semantic Versioning, Semantic MediaWiki.
• WikiApiary is going to support Semantic Versioning for extension tracking. See WikiApiary/WikiApiary#35.

Thank you for considering.

Fix Store update when uploading a file (Bug 19487)

[0] has been open and broken since 2009. The reason why the normal update hooks don't work is because there aren't executed in case of a file upload.

Investigate UploadVerifyFile (should be used as $mime is delivered by the interface) [1] or UploadComplete (doing $image->getLocalFile()->mime doesn't really work) [2].

Store update

$this->globals['wgHooks']['UploadVerifyFile'][] = function ( $upload, $mime, &$error ) use( $context ) {

    $title = $upload->getLocalFile()->getTitle();

    $contentParser = $context->getDependencyBuilder()->newObject( 'ContentParser', array(
        'Title' => $title
    ) );

    $contentParser->parse();

    $parserData = $context->getDependencyBuilder()->newObject( 'ParserData', array(
        'Title'        => $title,
        'ParserOutput' => $contentParser->getOutput()
    ) );

    $parserData->updateStore();

    return true;
};

Mime annotation

While resolving [0], the core should also add two new predefined properties _MIME and _MEDIA and annotate those for uploaded files with something like:

/**
 * @var PropertyAnnotator $propertyAnnotator
 */
$propertyAnnotator = $context->getDependencyBuilder()->newObject( 'FilePropertyAnnotator', array(
    'SemanticData'  => $parserData->getSemanticData(),
    'MimeType' => $mime
    'MediaType'   => $image->getLocalFile()->media_type
) );

$propertyAnnotator->attach( $parserData )->addAnnotation();

Reference

[0] https://bugzilla.wikimedia.org/show_bug.cgi?id=19487
[1] https://www.mediawiki.org/wiki/Manual:Hooks/UploadVerifyFile
[2] https://www.mediawiki.org/wiki/Manual:Hooks/UploadComplete
[3] https://gerrit.wikimedia.org/r/#/c/50943/

See also

#128

Factbox enable to display historical data ('oldid')

Currently it is not possible to display historical data in a Factbox [0] due to the ParserOuput being static but changes to SMW\OutputPageParserOutput with (see below) should allow to display those data with its correct Factbox content.

protected function performUpdate() {

$parserOutput = $this->parserOutput;

if ( $this->outputPage->getContext()->getRequest()->getInt( 'oldid' ) ) {

    $parserData = $this->withContext()->getDependencyBuilder()->newObject( 'ParserData', array(
        'Title'        => $this->outputPage->getTitle(),
        'ParserOutput' => $parserOutput
    ) );

    $contentProcessor = $this->withContext()->getDependencyBuilder()->newObject( 'ContentProcessor', array(
        'ParserData' => $parserData
    ) );

    $text = $parserOutput->getText();

    $contentProcessor->parse( $text );
    $parserOutput = $parserData->getOutput();
}

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/includes/Factbox.php#L244

Event Calendar Result Format Displays Error Instead of "Weekly" title

The event calendar format from Semantic Result Formats shows an error message in the "week" view title. It appears to be a parsing issue.

This can be seen on the demo page on SMW.org and I was able to reproduce it in my own local environment.

https://www.semantic-mediawiki.org/wiki/Demo:Event_calendar

Analyse _NEWP annotation regression

Was found during #164 testing.

Cause

it should be called $this->revision->getParentId() === null and not $this->revision->getParentId() !== '' because Revision returns

     * @return Integer|null
     */
    public function getParentId() {
        return $this->mParentId;
    }

Solution

public function isNewPage() {

    if ( $this->isFilePage() ) {
        return isset( $this->auxiliary->reupload ) ? !$this->auxiliary->reupload : false;
    }

    return $this->revision ? $this->revision->getParentId() === null : false;
}

As for the File object, the reupload needs to be used in order to decide whether it is new or not.

(Bug 60866) SMWSql3SmwIds::getDatabaseIdAndSort `smw_object_ids` doesn't exist

See [1].

MediaWiki 1.22.1
PHP 5.3.24 (cgi-fcgi)
MySQL 5.5.33-log

Semantic MediaWiki (Version 1.9.0.2)

A database query error has occurred. This may indicate a bug in the software.

    Query:
    SELECT smw_id,smw_sortkey FROM `tips-trickssmw_object_ids` WHERE smw_title
= 'Deneme' AND smw_namespace = '0' AND smw_iw = '' AND smw_subobject = '' LIMIT
1
    Function: SMWSql3SmwIds::getDatabaseIdAndSort
    Error: 1146 Table 'kompitirsoyle.tips-trickssmw_object_ids' doesn't exist
(<database server>)

Notice: Uncommitted DB writes (transaction from DatabaseBase::query
(WikiPage::pageData)). in ...html/tips-tricks/includes/db/Database.php on line
3944


Additionally: 
$wgDBprefix = tips-tricks
$wgDBname = "kompitirsoyle"

[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=60866

Remove wfGetDB usage from Store

Remove wfGetDB usage from Store and derive DB instance directly from store->getDatabase()

SMW_Store.php:                  $dbw = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3.php:             $qe = new SMWSQLStore3QueryEngine( $this, wfGetDB( DB_SLAVE ) );
SQLStore/SMW_SQLStore3.php:                     wfGetDB( DB_SLAVE ),
SQLStore/SMW_SQLStore3.php:                     wfGetDB( DB_SLAVE ),
SQLStore/SMW_SQLStore3.php:                     wfGetDB( DB_SLAVE ),
SQLStore/SMW_SQLStore3.php:                     wfGetDB( DB_SLAVE ),
SQLStore/SMW_SQLStore3.php:             $qe = new SMWSQLStore3QueryEngine( $this, wfGetDB( DB_MASTER ) );
SQLStore/SMW_SQLStore3.php:             $qe = new SMWSQLStore3QueryEngine( $this, wfGetDB( DB_MASTER ) );
SQLStore/SMW_SQLStore3.php:             $db = wfGetDB( DB_SLAVE );
SQLStore/SMW_SQLStore3.php:                     $db = wfGetDB( DB_SLAVE ); /// TODO avoid doing this here again, al
SQLStore/SMW_SQLStore3.php:             $db = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3_SetupHandlers.php:               $dbw = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3_SetupHandlers.php:               $dbr = wfGetDB( DB_SLAVE );
SQLStore/SMW_SQLStore3_SetupHandlers.php:                               $dbw = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3_Writers.php:                     $db = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3_Writers.php:             $dbw = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3_Writers.php:             $db = wfGetDB( DB_MASTER );
SQLStore/SMW_SQLStore3_Writers.php:             $db = wfGetDB( DB_SLAVE );
SQLStore/SMW_SQLStore3_Writers.php:             $db = wfGetDB( DB_MASTER ); // now we need to write something

See also

#159
#140
#112
#103

Create a testers group part of our GH org

This would allow us to ping all testers easily.

Suggested by kgh.

Improve regression detection and tests

Regressions tests are meant to cover a larger area of functionality and integration without focusing on a single implementation.

Regression tests are not meant to replace individual unit tests but to aid further verifiability from a different input level using a holistic approach to compare expected results.

To describe possible regression scenarios and make it easier for end-user and developers, we use MW's XML import/export functionality to enable the import of page content while making the test system and release independent.

• Scenarios are imported via the import/export functionality (should always work)
• Individual cases can be encapsulated, imported and verified against the expected outcome/output
• Users can import the XML's on their own system to acknowledge is functionality or identify related issues

Create a regression test

Regression tests run on a "live" DB to detect behavioural deviations with the import being expected to run all necessary activities deployed with a XML file.

Support pages (Property or Template namespace) should be specified and imported first to ensure a proper setup and recognition of different property types within the DB.

Importing from a Custom namespace should be avoided in order to ensure that a XML can be used on any system without the knowledge of that specific namespace.

Under normal circumstances regression tests should not break and a once specified XML (together with its test asserts) should not be adopted. If a different aspect of a regression needs elaboration a new scenario should be created.

• Export the regression scenario (normally as a wikipage that contains a specific issue together with related pages from the Property or Template namespace) as XML
• Use MwRegressionTestCase to describe the expected data and behaviour after an import
• MwRegressionTestCase does not replace an acceptance tests (UI tests such selenium etc.) but rather focuses on expected data objects within a wikipage or Store

Tests

• TimeDataTypeRegressionTest, #112
• CategoryInstanceAndCategoryHierarchyRegressionTest
• RecordDataTypeRegressionTest
• ConceptAndConceptQueriesRegressionTest, #156
• Redirect, #231, #234
• Subobjects
• Statistics
• Queries
• Subproperty of
• Maintenance script #267, #253

Tests could be designed for different languages because right now our tests are language-agnostic.

Related issues

• #124
• #107

Benchmarking

While a poor replacement for a real benchmark suite, nonetheless deployed XML's can be used to benchmark existing functionality on different systems, installations, and releases and help identify whether significant changes occurred during releases.

    $importer = new MwImporter( $sourceFile );

    if ( $importer->run() ) {
        $time = $importer->getImportTimeAsSeconds();
    }

composer.json "require composer/installers" breaks PHPUnit test run

The entry breaks

    "require": {
            "composer/installers": ">=1.0.1",
    },

local PHPUnit test run (php phphunit ...) with

Fatal error: Class 'Composer\Installers\Test\TestCase' not found in ...extensions
\SemanticMediaWiki\vendor\composer\installers\tests\Composer\Installers\Test\CakePHPInstallerTest.php on line 9

Have "announce wiki" use WikiApiary

The "announce wiki" functionality on Special:SMWAdmin uses some crappy old MediaWiki extension deployed on SMW.org. I'd rather see this extension be removed. From a user perspective it also makes more sense to use WikiApiary.

cc @thingles

New config should be documented

Config vars added in c2a0723 are not mentioned in the release notes

PHP Error: Invalid argument supplied for foreach

I did an update using composer this morning (git says I’m on fc84edf) and started seeing this warning/fatal in my error log:

PHP message: PHP Warning: Invalid argument supplied for foreach() in /.../w/extensions/SemanticMediaWiki/includes/storage/SQLStore/SMW_SQLStore3_Readers.php on line 328
PHP message: PHP Fatal error: Call to a member function free_result() on a non-object in /.../w/includes/db/DatabaseMysqli.php on line 131

Resource loading on smw.org is distorted

Navigation icons to switch between months do not appear to be included in release.

This can be seen on the following example:

https://www.semantic-mediawiki.org/w/index.php?title=Special:RunQuery&form=Query%2FScenario&target=&Query%2FScenario%5Bscenario%5D=Calendar&Query%2FScenario%5Blang%5D=en&wpRunQuery=true

Request: Add Special Attribute "First Editor is" / "_FEDT"

On wikis, where articles for specific topic are mainly maintained by one person, but once in a while changed by admins or bots, showing the last Editor does not help in querying e.g. for all of that persons contributions in a specific category etc.

As I figured, this may only be a small code Extension in SMW_ParseData.php (and the language files), e.g. something like this (insert on line 255):

case '_FEDT' :
$user = User::newFromId( $title->getFirstRevision()->getUser() );
$value = SMWDIWikiPage::newFromTitle( $user->getUserPage() );
break;

Support unique constraint for property

It would be really useful if Semantic MediaWiki could support a unique constraint for a property. I would suggest a property setting like:

[[Has constraint::unique]]

The effect seems obvious, to require the property be unique.

I currently attack this problem in the Template and do an ask query for a property with count and if the value is not zero, I often display a warning about a duplicate being detected and might also put it in a category. It would be interesting if violating a constraint could do something like add to a category, but just enforcing the uniqueness would be a great start.

It's possible this has all been discussed but figured I would throw it out there for consideration.

Other constraints could be envisioned using this syntax.

SMW 1.9 upgrade troubleshooting guide when installing with Composer

After the upgrade of an existing SMW installation the following messages can occur (its depends solely on the LocalSettings configuration):

$smwgNamespacesWithSemanticLinks usage

According to [a0, a1] which as been noted with the following contents:

---

Since we were forced to migrate to composer[1], on translatewiki.net SMW has stopped working completely for our main case, because properties are not set at all.[2] This means we're unable to manage our hundreds of i18n support requests;[3] encouraged by Jeroen, I created a test case which doesn't make use of templates[4] and I'm asking help on how to make it work. Initially we tried refreshing data as docs suggest and got stuck with the insuccess of the process[5] but local hacks brought the script to completion and still no properties are set from LQT namespaces.

We therefore think that $smwgNamespacesWithSemanticLinks is not working; I see the help page for it was created few days ago (thanks!)[6] but it doesn't really say how it interacts with composer. Nikerabbit set it via $wgExtensionFunctions, which docs declare "liable to fail in odd ways",[7] but was (allegedly) recommended by Jeroen and is not explained anywhere, nor I managed to get answers to questions about it since December. What to do? Our sysadmins have not had time to debug SMW in the last few months, I hope you can offer a solution they can apply.

Nemo

[1] Diff: https://git.wikimedia.org/commitdiff/translatewiki.git/2f13c8ef01e5c33d544c4d6699a3dad8c7097725
Current config: https://git.wikimedia.org/blob/translatewiki.git/HEAD/TranslatewikiSettings.php
Current composer.json: https://git.wikimedia.org/blob/translatewiki.git/HEAD/composer.json
[2] See for instance "Developer". https://translatewiki.net/wiki/Special:Properties
[3] https://translatewiki.net/wiki/Support/Open_requests
https://translatewiki.net/wiki/Template:Support
[4] https://translatewiki.net/w/i.php?title=Summary:Support/HTML_tags_or_wiki_text%3F
[5] https://bugzilla.wikimedia.org/show_bug.cgi?id=58969
[6] https://semantic-mediawiki.org/wiki/Help:$smwgNamespacesWithSemanticLinks
[7] https://www.mediawiki.org/wiki/Manual:$wgExtensionFunctions

---

[a0] https://bugzilla.wikimedia.org/show_bug.cgi?id=58969
[a1] http://p.defau.lt/?0iJEtsTkjCpwDIWF1UivQQ

1.9.0.2 release

• Update rel notes
• Bump version nr
• Create tag
• Create tarballs and upload to SF
• Award the barn-bunny of fluffiness to kgh

Migrate roadmap items onto this issue tracker

Migrate the items listed on our old roadmap onto this issue tracker. This results in a more holistic experience, allows people to filter based on tags (ie just show the easy items) and assign items to specific milestones.

1.9.1 release

• Rename 1.9.0.3 to 1.9.1 (since it has a new feature) including @since tags
• Finalize rel notes
• Update version number
• Create tag
• Create tarballs
• Update release template
• Announce on wiki
• Announce via mail
• Run composer update for smw.o
• Do a release tweet on Twitter

SMW\Store abstract vs. interface

Currently SMW\Store is specified as an abstract class which gives all sorts of trouble (e.g. being dependent on implementation details) when trying to mock the Store.

Naming issue

• http://verraes.net/2013/09/sensible-interfaces/

First proposal

Implement an interface called SMW\Store that specifies the required methods and move all operational code from the abstract class into a base class called SMW\BaseStore (or SMW\Store\BaseStore). Store implementation should derive from the BaseStore or implement SMW\Store.

interface Store {

}

abstract class BaseStore implements Store {

}

class SMWSQLStore3 extends BaseStore {

}

Second proposal

Leave the SMW\Store as is (as abstract class) and instead use a SMW\StoreInterface to specify its methods.

abstract class Store implements StoreInterface {

}