benlau / qsyncable

A solution of nested Json List Model

QSyncable - Synchronize data between models

QSyncable provides a QML friendly list model (based on QAbstractItemModel) as a wrapper of any data structure from C++ / Javascript. Instead, to access data from the original source per query, it duplicates a copy of data locally, and keep updated by an average O(n) synchronization algorithm. Every update is carried by passing a full copy of data snapshot. It will find out the diff and transform into a list of change operations like insertion, removal and move. It will guarantee the behaviour is identical to the original QML ListModel. Therefore, UI components could react to the changes correctly.

An immediate benefit of using QSycnable is the simplification of the data pipeline. If you need your UI to respond to changes like insertion/removal correctly, you must update the ListModel by the corresponding method explicitly. QSyncable combines all kinds of update methods into a single way. Such that user doesn’t need to care about their differences and setup data binding by just a single connection.

Moreover, QSyncable could also be used as a solution for the nested list model.

Reference:

1. Efficient models for QML with QSyncable - Google Slides

How does it work?

DiffRunner (QSDiffRunner) compares two QVariantList to produce a patch for transforming one of a list to another list with minimum no. of steps. The result can be applied on a QSListModel. DiffRunner uses an average O(n) algorithm and therefore it should be fast enough for regular UI application.

ListModel (QSListModel) is an implementation of QAbstactItemModel. It stores data in a list of QVariantMap. It will emit insert, remove, move and data changed signals according to the patch applied.

QSyncable provides the two classes above for a user to convert their own data structure to a QML friendly list model. Usually, there are several ways to update a list model. QSyncable combines all of the update methods into a single process - patching.

Whatever the data source has been changed, regardless of update method, the user converts it into a QVariantList and pass it to DiffRunner to compare with the current snapshot. Then apply the generated patch on QSListModel. UI component will be refreshed according to the emitted signals from QSListModel.

The diagram below shows an example application architecture using QSyncable:

In QSyncable application, ListModel only keeps a copy of the data. it is meaningless for UI components to modify it. Instead, UI components should ask to update the data source and trigger synchronization afterwards. The component for “updates” and “queries” is in fact separated. (More information in this article )

Design Principle - Separation of "updates" and "queries"

QSyncable is designed to solve a fundamental problem of C++ & QML application: How to share data between C++ and QML?

QObject list model is definitely a bad idea. It is terrible to manage their life cycle and ownership (QML / C++ scope). You should be aware of garbage collection in your QObject list model even it is written in C++.

Using a variant list model is better, but it is not C++ friendly. And it is difficult to handle nested list model.

In fact, the problem will be simple if you separate “updates” and “queries” into different components. First of all, you don’t even need to consider QObject list model approach. It has no any advantage of using QObject list model if you use another component for the update.First of all, you don’t even need to consider QObject list model approach. It has no any advantage of using QObject list model if you use other component for update.

Moreover, it is not necessary to use a variant list model as a central data source. You may use any data structure you like. Leave variant list model for presentation only.

QSyncable takes a step further. It simplifies the process to update the variant list model from a data source by combining insertion, removal, move and change operations into a single process - patching, while maintaining the correctness of UI response. It solves not only the problem of C++ and QML data sharing, but also a solution of nested list model within QML.

Reference:

1. What the Flux? (On Flux, DDD, and CQRS) — Jack Hsu

Why use QSyncable for C++?

(1) The function of QQmlListProperty is limited.

(2) Implement your own QAbstactItemModel is troublesome.

You need to understand how QAbstactItemModel works and emit insert, remove, move and update signals correctly. Otherwise, UI components like ListView will not react correctly.

(3) Use implicit sharing class over QObject

QObject is not copyable and you need to manage its life cycle. It is not really a good solution as a data storage class.

Implicit sharing class is recommended for this purpose. Because it can safely be copied across threads, like any other QVariant classes. Once you find that your data is too big for processing, you may pass it to a thread and work in the background.

Implicit Sharing | Qt Core 5.5

(4) Works for any data structure

You just need to write a conversion function, QSyncable will do the rest for you.

(5) Simple update method

No matter what kind of update happen, just convert your data structure to QVariantList, pass it to DiffRunner, then patch a model.

Why use QSyncable for QML?

(1) Use JsonListModel to wrap your Javascript object.

(2) Able to work as a nested list model.

(3) Simple data pipeline.

Algorithm

The average time complexity of the diff algorithm in QSDiffRunner is O(n), where n is the sum of the number of items in comparison. But it is not the worst case time complexity. It is O(n + m log m), where m is the no. of block move operations.

However, unless you are doing a reverse or random shuffle. Moving a single block of items, regardless of the number, from one to another place will remain in O(n) time complexity.

If you really need to do a reverse of a list or random shuffle with a large amount of data, you may consider to move it to a thread or set the key field to null. QSDiffRunner will ignore the insertion, removal, and move checking if the key field is not set.

Condition	Time Complexity
Insert to an empty list	O(1)
Clear the list	O(1)
Insert items to any position	O(n)
Remove a block of items	O(n)
Move a block of items	O(n)
Insert, remove, updates	O(n)
Reverse the list	O(n + m log m)
Random shuffle	O(n + m log m)

Installation

Download a release and bundle the folder within your source tree.

Or:

qpm install com.github.benlau.qsyncable

Class Reference

QSyncable - Class Reference

Example

faketrello - Simulate a Trello card board.

Future Development

1. FastDiffRunner

It is a diff runner to work with implicit sharing class (Immutable data type). It is able to achieve an O(1) comparison if data is not changed and it is not necessary to convert to QVariantMap type before comparison. It will be much faster than the original DiffRunner.

Link: FastDiffRunner Proposal

Status: Not started as the priority is low when compared to another project.

qsyncable's Issues

"QSDiffRunner.compare() - Duplicated or missing key" leads to endless loop.

Sometimes my program prints this. It happens very rarely but when it does happen the program freezes and uses 100% CPU forever. It seems to be getting suck inside a loop in the diff runner algorithm. I was able to obtain a backrace:

#0  0x00007fae8b4b1e1f in  () at /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#1  0x00007fae8b4a9279 in QVariant::QVariant(QVariant const&) () at /usr/lib/x86_64-linux-gnu/libQt5Core.so.5
#2  0x000056348bb41922 in QMapNode<QString, QVariant>::copy(QMapData<QString, QVariant>*) const ()
#3  0x000056348bb41991 in QMapNode<QString, QVariant>::copy(QMapData<QString, QVariant>*) const ()
#4  0x000056348bb41991 in QMapNode<QString, QVariant>::copy(QMapData<QString, QVariant>*) const ()
#5  0x000056348bb41991 in QMapNode<QString, QVariant>::copy(QMapData<QString, QVariant>*) const ()
#6  0x000056348bb4adee in QMap<QString, QVariant>::detach_helper() ()
#7  0x000056348bb51941 in QSDiffRunnerAlgo::compare(QList<QVariant> const&, QList<QVariant> const&) ()
#8  0x000056348bb404f5 in QSDiffRunner::compare(QList<QVariant> const&, QList<QVariant> const&) ()
#9  0x000056348bb556c0 in QSJsonListModel::sync() ()
#10 0x000056348bb557f3 in QSJsonListModel::setSource(QList<QVariant> const&) ()
#11 0x000056348bbf2f25 in QSJsonListModel::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#12 0x000056348bbf2fc3 in QSJsonListModel::qt_metacall(QMetaObject::Call, int, void**) ()
#13 0x00007fae8c12478e in QQmlVMEMetaObject::metaCall(QObject*, QMetaObject::Call, int, void**) () at /usr/lib/x86_64-linux-gnu/libQt5Qml.so.5
#14 0x00007fae8c13b242 in  () at /usr/lib/x86_64-linux-gnu/libQt5Qml.so.5
#15 0x00007fae8c138d04 in QQmlPropertyPrivate::write(QObject*, QQmlPropertyData const&, QVariant const&, QQmlContextData*, QFlags<QQmlPropertyData::WriteFlag>) () at /usr/lib/x86_64-linux-gnu/libQt5Qml.so.5
#16 0x00007fae8c1028d9 in QV4::QObjectWrapper::setProperty(QV4::ExecutionEngine*, QObject*, QQmlPropertyData*, QV4::Value const&) () at /usr/lib/x86_64-linux-gnu/libQt5Qml.so.5
#17 0x00007fae8d2c6540 in  ()
#18 0x0000000000000020 in  ()
#19 0x00032000000000c8 in  ()
#20 0x0000000000000000 in  ()

This is most likely caused by a transient error on the remote server. In any case, I think the algorithm should never get stuck regardless of what input is sent to it.

When call JsonListModel::clear(), it dosen't work for source

QSListModel::clear() is inherited by QSJsonListModel, but isn't overwritten or synchronized. So it resets QSListModel::m_storage but dosen't work for QSJsonListModel::m_source. Ofcourse, I can see QSJsonListModel dosen't call sync() when QSListModel::m_storage change,
you may have wanted it that way. By the way, I want QSJsonListModel::m_source has some way to be in sync with QSListModel::m_storage, for example overwrite and call QSListModel::submit().

screenshot contains non-free embedded ICC profile

The file <examples/faketrello/docs/screenshot.png> contains an ICC profile, copyrigh Apple Inc. with no licensing - i.e. that embedded code is copyright-protected and is not licensed for free redistribution or reuse.

Please strip the ICC profile from that image.

If curious (as others have been when I filed similar reports), the way I detect such issues is with the command-line tool exiftool part of the Perl library Image::ExifTool, available e.g. in Debian-based distributions as package libimage-exiftool-perl.

'faketrello' example feature req -- persist model to file, Board::load() recreates model

In faketrello example, Board::load() currently loads board statically from code. It would be a more complete example app if it showed "end to end" usage of the model, from initial creation, modification, and persistence on application close.

What's the simplest implementation of a "print method" for the model so that it can be persisted to a file? For example a method to output the current model as JSON?

How could Board::load() read such a file, and then reconstruct the model from the JSON-persisted file?

Rename JsonListModel

Hello, great lib as always ;)

But I'd like to suggest renaming JsonListModel to something else, or at least add a new name alias.
The reasoning behind this is that it has little to do with JSON, since you don't pass it a JSON string as a source, but a js array of objects (or a QVariantList from c++).

I don't have a lot of name ideas, maybe SyncableModel ?

[RFC] FastDiffRunner

Hello,

I am going to write a FastDiffRunner to offer a diff runner with better performance. And here is the proposed API. If anyone is interested, please feel free to comment and making suggestion.

Objective

The current implementation of DiffRunner requires the user to convert a C++ data structure into QVariantMap then perform a comparison. It is an easy to use solution and fast enough for the small amount of data. But since the average time complexity is O(n) even the data is unchanged. Therefore, it may be difficult to handle a huge amount of data.

To solve this problem, a new FastDiffRunner is proposed. It uses the implicitly shared data class (Immutable type) in Qt. The big-O to compare data without changes is O(1). And it doesn't convert data to QVariantMap for comparison. It should be much faster than the original DiffRunner.

Preparation

1. Immutable Data

FastDiffRunner only works with an immutable data type. User should create their data structure by File -> New File or Project -> C++ Class. Tick Include QShared Data in Qt Creator

Example (Generated by Qt Creator)

class CustomDataData;

class CustomData
{
public:
    CustomData();
    CustomData(const CustomData &);
    CustomData &operator=(const CustomData &);
    ~CustomData();

private:
    QSharedDataPointer<CustomDataData> data;
};

2. Implements isSharedWith() and key() function

class CustomDataData;

class CustomData
{
    Q_GADGET

public:
    CustomData();
    CustomData(const CustomData &);
    CustomData &operator=(const CustomData &);
    ~CustomData();

    bool isSharedWith(const CustomData& other);

    Q_INVOKABLE QVariant key(); // Optional function. 

private:
    QSharedDataPointer<CustomDataData> data;
};

inline bool CustomData::isSharedWith(const CustomData &other)
{
    return this->data == other.data;
}

3. Implements custom convertor function [Optional]

QVariantMap convert(const CustomData& data) {
  /// Insert your code here
}

If you have declared your property via Q_PROPERTY(), then it is not needed.

Usage

The interface of FastDiffRunner is almost same as the DiffRunner, expect you may need apply a custom data convertor function and doesn't need to set key field.

QList<CustomData> previous, current; 

// load previous and current 

QSFastDiffRunner<CustomData> runner(convert); 

QList<QSPatch> patches = runner.compare(previous, current);

runner.patch(listModel, patches);

Pros & Cons

Pros

1. O(1) comparison for unchanged data

Cons

1. It don't work for JavaScript

Request for Comment

If you have any suggestion toward the design, please feel free to leave a comment here. Thx

Discussion

Discussion 1

Interface Class

Should it use an abstract interface instead of just using Q_GADGET?

Example

class QSImmutable {
  Q_GADGET
public:
  bool iSharedWith(const QSImmutable*) = 0;
  QVariant key() const;
  bool hasKey() const; // User should implement this function if key is present in the data structure
}