mirkokiefer / syncing-thesis

Syncing strategies for mobile apps. A Protocol for Peer-to-Peer Data Synchronization.

Syncing Strategies for Mobile Devices

Histo: A Protocol for Peer-to-Peer Data Synchronization in Mobile Apps

Mirko Kiefer, published 2013 at Database Systems Research Group, Heidelberg University

Building collaborative apps that sync data among a group of users is hard.
The widespread adoption of mobile devices with limited network access requires the offline availability of data and apps.
Some aspects of syncing are often application specific and can therefore not be solved in a generic way.
However there are recurring patterns that can be used to build application specific solutions. The goal of this thesis is to develop a syncing framework that speeds up the development of collaborative apps.

Abstract

Inspired by distributed version control for source code we develop Histo, which is a framework for peer-to-peer data synchronization in mobile applications.

Our objectives when developing Histo are driven by exemplaric workflows in a collaborative task manager. We see offline support as a key requirement of modern mobile applications. Histo is designed to keep all data locally on the client device to ensure a user is not blocked from using an application.

Every application has a different data model, we develop methods to support flexible data models. We show that by mapping an application’s data to a hierarchical model, we can efficiently synchronize data.

Locking mechanisms are shown to be non-feasible when working with loosely connected devices. The synchronization protocol, which is at the heart of Histo, is therefore designed to work without any locking logic. We instead rely on a concept of optimistic synchronization, which guarantees eventual consistency of an application’s data. Handling concurrent edits and resulting conflicts correctly is another core element of Histo. Histo merges concurrent edits and identifies conflicts using three-way-merging. We find that the only way to implement three-way-merging in a distributed setting is by tracking the edit history on each device.

Histo’s synchronization protocol is shown to perform well in a range of network topolo- gies. The most extreme cases being client-server and peer-to-peer. For each network topology, we show how assumptions we make on the data history allows to minimize the amount of data stored on each device.

We focus on developing a practical solution that works on a broad range of devices. Histo is therefore implemented with open web standards. To our knowledge there are no publicly available solutions with these objectives.

We see potential to continue the work by exploring other application scenarios and adding support for different types of data. This could include conflict handling support for specialized data structures like those of a spreadsheet or a text editor.

• Abstract
• 1 Introduction
  • 1.1 Motivation
  • 1.2 Objectives and Approach
  • 1.3 Structure of the Thesis
• 2 Background
  • 2.1 Defining Data Synchronization
  • 2.2 Transactions and Consistency
  • 2.3 Stream-Based Synchronization
  • 2.4 History-Based Synchronization
  • 2.5 Three-Way Merging
  • 2.6 Lowest Common Ancestor
  • 2.7 Content Adressable Storage
  • 2.8 HTML5 and Offline Applications
    • 2.8.1 Web Storage
    • 2.8.2 Web SQL Database
    • 2.8.3 Indexed Database
    • 2.8.4 Cache Manifests
• 3 Application Scenario - A Collaborative Task Manager
  • 3.1 User Story 1: Creating Projects
  • 3.2 User Story 2: Creating and Editing Tasks
  • 3.3 User Story 3: Commenting on Tasks
  • 3.4 User Story 4: User Workflows
  • 3.5 Data Schema
• 4 Requirements
  • 4.1 Flexible Data Model Support
  • 4.2 Optimistic Synchronization
  • 4.3 Causality Preservation and Conflicts
  • 4.4 Flexible Network Topologies
  • 4.5 Integration with Existing Application Logic
  • 4.6 Cross-Platform
  • 4.7 Other Requirements
  • 4.8 Summary
• 5 Architecture of CouchDB
  • 5.1 Overview
  • 5.2 Synchronization Protocol
  • 5.3 Fulfillment of Requirements
• 6 Architecture of Histo
  • 6.1 Hierarchical Data Model Mapping
  • 6.2 Committing Objects
  • 6.3 Synchronization Protocol
    • 6.3.1 Update Detection and Propagation
    • Robustness
    • 6.3.2 Merging
    • Robustness
  • 6.4 Update Detection Between Branches
    • 6.4.1 Commit History Difference
    • 6.4.2 Application Data Change Detection
  • 6.5 Reconciliation Through Three-Way Merging
    • 6.5.1 Merge Scenario
    • 6.5.2 Differencing
    • 6.5.3 Diff Merging
    • 6.5.4 Patching
    • 6.5.5 Hierarchical Merging
  • 6.6 Handling Conflicts
  • 6.7 Synchronization Topologies
    • 6.7.1 Master - Client
    • 6.7.2 Client - Client
    • 6.7.3 Multi Master - Client
    • 6.7.4 Hierarchical
• 7 Realization
  • 7.1 Technology
  • 7.2 Fullfillment of Requirements
  • 7.3 Correctness and Robustness
• 8 Conclusion and Outlook
• Bibliography

syncing-thesis's Issues

Implement Tasks App with CouchDB

diff-merge-patch

Document the library and apply to "Models".

Real world examples

Hi Mirko,
I didn't read the whole thesis but I enjoyed skimming over some sections :) Nice work!
I wonder if you ever see a system using an approach like Histo? Do you have any experiences with it? And how would you solve it today? What kind of syncing would you choose for a real-world system?

Explain syncing problems in more detail

Use some of this as a basis:
http://culturedcode.com/things/blog/2010/12/state-of-sync-part-1.html

Content Addressable Store

explain and refer to examples like git

State-based synchronization

Most Recent Common Ancestor

describe problem and algorithm with graphics

Implement Tasks App with Histo

Histo could become a tool on its own or more like CouchDB plugin...

CouchDB architecture

• Describe high-level architecture of CouchDB (append-only, HTTP, replication)
• Analyse with regards to requirements of Syncing Engine