halfak / deltas

A library for generating deltas of the difference between two sequences of tokens.

Deltas

An open licensed (MIT) library for performing generating deltas (A.K.A sequences of operations) representing the difference between two sequences of comparable tokens.

• Installation: pip install deltas
• Repo: http://github.com/halfak/Deltas
• Documentation: http://pythonhosted.org/deltas
• Note this library requires Python 3.3 or newer

This library is intended to be used to make experimental difference detection strategies more easily available. There are currently two strategies available:

deltas.sequence_matcher.diff(a, b):

A shameless wrapper around difflib.SequenceMatcher to get it to work within the structure of deltas.

deltas.segment_matcher.diff(a, b, segmenter=None):

A generalized difference detector that is designed to detect block moves and copies based on the use of a Segmenter.

Example:

>>> from deltas import segment_matcher, text_split >>> >>> a = text_split.tokenize("This is some text. This is some other text.") >>> b = text_split.tokenize("This is some other text. This is some text.") >>> operations = segment_matcher.diff(a, b) >>> >>> for op in operations: ... print(op.name, repr(''.join(a[op.a1:op.a2])), ... repr(''.join(b[op.b1:op.b2]))) ... equal 'This is some other text.' 'This is some other text.' insert ' ' ' ' equal 'This is some text.' 'This is some text.' delete ' ' ''

deltas's Issues

SegmentMatcher diffengine sometimes returns no operations

>>> from deltas import SegmentMatcher
>>> from deltas import wikitext_split
>>> sm = SegmentMatcher(tokenizer=wikitext_split)
>>> [len(item) for item in sm.process("")]
[]
>>> operations, a, b = sm.process("")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: need more than 0 values to unpack

Remove dependency on "nose" from "util.py"

The first line of the script has this: from nose.tools import eq_.
(I found this while running the example from Revision-Scoring)

Possible incompatibility with python 2.7

I get the following error,

import deltas.tokenizers

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/deltas/__init__.py", line 3, in <module>
    from .algorithms.diff_engine import DiffEngine
  File "/usr/local/lib/python2.7/dist-packages/deltas/algorithms/__init__.py", line 18, in <module>
    from .segment_matcher import SegmentMatcher
  File "/usr/local/lib/python2.7/dist-packages/deltas/algorithms/segment_matcher.py", line 20, in <module>
    from . import sequence_matcher
  File "/usr/local/lib/python2.7/dist-packages/deltas/algorithms/sequence_matcher.py", line 10, in <module>
    from ..tokenizers import text_split
  File "/usr/local/lib/python2.7/dist-packages/deltas/tokenizers/__init__.py", line 16, in <module>
    from .tokenizer import Tokenizer, RegexTokenizer
  File "/usr/local/lib/python2.7/dist-packages/deltas/tokenizers/tokenizer.py", line 6, in <module>
    from .token import Token
  File "/usr/local/lib/python2.7/dist-packages/deltas/tokenizers/token.py", line 8, in <module>
    class Token(str):
TypeError: Error when calling the metaclass bases
    nonempty __slots__ not supported for subtype of 'str'