GithubHelp home page GithubHelp logo

matrix-org / sydent Goto Github PK

View Code? Open in Web Editor NEW
284.0 36.0 78.0 2.17 MB

Sydent: Reference Matrix Identity Server

Home Page: http://matrix.org

License: Apache License 2.0

Python 92.56% Shell 0.64% Dockerfile 0.42% HTML 0.23% Jinja 6.15%
python matrix hacktoberfest

sydent's Introduction

What is Sydent?

Sydent is an identity server for the Matrix communications protocol. It allows Matrix users to prove that they own an email address or phone number, and allows _other_ Matrix users to look them up using that email address or phone number.

Do I need to run Sydent to run my own homeserver?

Short answer: no.

Medium answer: probably not. Most homeservers and clients use the Sydent instance run by matrix.org, or use no identity server whatsoever.

Longer answer: if you want to allow user lookup via emails and phone numbers in a private federation of multiple homeservers, Sydent _might_ be useful for you. If you want your homeserver to be able to verify phone numbers via SMS and you have an API token for the OpenMarket HTTP SMS API, then Sydent might be useful for you.

Installation

Installing the system dependencies

To install Sydent's dependencies on a Debian-based system, run:

sudo apt-get install build-essential python3-dev libffi-dev \
                     sqlite3 libssl-dev python3-virtualenv libxslt1-dev

From here, you can either install Sydent by using a PyPI release, or by recreating Sydent's locked runtime environment.

Installing the latest Sydent release from PyPI

To create the virtual environment in which Sydent will run:

virtualenv -p python3 ~/.sydent
source ~/.sydent/bin/activate
pip install --upgrade pip
pip install --upgrade setuptools

Sydent and its dependencies can be installed using pip by running:

pip install matrix-sydent

With the virtualenv activated, you can run Sydent using:

python -m sydent.sydent

Installing from source

Alternatively, Sydent can be installed using poetry from a local git checkout. First install poetry. See poetry's documentation for details; we recommend installing via pipx. Once that's done:

git clone https://github.com/matrix-org/sydent.git
cd sydent
poetry install --no-dev
# For development, pull in extra tools with
# poetry install

To start Sydent:

poetry run sydent

Running Sydent

When Sydent is first run, it will create a configuration file in sydent.conf with some defaults. If a setting is defined in both the [DEFAULT] section and another section in the configuration file, then the value in the other section is used.

You'll most likely want to change the server name (server.name) and specify an email server (look for the settings starting with email.).

By default, Sydent will listen on 0.0.0.0:8090. This can be changed by changing the values for the configuration settings clientapi.http.bind_address and clientapi.http.port.

Sydent uses SQLite as its database backend. By default, it will create the database as sydent.db in its working directory. The name can be overridden by modifying the db.file configuration option. Sydent is known to be working with SQLite version 3.16.2 and later.

Listening for HTTPS connections

Most homeservers and clients will expect identity servers to be reachable using HTTPS.

Sydent does not currently support listening for HTTPS connection by itself. Instead, it is recommended to use a reverse proxy to proxy requests from homeservers and clients to Sydent. It is then possible to have this reverse proxy serve Sydent's API over HTTPS.

When using a reverse proxy, it is recommended to limit the requests proxied to Sydent to ones which paths start with /_matrix/identity for security reasons.

An exception to this is Sydent's internal replication API, see docs/replication.md.

SMS originators

Defaults for SMS originators will not be added to the generated config file, these should be added to the [sms] section of that config file in the form:

originators.<country code> = <long|short|alpha>:<originator>

Where country code is the numeric country code, or default to specify the originator used for countries not listed. For example, to use a selection of long codes for the US/Canada, a short code for the UK and an alphanumertic originator for everywhere else:

originators.1 = long:12125552368,long:12125552369
originators.44 = short:12345
originators.default = alpha:Matrix

Docker

A Dockerfile is provided for sydent. To use it, run docker build -t sydent . in a sydent checkout. To run it, use docker run --env=SYDENT_SERVER_NAME=my-sydent-server -p 8090:8090 sydent.

Persistent data

By default, all data is stored in /data. To persist this to disk, bind /data to a Docker volume.

docker volume create sydent-data
docker run ... --mount type=volume,source=sydent-data,destination=/data sydent

But you can also bind a local directory to the container. However, you then have to pay attention to the file permissions.

mkdir /path/to/sydent-data
chown 993:993 /path/to/sydent-data
docker run ... --mount type=bind,source=/path/to/sydent-data,destination=/data sydent

Environment variables

Warning

These variables are only taken into account at first start and are written to the configuration file.

Variable Name Sydent default Dockerfile default
SYDENT_SERVER_NAME empty empty
SYDENT_CONF sydent.conf /data/sydent.conf
SYDENT_PID_FILE sydent.pid /data/sydent.pid
SYDENT_DB_PATH sydent.db /data/sydent.db

Internal bind and unbind API

It is possible to enable an internal API which allows for binding and unbinding between identifiers and matrix IDs without any validation. This is open to abuse, so is disabled by default, and when it is enabled, is available only on a separate socket which is bound to localhost by default.

To enable it, configure the port in the config file. For example:

[http]
internalapi.http.port = 8091

To change the address to which that API is bound, set the internalapi.http.bind_address configuration setting in the [http] section, for example:

[http]
internalapi.http.port = 8091
internalapi.http.bind_address = 192.168.0.18

As already mentioned above, this is open to abuse, so make sure this address is not publicly accessible.

To use bind:

curl -XPOST 'http://localhost:8091/_matrix/identity/internal/bind' -H "Content-Type: application/json" -d '{"address": "[email protected]", "medium": "email", "mxid": "@matthew:matrix.org"}'

The response has the same format as /_matrix/identity/api/v1/3pid/bind.

To use unbind:

curl -XPOST 'http://localhost:8091/_matrix/identity/internal/unbind' -H "Content-Type: application/json" -d '{"address": "[email protected]", "medium": "email", "mxid": "@matthew:matrix.org"}'

The response has the same format as /_matrix/identity/api/v1/3pid/unbind.

Replication

It is possible to configure a mesh of Sydent instances which replicate identity bindings between each other. See docs/replication.md.

Discussion

Matrix room: #sydent:matrix.org.

sydent's People

Contributors

anoadragon453 avatar ara4n avatar azrenbeth avatar babolivier avatar benbz avatar callahad avatar clokep avatar dbkr avatar dependabot[bot] avatar dkasak avatar dklimpel avatar erikjohnston avatar gernest avatar h-shay avatar illicitonion avatar kittykat avatar lampholder avatar leonerd avatar michaelkaye avatar oddvar avatar pc25 avatar piotrkozimor avatar reivilibre avatar richvdh avatar sim6 avatar squahtx avatar sunweaver avatar t3chguy avatar tristanlins avatar turt2live avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sydent's Issues

Enabling `federation.verifycerts` breaks federation requests

Attempts at outbound federation when enabled results in:

ValueError: Specify client CA certificate information if and only if enabling certificate verification

This is due to setting CertificateOptions incorrectly at:

self._options = ssl.CertificateOptions(verify=config.getboolean("http", "federation.verifycerts"))

as twisted require if verify=True that you also specify CAs.

The project lacks documentation for implementing identity server in real life

I started the server with python -m sydent.sydent - everything seems to work but there is no info how I should configure web client or Android app to use the identity server.

Currently synapse gives this error:

2016-11-27 16:57:30,814 - synapse.http.server - 139 - ERROR - POST-53- Failed handle request synapse.http.server._async_render on <synapse.rest.ClientRestResource object at 0x7f5694cc1f90>: <SynapseRequest at 0x7f568ddf1518 method=POST uri=/_matrix/client/r0/account/3pid/email/requestToken clientproto=HTTP/1.1 site=8448>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/synapse/http/server.py", line 116, in wrapped_request_handler
    yield request_handler(self, request, request_metrics)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/http/server.py", line 255, in _async_render
    callback_return = yield callback(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/rest/client/v2_alpha/account.py", line 195, in on_POST
    ret = yield self.identity_handler.requestEmailToken(**body)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python2.7/dist-packages/synapse/handlers/identity.py", line 158, in requestEmailToken
    params
  File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/usr/lib/python2.7/dist-packages/synapse/http/client.py", line 128, in post_urlencoded_get_json
    defer.returnValue(json.loads(body))
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 505, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
JSONDecodeError: Expecting value: line 2 column 1 (char 1)

Should I specify port (8090) too or will the server detect itself? If the port is specified in web client then it starts to spin but there is no response nor anything interesting in the logs. Really no idea what I'm missing here..

Federation requests use deprecated code & get host header wrong

Federation requests now use new code from synapse to do the requests properly with .well-known lookups, but they also still call the old, simplistic implementation:

server = yield self._pickServer(domain)

This means the old impl replaces the domain with a server from an SRV lookup which means the new code can't do its job properly.

Looks like the call to _pickServer and all that code should just be removed and domain put straight into the URL, and 'https' replaced with 'matrix', and then it should work.

Practically, this is causing invites not to turn up on matrix.org (and oresumably any other servers with similar setup) because the onbind requests don't go through.

Support for MSC2140

Support the terms of service API for ISes, including:

  • /terms endpoint
  • OpenID registration
  • /v2/ api prefix with authentication and requirement that terms be agreed to

/lookup sometime returns bad signatures on vector.im

When looking up the MXID for an email I registered some time ago ([email protected], on 22/08/2018), and trying to validate the signature that Sydent returns in the response, I get Signature was forged or corrupt. The public key used to verify is ta8IQ0u1sp44HVpxYi7dFOdS/bfwDjcy4xLFlfY5KOA, which is the one currently shown when querying https://vector.im/_matrix/identity/api/v1/pubkey/ed25519:0.

I can reproduce this issue with all of my some-months-old 3PIDs, but if I register a new email address ([email protected], registered today around 2PM BST), the signature is correct.

Phone number as a 3PID

Firstly, thanks for your work.

Please could you add support to use phone number as a 3PID?

Thanks again.

Feature request: a way to manage your email address' association with mxids independently of a homeserver

Problems that can exist today:

  • email address is associated with multiple mxids (on multiple homeservers)
    • if the most recent? association is with an HS that doesn't support invites, invites to your email address will fail
    • email invites likely go to the wrong mxid whatever happens
  • email address is associated with a mxid on a dead HS
    • we try and fail to send the matrix invite without falling back to an email invite

run sydent error

(.sydent) [wangjh@localhost .sydent]$ python -m sydent.sydent
INFO:sydent.db.sqlitedb:Using DB file sydent.db
WARNING:sydent.http.httpcommon:No HTTPS private key / cert found: not starting replication server or doing replication pushes
INFO:sydent.http.httpserver:Starting Client API HTTP server on port 8090
INFO:twisted:Site starting on 8090
INFO:twisted:Starting factory <twisted.web.server.Site instance at 0x7f366710c2d8>
Unhandled error in Deferred:
CRITICAL:twisted:Unhandled error in Deferred:

CRITICAL:twisted:
Traceback (most recent call last):
File "/home/wangjh/.sydent/lib/python2.7/site-packages/twisted/internet/defer.py", line 149, in maybeDeferred
result = f(*args, **kw)
File "/home/wangjh/.sydent/lib/python2.7/site-packages/sydent/replication/pusher.py", line 76, in scheduledPush
peers = self.peerStore.getAllPeers()
File "/home/wangjh/.sydent/lib/python2.7/site-packages/sydent/db/peers.py", line 52, in getAllPeers
res = cur.execute("select p.name, p.port, p.lastSentVersion, pk.alg, pk.key from peers p, peer_pubkeys pk "
OperationalError: no such table: peers

What wrong?

ephemeral_public_keys table grows indefinitely

The ephemeral_public_keys table gains a new row whenever a 3pid invite (via email) is sent. The token is sent as part of the email and is validated upon clicking the link.

However, after validation we do not clear the token. While this allows invites to work multiple times (for some reason?) it does lead to an ever-growing table.

It's not unheard of to expire validation emails after a few hours or days, and we should probably do that here.

SMTPlib relies on unauthenticated mail relay services

In the current implementation (https://github.com/matrix-org/sydent/blob/master/sydent/util/emailutils.py#L65-L67) sydent uses only an unauthenticated mail delivery path via an available relay host. Unfortunately, most transactional email services like Cuttlefish or Mailgun only offer authenticated routes.

Are there plans to provide an immutable infrastructure / 12factor.net compliant modeling of configuration via environment variables that includes credentials similar to http://stackoverflow.com/a/64890/1959568?

Database empty at install

Hi Folks,

the database will be empty after an clean install.

pip install https://github.com/matrix-org/sydent/tarball/master

python -m sydent.sydent
INFO:sydent.db.sqlitedb:Using DB file sydent.db
WARNING:sydent.http.httpcommon:No HTTPS private key / cert found: not starting replication server or doing replication pushes
INFO:sydent.http.httpserver:Starting Client API HTTP server on port 8090
INFO:twisted:Site starting on 8090
INFO:twisted:Starting factory <twisted.web.server.Site instance at 0x7f003e44ed40>
Unhandled error in Deferred:
CRITICAL:twisted:Unhandled error in Deferred:

CRITICAL:twisted:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/sydent/replication/pusher.py", line 76, in scheduledPush
peers = self.peerStore.getAllPeers()
File "/usr/local/lib/python2.7/dist-packages/sydent/db/peers.py", line 52, in getAllPeers
res = cur.execute("select p.name, p.port, p.lastSentVersion, pk.alg, pk.key from peers p, peer_pubkeys pk "
OperationalError: no such table: peers

VersionConflict: (six 1.8.0 (/usr/lib/python2.7/dist-packages), Requirement.parse('six>=1.9'))

When trying to build a docker image for sydent, the following error occurs during build time:

Processing /sydent-master
    Complete output from command python setup.py egg_info:

    Installed /tmp/easy_install-kJtNbm/mock-2.0.0/.eggs/pbr-1.10.0-py2.7.egg

    Installed /tmp/pip-WHdttC-build/.eggs/mock-2.0.0-py2.7.egg
    Searching for setuptools_trial
    Reading https://pypi.python.org/simple/setuptools_trial/
    Best match: setuptools-trial 0.6.0
    Downloading https://pypi.python.org/packages/6c/d8/2ec30d50a37c90f8b48b22a50e96d4c50fec7e05ca1557bb814dc219ffc3/setuptools_trial-0.6.0.tar.gz#md5=bca42190adf631b213846c2b33f3676f
    Processing setuptools_trial-0.6.0.tar.gz
    Writing /tmp/easy_install-1I6r7A/setuptools_trial-0.6.0/setup.cfg
    Running setuptools_trial-0.6.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-1I6r7A/setuptools_trial-0.6.0/egg-dist-tmp-a4zumg
    zip_safe flag not set; analyzing archive contents...
    Moving setuptools_trial-0.6.0-py2.7.egg to /tmp/pip-WHdttC-build/.eggs

    Installed /tmp/pip-WHdttC-build/.eggs/setuptools_trial-0.6.0-py2.7.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-WHdttC-build/setup.py", line 48, in <module>
        long_description=read("README.rst"),
      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 269, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 313, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 826, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1085, in best_match
        dist = working_set.find(req)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 695, in find
        raise VersionConflict(dist, req)
    pkg_resources.VersionConflict: (six 1.8.0 (/usr/lib/python2.7/dist-packages), Requirement.parse('six>=1.9'))

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-WHdttC-build/
The command '/bin/sh -c curl -fSL https://github.com/matrix-org/sydent/archive/$BV_SYN.zip -o s.zip     && unzip s.zip     && rm s.zip     && cd /sydent-$BV_SYN     && pip install .     && echo $BV_SYN > 
/sydent.version     && rm -rf /sydent-$BV_SYN' returned a non-zero code: 1

The expected behaviour would be to install just fine.

Resources

Add Russia to supported countries for Matrix Identity server in SMS gateway with +7 phone number code

Seems OpenMarket already have support for send SMS to Russia with +7 phone number code: https://www.openmarket.com/docs/Content/globalcoverage/country/russia.htm
But in Matrix we see the error: Phone numbers in this country are not currently supported (Status 400).

Seems that Matrix list of supported countries now is hard-coded in Sydent:

originators.1 = long:12125552368,long:12125552369
originators.44 = short:12345
originators.default = alpha:Matrix

So maybe simply adding +7 to supported codes will add Russia support?

running issue

hi,
i got this error..

CRITICAL:twisted:Unhandled Error
Traceback (most recent call last):
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 571, in dataReceived
why = self.lineReceived(line)
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/web/http.py", line 1811, in lineReceived
self.allContentReceived()
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/web/http.py", line 1906, in allContentReceived
req.requestReceived(command, path, version)
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/web/http.py", line 771, in requestReceived
self.process()
--- ---
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/web/server.py", line 190, in process
self.render(resrc)
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/web/server.py", line 241, in render
body = resrc.render(self)
File "/home/udee/.sydent/local/lib/python2.7/site-packages/twisted/web/resource.py", line 250, in render
return m(request)
File "/home/udee/.sydent/local/lib/python2.7/site-packages/sydent/http/servlets/init.py", line 69, in inner
return json.dumps(f(*args, **kwargs)).encode("UTF-8")
File "/home/udee/.sydent/local/lib/python2.7/site-packages/sydent/http/servlets/emailservlet.py", line 55, in render_POST
email, clientSecret, sendAttempt, nextLink, ipaddress=ipaddress
File "/home/udee/.sydent/local/lib/python2.7/site-packages/sydent/validators/emailvalidator.py", line 58, in requestToken
sendEmail(self.sydent, 'email.template', emailAddress, substitutions)
File "/home/udee/.sydent/local/lib/python2.7/site-packages/sydent/util/emailutils.py", line 58, in sendEmail
mailString = open(mailTemplateFile).read() % allSubstitutions
exceptions.TypeError: float argument required, not dict

anyone know ?

Private contact discovery

Description

This week, Signal released an implementation of "private contact discovery". The goal of such as mechanism is to keep the user contact lists hidden from the Signal service. They use SGX to attest that the Open Source server implementation that can reviewed and tested is the one deployed in the real service. For more information, look at https://signal.org/blog/private-contact-discovery/

I was wondering if this is something Matrix could implement too: if the Home Server can make use of SGX (i.e. it is available in the hosting machine).

Email templates need fixing

  • Not all email template values are in the default config (part of #101)
  • email.email.template and email.email.invite_template config values seem to be the same thing?

"no such table"

Unfortunately, I'm currently getting this error during installation and I'm not sure why.

python -m sydent.sydent

2018-08-05 23:33:35,619 - __main__ - 116 - INFO - Starting Sydent server
2018-08-05 23:33:35,619 - sydent.db.sqlitedb - 29 - INFO - Using DB file sydent.db
2018-08-05 23:33:35,620 - sydent.db.sqlitedb - 43 - INFO - Running schema files...
2018-08-05 23:33:35,621 - sydent.db.sqlitedb - 73 - INFO - Migrating schema from v0 to v1
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/.sydent/lib/python2.7/site-packages/sydent/sydent.py", line 216, in <module>
    syd = Sydent()
  File "/root/.sydent/lib/python2.7/site-packages/sydent/sydent.py", line 123, in __init__
    self.db = SqliteDatabase(self).db
  File "/root/.sydent/local/lib/python2.7/site-packages/sydent/db/sqlitedb.py", line 40, in __init__
    self._upgradeSchema()
  File "/root/.sydent/local/lib/python2.7/site-packages/sydent/db/sqlitedb.py", line 76, in _upgradeSchema
    cur.execute("ALTER TABLE local_threepid_associations RENAME TO old_local_threepid_associations");
sqlite3.OperationalError: no such table: local_threepid_associations

support for MSC1711

enforce proper cert checking. Blocked on getting people to switch over for now, but we could make it optional.

/lookup requests don't seem to work

/lookup doesn't seem to work while /bulk_lookup does.

/lookup pulls an sgAssoc from the DB while /bulk_lookup just pulls an MXID associated with a address. An sgAssoc looks like the following:

{
  "address": "[email protected]",
  "medium": "email",
  "mxid": "@ears:matrix.org",
  "not_before": 1428825849161,
  "not_after": 4582425849161,
  "ts": 1428825849161,
  "signatures": {
    "matrix.org": {
      "ed25519:0": "ENiU2YORYUJgE6WBMitU0mppbQjidDLanAusj8XS2nVRHPu+0t42OKA/r6zV6i2MzUbNQ3c3MiLScJuSsOiVDQ"
    }
  }
}

/lookup grabs an assoc from the DB with the following code:

sgassoc = globalAssocStore.signedAssociationStringForThreepid(medium, address)

Which calls this SQL:

def signedAssociationStringForThreepid(self, medium, address):
cur = self.sydent.db.cursor()
# We treat address as case-insensitive because that's true for all the threepids
# we have currently (we treat the local part of email addresses as case insensitive
# which is technically incorrect). If we someday get a case-sensitive threepid,
# this can change.
res = cur.execute("select sgAssoc from global_threepid_associations where "
"medium = ? and lower(address) = lower(?) and notBefore < ? and notAfter > ? "
"order by ts desc limit 1",
(medium, address, time_msec(), time_msec()))
row = res.fetchone()
if not row:
return None
sgAssocBytes = row[0]
return sgAssocBytes

My hunch is sgAssocs aren't getting properly added to the DB.

unicode errors in logs

Unhandled Error
Traceback (most recent call last):
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/web/http.py", line 1656, in lineReceived
    self.allContentReceived()
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/web/http.py", line 1731, in allContentReceived
    req.requestReceived(command, path, version)
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/web/http.py", line 827, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/web/server.py", line 189, in process
    self.render(resrc)
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/web/server.py", line 238, in render
    body = resrc.render(self)
  File "/home/vectoris/venv-sydent/local/lib/python2.7/site-packages/twisted/web/resource.py", line 250, in render
    return m(request)
  File "sydent/http/servlets/lookupservlet.py", line 42, in render_GET
    sgassoc = globalAssocStore.signedAssociationStringForThreepid(medium, address)
  File "sydent/db/threepid_associations.py", line 73, in signedAssociationStringForThreepid
    (medium, address, time_msec(), time_msec()))
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Running with HTTPS

Is it possible to run it with HTTPS? I deployed identity server in localhost, then deployed Matrix server and specified identity URL in homeserver.yaml:

trusted_third_party_id_servers:
- localhost:8090

After registration attempt Matrix crashing with:

2018-01-22 09:57:53,715 - synapse.http.client - 112 - INFO - POST-1- Error sending request to POST https://localhost:8090/_matrix/identity/api/v1/validate/msisdn/requestToken: ConnectError Cannot assign requested address

Is it possible to specify in Matrix settings to connect to identity server via HTTP? Or how to run identity via HTTPS?

DatabaseError: malformed database schema (medium_lower_address)

Trying to run sydent on Debian Jessie using SQLite version 3.18.0 results in an error.

This has previously been mentioned in #26, however the original issue is different to this one, so logging it as a new issue.

(sydent)gooseberry@servername:/opt/sydent# python -m sydent.sydent
INFO:sydent.db.sqlitedb:Using DB file sydent.db
WARNING:sydent.http.httpcommon:No HTTPS private key / cert found: not starting replication server or doing replication pushes
INFO:sydent.http.httpserver:Starting Client API HTTP server on port 8090
INFO:twisted:Site starting on 8090
INFO:twisted:Starting factory <twisted.web.server.Site instance at 0x7fda3b6c2950>
Unhandled error in Deferred:
CRITICAL:twisted:Unhandled error in Deferred:

CRITICAL:twisted:
Traceback (most recent call last):
File "/opt/sydent/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/opt/sydent/local/lib/python2.7/site-packages/sydent/replication/pusher.py", line 76, in scheduledPush
peers = self.peerStore.getAllPeers()
File "/opt/sydent/local/lib/python2.7/site-packages/sydent/db/peers.py", line 52, in getAllPeers
res = cur.execute("select p.name, p.port, p.lastSentVersion, pk.alg, pk.key from peers p, peer_pubkeys pk "
DatabaseError: malformed database schema (medium_lower_address) - near "(": syntax error
^CINFO:twisted:Received SIGINT, shutting down.

Config file values need explanation

Currently the config file is a bit of a mess. It has a collection of "default" values, but this is not the complete list of values. They are in an odd order (email things are not together for instance), and figuring out what each value does requires looking in the code (and there are no examples anywhere).

Ideally we'd have explanations in the generated config file, similar to Synapse, with examples for each one, and it would be organised better (correct ordering).

This isn't that hard, but someone needs to go ahead and do it, else every time someone sets up a Sydent dev environment they spent a couple hours figuring out how the config file works.

Remote replication gets wedged when a single push to a server fails

When a Sydent is replicating to a few different servers, it does so sequentially. If one of those operations raises an exception, Sydent stops and doesn't continue pushing to the others. This can make it seem like a lot of the replication is broken, when actually it's just one server.

We should print the exception but continue iterating.

Uppercase in username

Hi everyone,
I don't know if this is the right place to ask, and I don't even know if what I am reporting really is a bug. If I'm mistaken, I apologise in advance.

My problem

By mistake, I registered (some time ago) on the matrix.org server a username with uppercase and lowercase letters; let's say Me. It seems that the login interface is case-insensitive, because I am able to log in with username me on Riot web, Riot Android and Riot desktop (for Linux).
However, the Matrix ID appears to be case-sensitive: people can send me messages at @Me:matrix.org but I never received anything sent to @me:matrix.org.

What I would like

What I really wanted was @me:matrix.org, but I am unable to register the me username, since the only registration interface I know is the one provided through Riot (which is case-insensitive). Moreover, the last time I checked, the ability to change a Matrix ID was still at the discussion stage (I don't even know if it's on the roadmap).

My questions

  1. Are Matrix IDs meant to be case-sensitive?
  2. How can I register me, i.e. a lowercase version of my current username?

Error Trying to Send Verification Email

Hi,

I have Synapse, Sydent and Riot (Web) running. Creating a new account on Riot, Sydent sends a verification email right?.

Well I'm stuck there because of the following error

2019-03-01 10:01:11,782 - twisted - 154 - CRITICAL - Traceback (most recent call last): File "/home/<user>/.sydent/lib/python2.7/site-packages/twisted/web/server.py", line 199, in process self.render(resrc) File "/home/<user>/.sydent/lib/python2.7/site-packages/twisted/web/server.py", line 259, in render body = resrc.render(self) File "/home/<user>/.sydent/lib/python2.7/site-packages/twisted/web/resource.py", line 250, in render return m(request) File "/home/<user>/.sydent/lib/python2.7/site-packages/sydent/http/servlets/__init__.py", line 69, in inner return json.dumps(f(*args, **kwargs)).encode("UTF-8") File "/home/<user>/.sydent/lib/python2.7/site-packages/sydent/http/servlets/emailservlet.py", line 55, in render_POST email, clientSecret, sendAttempt, nextLink, ipaddress=ipaddress File "/home/<user>/.sydent/lib/python2.7/site-packages/sydent/validators/emailvalidator.py", line 38, in requestToken logger.basicConfig(filename='sydent.log',level=logging.DEBUG) AttributeError: 'Logger' object has no attribute 'basicConfig'
I can't understand the problem of the logger, the python logger has basicConfig and the emailvalidator.py has the logger. I already changed the path of the log for and a specific path.

Any help is appreciated

Error while running python -m sydent.sydent

(.sydent) [root@izj6cbfvkmy1wfsmnlq8snz ~]# python -m sydent.sydent
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/root/.sydent/lib/python2.7/site-packages/sydent/sydent.py", line 43, in <module>
    from http.servlets.msisdnservlet import MsisdnRequestCodeServlet, MsisdnValidateCodeServlet
  File "/root/.sydent/lib/python2.7/site-packages/sydent/http/servlets/msisdnservlet.py", line 110
    msg: "Verification failed: Your request was invalid."
       ^
SyntaxError: invalid syntax

Installation fails with "Python.h: No such file or directory"

OK, so

add-apt-repository -s "https://matrix.org/packages/debian/ testing main"
gpg --keyserver pgp.mit.edu --recv-keys AD0592FE47F0DF61
apt-get install matrix-synapse nodejs nodejs-legacy npm
apt-get syutil
mkdir -p /opt
cd /opt
git clone https://github.com/matrix-org/sydent.git
cd /opt/sydent
apt install virtualenv
virtualenv env
source env/bin/activate
pip install setuptools_trial mock    # this fails
python setup.py test                      # this fails too

So the first failure is

  Failed building wheel for Twisted
Command "/opt/sydent/env/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-Fmu_QK/Twisted/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-Utbikr-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/sydent/env/include/site/python2.7/Twisted" failed with error code 1 in /tmp/pip-build-Fmu_QK/Twisted/

Here the full log. sydent.log.txt
My suggestions:

  • please kindly update the readme for sydent
  • ideally package up sydent as well

Implement MSC 1915 unbind via sid

MSC1915 says that an unbind can either be authenticated by a) checking that the request is signed by the HS (done) or b) that it includes a sid and client_secret that demonstrates that the requester owns the 3PID in question

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.