GithubHelp home page GithubHelp logo

ma1uta / ma1sd Goto Github PK

View Code? Open in Web Editor NEW

This project forked from kamax-matrix/mxisd

165.0 165.0 55.0 1.67 MB

Federated Matrix Identity Server (formerly fork of kamax/mxisd)

License: GNU Affero General Public License v3.0

Dockerfile 0.05% Shell 0.76% Java 99.04% HTML 0.15%

ma1sd's People

Contributors

aaronraimist avatar abeluck avatar adrnam avatar doofy avatar eyecreate avatar higgs1 avatar joshuaboniface avatar kiorky avatar lub avatar ma1uta avatar mattcen avatar maxidorius avatar mrjohnson22 avatar ne0sight avatar nullisnot0 avatar q-wertz avatar stygianguest avatar teutat3s avatar thegcat avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ma1sd's Issues

Update to 1.9.0 breaks authentication via LDAP (ma1sd / rest client)

Not sure whether this is a synapse issue or a ma1sd issue, so I cross-post this here as well:

Description

After updating synapse to 1.9.0 no user can login. Users are stored in a LDAP database, authentication via REST-API / ma1sd.

Steps to reproduce

Open Riot Web
Log in with any existing user
Error message: Fehler: Problem bei der Kommunikation mit dem angegebenen Home-Server. (M_UNKNOWN)

Version information

Synapse 1.9.0 installed via official repo.
Ubuntu 18.04 LTS

Log file of the issue:

2020-01-23 14:09:46,680 - synapse.access.https.8448 - 233 - INFO - POST-44 - 87.79.201.63 - 8448 - Received request: POST /_matrix/client/r0/login
2020-01-23 14:09:46,681 - synapse.rest.client.v1.login - 176 - INFO - POST-44 - Got login request with identifier: {'type': 'm.id.user', 'user': 'admin'}, medium: None, address: None, user: None
2020-01-23 14:09:46,681 - rest_auth_provider - 46 - INFO - POST-44 - Got password check for @admin:jaychat.de
2020-01-23 14:09:46,695 - rest_auth_provider - 62 - INFO - POST-44 - User @admin:jaychat.de authenticated
2020-01-23 14:09:46,699 - rest_auth_provider - 76 - INFO - POST-44 - User @admin:jaychat.de already exists, registration skipped
2020-01-23 14:09:46,699 - rest_auth_provider - 79 - INFO - POST-44 - Handling profile data
2020-01-23 14:09:46,700 - synapse.http.server - 110 - ERROR - POST-44 - Failed handle request via 'LoginRestServlet': <XForwardedForRequest at 0x7f18edeedda0 method='POST' uri='/_matrix/client/r0/login' clientproto='HTTP/1.0' site=8448>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: {}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: ('@admin:jaychat.de', '$2b$12$YEY5SVN7Yy43yFq3f6saLuAR/uaNK5m4Res6hgnLLc/70KODPt7ZW')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: @admin:jaychat.de

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 78, in wrapped_request_handler
    await h(self, request)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/http/server.py", line 331, in _async_render
    callback_return = await callback_return
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v1/login.py", line 150, in on_POST
    result = await self._do_other_login(login_submission)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/rest/client/v1/login.py", line 281, in _do_other_login
    identifier["user"], login_submission
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/synapse/handlers/auth.py", line 622, in validate_login
    is_valid = yield provider.check_password(qualified_user_id, password)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/opt/venvs/matrix-synapse/lib/python3.6/site-packages/rest_auth_provider.py", line 82, in check_password
    store = yield self.account_handler.hs.get_profile_handler().store
AttributeError: 'ModuleApi' object has no attribute 'hs'

Wrong handling of token expiration

Hello @ma1uta,
First thank you for your contribution and your fork !
While testing your recent commits I came across the following error when requesting the /_matrix/identity/v2/hash_details endpoint:
[XNIO-1 I/O-2] ERROR io.kamax.mxisd.http.undertow.handler.AuthorizationHandler - Account for '@mjattiot:agoria.dev.opensense.io' from: agoria.dev.opensense.io
[XNIO-1 I/O-2] ERROR io.undertow.request - UT005071: Undertow request failed HttpServerExchange{ GET /_matrix/identity/v2/hash_details} io.kamax.mxisd.exception.InvalidCredentialsException: Supplied credentials are invalid

To my understanding, it comes from and error in the following code and line:

if (account.getExpiresIn() < System.currentTimeMillis()) {

You are comparing the "expires_in" key coming from the /openid/request_token endpoint (which default to 3600 sec) to System.currentTimeMillis() (which is a timestamp). As such account.getExpiresIn() is always lesser than System.currentTimeMillis() at line 61 and always response with invalid credentials.

Could you patch this up ?

Thank you !

Issue with auth

Still trying to get this working....
When logging in with an email I see this in the log (emails and urls sanitised)
Related to #18 (which is still causing me issues)

When a 3PID resolved in more than one backend, what should happen?

In this case [email protected] exists in both the Synapse and Sql databases.
As they are different systems, it resolves to different MatrixIDs
@UserName:matrix.mysite.org.uk
and
@uname:matrix.mysite.org.uk

But it tries to login to the wrong one, does not try the other. Accepts the password then glibly fails with 403 and no further explanation.
I would expect it to create the non-existent MatrixID

[XNIO-1 task-5] INFO io.kamax.mxisd.auth.AuthManager - Proxy resolution: http://matrix.mysite.org.uk/_matrix/client/r0/login to http://localhost:8008/_matrix/client/r0/login
[XNIO-1 task-7] INFO io.kamax.mxisd.auth.AuthManager - Login request with medium 'email' and address '[email protected]'
[XNIO-1 task-7] INFO io.kamax.mxisd.backend.sql.SqlThreePidProvider - SQL lookup
[XNIO-1 task-7] INFO io.kamax.mxisd.backend.sql.SqlThreePidProvider - SQL query: SELECT 'email' as medium, username as uid FROM auth_fullnames WHERE medium = ? and address = ?
[XNIO-1 task-7] INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 2, acquireRetryAttempts -> 10,
  acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null,
  connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> oksi7pac97w6yb129aiab|42a0cc7d,
  debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> null, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false,
  forceSynchronousCheckins -> false, forceUseNamedDriverClass -> false, identityToken -> mksk7pac97w6yb765aiab|41a0bb7d, idleConnectionTestPeriod -> 0, initialPoolSize -> 3,
  jdbcUrl -> jdbc:sqlite:/opt/cambsac/cambsacdb, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 10,
  maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 1, numHelperThreads -> 3, preferredTestQuery -> null, privilegeSpawnedThreads -> false, properties -> {},
  propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {},
  usesTraditionalReflectiveProxies -> false ]

[XNIO-1 task-7] INFO io.kamax.mxisd.backend.sql.SqlThreePidProvider - Found match: UserName
[XNIO-1 task-7] INFO io.kamax.mxisd.backend.sql.SqlThreePidProvider - Resolving as localpart
[XNIO-1 task-7] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Found 3PID mapping: {medium: 'email', address: '[email protected]', mxid: '@UserName:matrix.mysite.org.uk'}
[XNIO-1 task-7] INFO io.kamax.mxisd.auth.AuthManager - Proxy resolution: http://matrix.mysite.org.uk/_matrix/client/r0/login to http://localhost:8008/_matrix/client/r0/login
[XNIO-1 task-8] INFO io.kamax.mxisd.http.undertow.handler.auth.RestAuthHandler - Requested to check credentials for @UserName:matrix.mysite.org.uk
[XNIO-1 task-8] INFO io.kamax.mxisd.auth.AuthManager - Attempting authentication with store ExecAuthStore
[XNIO-1 task-8] INFO io.kamax.mxisd.backend.exec.ExecAuthStore - Performing authentication for @UserName:matrix.mysite.org.uk
[XNIO-1 task-8] INFO io.kamax.mxisd.backend.exec.ExecStore - Executing /etc/ma1sd/wagtail_auth.sh
[XNIO-1 task-8] INFO io.kamax.mxisd.auth.AuthManager - @UserName:matrix.mysite.org.uk was authenticated by ExecAuthStore, publishing 3PID mappings, if any
[XNIO-1 task-7] INFO io.kamax.mxisd.auth.AuthManager - http status = 403

Black- and Whitelist patterns are not useful

You should not use Pattern.compile when you did replace the pattern, which a user did define.

Please find a solution, so we can use the Pattern.compile functionality to setup more flexible solutions for white- and blacklisting, because at the moment, the blacklist is not applicable, if you want blacklist a whole toplevel as an example. And so on, you cant whitelist a whole country or whitelist everyone.

Please fill in a special char or a config variable, which indicates that we can use the pattern natively to Pattern.compile instead of your replacing function.

MSC2134 hash lookup database multiple 3pids mapping to a single mxid

Hello,

I noticed a unique key on mxid in table hashes. That way only one 3pid can be assigned to a single mxid.
I have replaced that index with
CREATE UNIQUE INDEX index_hashes_unique_1 ON hashes USING btree (mxid, medium, address);
and it seems fine for the time being.

Kind regards,
Stefan Nikolov

Missing 3PIDs in riot-web

I would like to point out a problem that may be due to a bug in ma1sd.

The original issue was first reported in the riot-web bug list, see here: https://github.com/vector-im/riot-web/issues/13033#

In short, the problem is that no 3PIDs are shown in the settings page of riot-web, as well as in riotX Android app. In the old riot Android app, 3PIDs are shown though.

When I noticed the followin error in the browsers javascript console:

Access to XMLHttpRequest at 'https://matrix.mydomain.de/_matrix/identity/v2/account/register' from origin 'https://riot.mydomain.de' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

it seemed to me the problem might be caused by a wrongly configured reverse proxy.

I contacted the developer of matrix-docker-ansible-deploy, as my installation is done via his script. You can find my report in his issue list at https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/505

It turned out that when riot-web accessed the identity server, an Access-Control-Allow-Origin header was missing, which denied access.
This problem was finally fixed, see https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/511

The reason why I am addressing you is that the absence of the old bug now reveals a new one: in the current version I get the message:
Unable to reach identity server at https://matrix.mydomain.de to check for 3PIDs bindings in Settings
Error: Unsupported identity server: bad response

Since I use ma1sd as identity server and since e.g. the login works without problems I wonder if the described behaviour might be a bug in ma1sd.

It would be great if you could comment on this.

Best, Tom

Doesn't seem to work with recent versions of riot

I was able to connect fine with desktop (though never with the android client) when I first hooked ma1sd up, but at one point riot upgraded and since then if I or anyone on my server tries to connect it says "Terms of service not accepted or the identity server is invalid".

This has been the case for a while but I figured I'd try again after seeing the latest commit and tested with git master, unfortunately the problem is still there though.

I'm not really sure how to debug but would be happy to try if someone could point me in the right direction-- the server shows a successful launch but no logging output for connection attempts.

Thanks!

IPV4

I just installed, configured and run ma1sd. I notice it uses port 8090 on IPV6. How do I change it to IPV4?

LDAP Groups

hi
Is it somehow possible to link LDAP Groups with ma1sd and synapse? e.g.

  • to create Communities and their members based on LDAP Groups
  • create Rooms with Members based on LDAP Groups.
  • Using LDAP Groups as roles in ma1sd

Is it planned to implement such features?

kind regards,
dennis

v2.3.0 always responds with 405

After upgrading to v2.3.0 and using the postgres backend for internal storage, ma1sd responds to any HTTP GET request with a status code of 405 and with no body.

Running curl -i http://localhost:8090/<anything> prints the following:

HTTP/1.1 405 Method Not Allowed
Connection: keep-alive
Content-Length: 0
Date: <date>

The standard output of ma1sd from boot time to after receiving requests is below. No new output is printed after it receives a HTTP request.

[main] INFO io.kamax.mxisd.config.YamlConfigLoader - Reading config from /etc/ma1sd/ma1sd.yaml
[main] INFO io.kamax.mxisd.config.YamlConfigLoader - Loaded config from /etc/ma1sd/ma1sd.yaml
[main] INFO App - ma1sd starting
[main] INFO App - Version: 2.3.0
[main] INFO App - Logging config:
[main] INFO App -   Default log level: debug
[main] INFO App -   Logging level set by the configuration: debug
[main] INFO io.kamax.mxisd.config.DirectoryConfig - --- Account config ---
[main] INFO io.kamax.mxisd.config.DirectoryConfig - Allow registration only for trust domain: true
[main] INFO io.kamax.mxisd.config.DirectoryConfig - --- Directory config ---
[main] INFO io.kamax.mxisd.config.DirectoryConfig - Exclude:
[main] INFO io.kamax.mxisd.config.DirectoryConfig -   Homeserver: false
[main] INFO io.kamax.mxisd.config.DirectoryConfig -   3PID: false
[main] INFO io.kamax.mxisd.config.DnsOverwriteConfig - --- DNS Overwrite config ---
[main] INFO io.kamax.mxisd.config.DnsOverwriteConfig - Homeserver:
[main] INFO io.kamax.mxisd.config.DnsOverwriteConfig - 	Client: []
[main] INFO io.kamax.mxisd.config.DnsOverwriteConfig - 	Federation: []
[main] INFO io.kamax.mxisd.config.FirebaseConfig - --- Firebase configuration ---
[main] INFO io.kamax.mxisd.config.FirebaseConfig - Enabled: false
[main] INFO io.kamax.mxisd.config.InvitationConfig - --- Invite config ---
[main] INFO io.kamax.mxisd.config.InvitationConfig - Expiration: {"after":10080}
[main] INFO io.kamax.mxisd.config.InvitationConfig - Resolution: {"recursive":true,"timer":5}
[main] INFO io.kamax.mxisd.config.InvitationConfig - Policies: {"if_sender":{"has_role":[]}}
[main] INFO io.kamax.mxisd.config.ldap.LdapConfig - --- Generic LDAP Config ---
[main] INFO io.kamax.mxisd.config.ldap.LdapConfig - Enabled: false
[main] INFO io.kamax.mxisd.config.RecursiveLookupBridgeConfig - --- Bridge integration lookups config ---
[main] INFO io.kamax.mxisd.config.RecursiveLookupBridgeConfig - Enabled: false
[main] INFO io.kamax.mxisd.config.MatrixConfig - --- Matrix config ---
[main] INFO io.kamax.mxisd.config.MatrixConfig - Domain: <my-homeserver>
[main] INFO io.kamax.mxisd.config.MatrixConfig - Identity:
[main] INFO io.kamax.mxisd.config.MatrixConfig - 	Servers: {"matrix-org":["https://matrix.org"]}
[main] INFO io.kamax.mxisd.config.MatrixConfig - API v1: false
[main] INFO io.kamax.mxisd.config.MatrixConfig - API v2: true
[main] INFO io.kamax.mxisd.config.ldap.LdapConfig - --- NetIQ eDirectory Config ---
[main] INFO io.kamax.mxisd.config.ldap.LdapConfig - Enabled: false
[main] INFO io.kamax.mxisd.config.threepid.notification.NotificationConfig - --- Notification config ---
[main] INFO io.kamax.mxisd.config.threepid.notification.NotificationConfig - Handlers:
[main] INFO io.kamax.mxisd.config.threepid.notification.NotificationConfig -   msisdn: raw
[main] INFO io.kamax.mxisd.config.threepid.notification.NotificationConfig -   email: raw
[main] INFO io.kamax.mxisd.config.rest.RestBackendConfig - --- REST backend config ---
[main] INFO io.kamax.mxisd.config.rest.RestBackendConfig - Enabled: false
[main] INFO io.kamax.mxisd.config.SessionConfig - --- Session config ---
[main] INFO io.kamax.mxisd.config.SessionConfig - Global Policy: {"validation":{"enabled":true},"unbind":{"enabled":true,"notifications":true}}
[main] INFO io.kamax.mxisd.config.ServerConfig - --- Server config ---
[main] INFO io.kamax.mxisd.config.ServerConfig - Name: <my-id-server>
[main] INFO io.kamax.mxisd.config.ServerConfig - Port: 8090
[main] INFO io.kamax.mxisd.config.ServerConfig - Public URL: https://<my-id-server>
[main] INFO io.kamax.mxisd.config.sql.SqlConfig - --- Synapse SQL Provider config ---
[main] INFO io.kamax.mxisd.config.sql.SqlConfig - Enabled: false
[main] INFO io.kamax.mxisd.config.ViewConfig - --- View config ---
[main] INFO io.kamax.mxisd.config.ViewConfig - Session: {"local":{"on_token_submit":{}},"on_token_submit":{"failure":"classpath:/templates/session/tokenSubmitFailure.html","success":"classpath:/templates/session/tokenSubmitSuccess.html"}}
[main] INFO io.kamax.mxisd.config.PolicyConfig - --- Policy Config ---
[main] INFO io.kamax.mxisd.config.PolicyConfig - Empty
[main] INFO io.kamax.mxisd.config.HashingConfig - --- Hash configuration ---
[main] INFO io.kamax.mxisd.config.HashingConfig -    Pepper length: 20
[main] INFO io.kamax.mxisd.config.HashingConfig -    Rotation policy: per_requests
[main] INFO io.kamax.mxisd.config.HashingConfig -    Hash storage type: in_memory
[main] INFO io.kamax.mxisd.config.HashingConfig -    Rotation after requests: 10
[main] INFO io.kamax.mxisd.config.HashingConfig -    Algorithms: [sha256]
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ChangelogDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ThreePidInviteIO
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: HistoricalThreePidInviteIO
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ThreePidSessionDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ASTransactionDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: AccountDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: AcceptedDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Workaround for postgresql on dao: AcceptedDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Table exists, do nothing
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: HashDao
[main] INFO io.kamax.mxisd.storage.crypto.FileKeyStore - Key store is already in directory format
[main] INFO com.mchange.v2.c3p0.C3P0Registry - Initializing c3p0-0.9.5.4 [built 23-March-2019 23:00:48 -0700; debug? true; trace: 10]
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig - --- E-mail Generator templates config ---
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig - Invite: Built-in (/threepids/email/invite-template.eml)
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig - Session:
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig -   Validation: classpath:/threepids/email/validate-template.eml
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig -   Unbind:
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig -     Notification: classpath:/threepids/email/unbind-notification.eml
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - --- E-mail SMTP Connector config ---
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Host: localhost
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Port: 25
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - TLS Mode: 0
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Login: null
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Has password: false
[main] INFO io.kamax.mxisd.hash.HashEngine - Start update hashes.
[main] INFO io.kamax.mxisd.hash.HashEngine - Populate hashes from the handler: io.kamax.mxisd.lookup.provider.DnsLookupProvider
[main] INFO io.kamax.mxisd.hash.HashEngine - Populate hashes from the handler: io.kamax.mxisd.lookup.provider.ForwarderProvider
[main] INFO io.kamax.mxisd.hash.HashEngine - Finish update hashes.
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Found 2 providers
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy -   - io.kamax.mxisd.lookup.provider.DnsLookupProvider
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy -   - io.kamax.mxisd.lookup.provider.ForwarderProvider
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Recursive lookup enabled: true
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - 127.0.0.0/8 is allowed for recursion
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - 10.0.0.0/8 is allowed for recursion
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - 172.16.0.0/12 is allowed for recursion
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - 192.168.0.0/16 is allowed for recursion
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - ::1/128 is allowed for recursion
[main] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Hash lookups enabled: true
[main] INFO io.kamax.mxisd.profile.ProfileManager - Profile Providers:
[main] INFO io.kamax.mxisd.notification.NotificationManager - Found handler raw for medium email
[main] INFO io.kamax.mxisd.notification.NotificationManager - --- Notification handler ---
[main] INFO io.kamax.mxisd.notification.NotificationManager - 	Handler for email: raw
[main] DEBUG io.kamax.mxisd.invitation.InvitationManager - Loading saved invites
[main] INFO io.kamax.mxisd.invitation.InvitationManager - Loaded saved invites
[main] INFO io.kamax.mxisd.invitation.InvitationManager - Setting up invitation mapping refresh timer
[main] INFO io.kamax.mxisd.directory.DirectoryManager - Directory providers:
[main] INFO io.undertow - starting server: Undertow - 2.0.27.Final
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.xnio.nio.NioXnio$2 (file:/usr/lib/ma1sd/ma1sd.jar) to constructor sun.nio.ch.EPollSelectorProvider()
WARNING: Please consider reporting this to the maintainers of org.xnio.nio.NioXnio$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] INFO App - ma1sd started
[Timer-0] DEBUG io.kamax.mxisd.invitation.InvitationManager - Invite expiration: started
[Timer-0] DEBUG io.kamax.mxisd.invitation.InvitationManager - No invite to expired, skipping

Ability to change LDAP lookup attribute

When using the LDAP auth module, ma1sd looks into LDAP for "samaccountname" by default. When using FreeIPA (based on OpenLDAP), this account attribute is not part of the ldap schema.

Best fix would be to to change the default filter to be adjustable.

I attempted to add a default filter and it just added it in addition to looking for samaccountname.

Enabling exec for autentication causes null pointer log messages

I wrote a little script to hook exec to Django for authentication.
I put the following config line in ma1uta.yaml

exec:
  enabled: true  
  auth:  
    enabled: true
    command: '/etc/ma1sd/wagtail_auth.sh'
    args: ['{localpart}']
    input:
      type: 'plain'
      template: '{password}'

This caused a flurry of null pointer exceptions

I found that adding

  directory:
    enabled: false
  identity:
    enabled: false
  profile:
    enabled: false

Fixed it.

**
Conclusion:
Enabling a backend enables the sql queries for API aspects of that backend for which there are no usefull defaults or valid config. If there is no useful default or valid config, then perhaps default value for enabled parameter should be false. I think this same issue exists for other backends.
**

For completeness (as a useful wiki entry for others) here is the auth script.
in /etc/ma1sd/wagtail_auth.sh

#!/bin/bash
. /opt/site/pyenv/bin/activate
cd /opt/site

RES=$(./manage.py authenticate $1 <&0)

if [[ "$RES" == "OK" ]]; then
  exit 0
else
  exit 1
fi

And it's python couterpart in
/opt/site/app/base/management/commands

from django.core.management.base import BaseCommand, CommandError
from django.contrib.auth.models import User
from django.contrib.auth.hashers import check_password
import sys
from select import select

class Command(BaseCommand):
    help = "Command line verification of user password"

    def add_arguments(self, parser):
        parser.add_argument('username', type=str)
        parser.add_argument('password', type=str, nargs='?', default='')

    def handle(self, *args, **options):
        username = options['username']
        password = options['password']

        if not len(password):
            # Try STDIN
            rlist, _, _ = select([sys.stdin], [], [], 2)
            if rlist:
                password = sys.stdin.readline()

        if not len(password):
            res = False
            # raise CommandError('No password supplied')

        try:
            user = User.objects.get(username__iexact=username)
        except User.DoesNotExist:
            res = False
            # raise CommandError('User %s does not exist' % username)
        else:
            res = check_password(password.strip(), user.password)

        self.stdout.write("OK" if res else "BAD")

Unshared emails are treated as shared

Emails bound to a mxid are returned by /_matrix/identity/v2/lookup even though they were never validated for sharing through ma1sd, and despite the fact that session_3pid table of ma1sd's backend storage is blank.

This happens to me when using the synapseSql identity store, the postgresql backend, and the v2 API with hashing. I haven't tried other identity stores, and I am restricted to using the postgresql backend.

My config:

matrix:
  domain: 'domain'
  v1: true
  v2: true
  ## Remove default matrix-org server
  identity:
    servers:
      myOtherServers: []

server:
  name: 'identity.domain'

key:
  path: '/var/lib/ma1sd/keys'

storage:
  backend: postgresql
  provider:
    postgresql:
      database: '//localhost/ma1sd'
      username: 'user'
      password: 'pass'

threepid:
  medium:
    email:
      identity:
        from: "noreply-id@domain"
        name: "Name"
      connectors:
        smtp:
          host: "localhost"
          tls: 0
          port: 25

hashing:
  enabled: true
  pepperLength: 20
  rotationPolicy: per_requests
  hashStorageType: in_memory
  algorithms:
    - sha256

synapseSql:
  enabled: true
  type: 'postgresql'
  connection: '//localhost/synapse?user=synapse_user&password=pass'
  lookup:
    query: 'select user_id as mxid, medium, address from user_threepids'
  legacyRoomNames: false

logging:
  root: debug
  app: debug

Change ma1sd systemd service config to stop spamming syslog

I find the default ma1sd configuration for systemd annoying as it spams the syslog, which IMO should only log important system level messages.

The following change might help (at least Debian) users manage their system better,

/etc/systemd/system/ma1sd.service

[Unit]
Description=ma1sd
After=syslog.target

[Service]
User=ma1sd
Group=matrix-synapse
ExecStart=/bin/sh -c 'exec /usr/bin/ma1sd -v -c /etc/ma1sd/ma1sd.yaml >>/var/log/matrix-synapse/ma1sd.log 2>&1'

[Install]
WantedBy=multi-user.target

Forward Provider not working

Dear,

When using ma1sd together with Synapse, I enabled the forward provider in order to be able to invite people via their 3pid. It is apparently not working: when I invite someone with a matrix.org account (via the 3pid), the binding is identified correctly (mxid <> 3pid) but the invitation eventually fails (riot says that there was a failure to create the room).

See below relevant log files and config files:

ma1sd configuration:

directory:
  exclude:
    homeserver: true

dns:
  overwrite:
    homeserver:
      client:
        - name: 'matrix.nilux.be'
          value: 'http://synapse.nilux.be:8008'

enable_registration: false

forward:
  servers:
    - 'matrix-org'

hashing:
  enabled: true
  pepperLength: 20
  rotationPolicy: per_requests
  hashStorageType: sql
  algorithms:
    - sha256 
  requests: 10

key:
  path: '/var/ma1sd/secret'

ldap:
  enabled: true
  filter: '(memberOf=CN=nac-matrix,OU=access,OU=groups,dc=nilux,dc=be)'
  connection:
    host: 'slapd.nilux.be'
    tls: true
    port: 636
    bindDn: 'CN=srvmxid,OU=services,dc=nilux,dc=be'
    bindPassword: 'redact'
    baseDNs:
      - 'OU=users,dc=nilux,dc=be'
  attribute:
    name: 'cn'
    threepid:
      email:
        - 'mail'
      msisdn:
        - 'mobile'
    uid:
      type: 'uid'
      value: 'uid'

matrix:
  domain: 'matrix.nilux.be'

storage:
  provider:
    sqlite:
      database: '/var/ma1sd/db.sqlite'

threepid:
  medium:
    email:
      identity:
        from: 'redacted'
        name: 'Matrix Identity Service'
      connectors:
        smtp:
          host: 'mail.nilux.be'
          tls: 3
          port: 465

nginx config:

# HTTPS server
#
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name matrix.nilux.be;

    ssl_certificate         /etc/nginx/certs/live/matrix.nilux.be/fullchain.pem;
    ssl_certificate_key     /etc/nginx/certs/live/matrix.nilux.be/privkey.pem;
    ssl_trusted_certificate /etc/nginx/certs/live/matrix.nilux.be/fullchain.pem;

    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 5m;
    ssl_stapling on;
    ssl_stapling_verify on;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers   HIGH:!aNULL:!MD5:!RC4;

    ssl_dhparam /etc/nginx/dhparams.pem;
    ssl_prefer_server_ciphers on;

    set $upstream synapse.nilux.be:8008;
    add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
    location ~ /.well-known/matrix/ {
        default_type "text/plain";
        rewrite /.well-known/matrix/(.*) /$1 break;
        root /var/www/matrix;
    }
                      # Enable the user to authenticate with ma1sd iso the matrix HS
    # https://github.com/ma1uta/ma1sd/blob/master/docs/features/authentication.md
    location /_matrix/client/r0/login {
        proxy_pass http://ma1sd.nilux.be:8090;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
    }

    # also used for authentication
    location /_matrix/identity {
        proxy_pass http://ma1sd.nilux.be:8090/_matrix/identity;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
    }

    # user_directory allows to search users in the directory using their 3pid
    # https://github.com/ma1uta/ma1sd/blob/master/docs/features/directory.md
    location /_matrix/client/r0/user_directory {
        proxy_pass http://ma1sd.nilux.be:8090/_matrix/client/r0/user_directory;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
    }

    # Enable 3pid invites to work also with central matrix accounts
    # https://github.com/ma1uta/ma1sd/blob/master/docs/features/identity.md
    location ~* ^/_matrix/client/r0/rooms/([^/]+)/invite$ {
        proxy_pass  http://ma1sd.nilux.be:8090;
        proxy_set_header	Host $host;
        proxy_set_header	X-Forwarded-For $remote_addr;
    }

    location /_matrix {
        add_header Referrer-Policy "same-origin";
        fastcgi_hide_header X-Powered-By;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://$upstream ;        
            }

    location /.well-known/acme-challenge/ {
        default_type "text/plain";
        root /var/www/letsencrypt;
    }}

ma1sd log during the invitation:

[XNIO-1 task-9] INFO io.kamax.mxisd.http.undertow.handler.identity.v1.SingleLookupHandler - Got single lookup request from ipv6_prefix:a01:aaaa:aaaa:11:1 with client Synapse/1.9.1 - Is recursive? false
[XNIO-1 task-9] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Host ipv6_prefix:a01:aaaa:aaaa:11:1 allowed for recursion: false
[XNIO-1 task-9] INFO io.kamax.mxisd.backend.ldap.LdapThreePidProvider - Performing LDAP lookup [email protected] of type email
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 1.3.6.1.4.1.18060.0.0.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.7
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.2
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.18
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 1.2.840.113556.1.4.319
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.3
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.10.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 1.2.840.113556.1.4.473
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.osgi.DefaultLdapCodecService - Registered pre-bundled control factory: 1.2.840.113556.1.4.474
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.18060.0.0.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.7
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.2
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.18
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.319
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.3
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.10.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.42.2.27.8.5.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.9
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 2.16.840.1.113730.3.4.10
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.3
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.4
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.3.6.1.4.1.4203.1.9.1.2
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.473
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.474
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.841
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.417
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.1413
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled control factory: 1.2.840.113556.1.4.528
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.1.8
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.8
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.3
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.6
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.18060.0.1.5
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.4203.1.11.1
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.4203.1.11.3
[XNIO-1 task-9] INFO org.apache.directory.api.ldap.codec.standalone.CodecFactoryUtil - Registered pre-bundled extended operation factory: 1.3.6.1.4.1.1466.20037
[XNIO-1 task-9] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - No 3PID mapping found
[XNIO-1 task-9] INFO io.kamax.mxisd.http.undertow.handler.identity.v1.SingleLookupHandler - No mapping was found, return empty JSON object
[NioProcessor-1] WARN org.apache.directory.ldap.client.api.LdapNetworkConnection - Outbound done
javax.net.ssl.SSLException: Outbound done
	at org.apache.mina.filter.ssl.SslFilter.messageReceived(SslFilter.java:528)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:641)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:1114)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:121)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:641)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:634)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:539)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1241)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1230)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[XNIO-1 task-10] INFO io.kamax.mxisd.invitation.InvitationManager - Handling invite for email:[email protected] from @hntourne:matrix.nilux.be in room !LPmmmWQxxrSCAchxZu:matrix.nilux.be
[XNIO-1 task-10] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Host Internal allowed for recursion: true
[XNIO-1 task-10] INFO io.kamax.mxisd.backend.ldap.LdapThreePidProvider - Performing LDAP lookup [email protected] of type email
[XNIO-1 task-10] INFO io.kamax.mxisd.lookup.provider.DnsLookupProvider - Performing DNS lookup for [email protected]
[XNIO-1 task-10] INFO io.kamax.mxisd.lookup.provider.DnsLookupProvider - Domain name for [email protected]: gmail.com
[XNIO-1 task-10] INFO io.kamax.mxisd.matrix.IdentityServerUtils - gmail.com is not an URL, using as-is
[XNIO-1 task-10] INFO io.kamax.mxisd.matrix.IdentityServerUtils - Discovering Identity Server for gmail.com
[XNIO-1 task-10] INFO io.kamax.mxisd.matrix.IdentityServerUtils - Performing SRV lookup
[XNIO-1 task-10] INFO io.kamax.mxisd.matrix.IdentityServerUtils - Lookup name: _matrix-identity._tcp.gmail.com
[XNIO-1 task-10] INFO io.kamax.mxisd.matrix.IdentityServerUtils - No SRV record for _matrix-identity._tcp.gmail.com
[XNIO-1 task-10] INFO io.kamax.mxisd.lookup.provider.ForwarderProvider - Using forward server https://matrix.org
[XNIO-1 task-10] INFO io.kamax.mxisd.lookup.provider.RemoteIdentityServerFetcher - Looking up email 3PID [email protected] using https://matrix.org
[XNIO-1 task-10] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Found 3PID mapping: {medium: 'email', address: '[email protected]', mxid: '@test2343:matrix.org'}
[XNIO-1 task-10] INFO io.kamax.mxisd.invitation.InvitationManager - Mapping for email:[email protected] already exists, refusing to store invite
[XNIO-1 task-10] INFO io.kamax.mxisd.http.undertow.handler.BasicHttpHandler - Request POST http://matrix.nilux.be/_matrix/identity/api/v1/store-invite - Error M_ALREADY_EXISTS: A mapping already exists for this 3PID
[XNIO-1 task-11] INFO io.kamax.mxisd.invitation.InvitationManager - Handling invite for email:[email protected] from @hntourne:matrix.nilux.be in room !LPmmmWQxxrSCAchxZu:matrix.nilux.be
[XNIO-1 task-11] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Host Internal allowed for recursion: true
[XNIO-1 task-11] INFO io.kamax.mxisd.backend.ldap.LdapThreePidProvider - Performing LDAP lookup [email protected] of type email
[XNIO-1 task-11] INFO io.kamax.mxisd.lookup.provider.DnsLookupProvider - Performing DNS lookup for [email protected]
[XNIO-1 task-11] INFO io.kamax.mxisd.lookup.provider.DnsLookupProvider - Domain name for [email protected]: gmail.com
[XNIO-1 task-11] INFO io.kamax.mxisd.matrix.IdentityServerUtils - gmail.com is not an URL, using as-is
[XNIO-1 task-11] INFO io.kamax.mxisd.matrix.IdentityServerUtils - Discovering Identity Server for gmail.com
[XNIO-1 task-11] INFO io.kamax.mxisd.matrix.IdentityServerUtils - Performing SRV lookup
[XNIO-1 task-11] INFO io.kamax.mxisd.matrix.IdentityServerUtils - Lookup name: _matrix-identity._tcp.gmail.com
[XNIO-1 task-11] INFO io.kamax.mxisd.matrix.IdentityServerUtils - No SRV record for _matrix-identity._tcp.gmail.com
[XNIO-1 task-11] INFO io.kamax.mxisd.lookup.provider.ForwarderProvider - Using forward server https://matrix.org
[XNIO-1 task-11] INFO io.kamax.mxisd.lookup.provider.RemoteIdentityServerFetcher - Looking up email 3PID [email protected] using https://matrix.org
[XNIO-1 task-11] INFO io.kamax.mxisd.lookup.strategy.RecursivePriorityLookupStrategy - Found 3PID mapping: {medium: 'email', address: '[email protected]', mxid: '@test2343:matrix.org'}
[XNIO-1 task-11] INFO io.kamax.mxisd.invitation.InvitationManager - Mapping for email:[email protected] already exists, refusing to store invite
[XNIO-1 task-11] INFO io.kamax.mxisd.http.undertow.handler.BasicHttpHandler - Request POST http://matrix.nilux.be/_matrix/identity/api/v1/store-invite - Error M_ALREADY_EXISTS: A mapping already exists for this 3PID

synapse log during the invitation:

Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,386 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22147 - ::ffff:192.168.10.67 - 8008 - Received request: OPTIONS /_matrix/client/r0/createRoom
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,389 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22147 - ::ffff:192.168.10.67 - 8008 - {None} Processed request: 0.001sec/0.001sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/createRoom HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,403 - synapse.access.http.8008 - 233 - INFO - POST-22148 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: POST /_matrix/client/r0/createRoom
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,895 - synapse.access.http.8008 - 302 - INFO - GET-22146 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 11.337sec/0.001sec (0.030sec, 0.010sec) (0.027sec/0.084sec/12) 613B 200 "GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63927_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,926 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22149 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: OPTIONS /_matrix/client/r0/keys/query
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,929 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22150 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63929_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,932 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22149 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {None} Processed request: 0.001sec/0.004sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/keys/query HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,934 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22150 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {None} Processed request: 0.001sec/0.003sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63929_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,941 - synapse.access.http.8008 - 233 - INFO - GET-22151 - ::ffff:192.168.10.67 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63929_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:54 server 6f4d10a4094a[784]: 2020-02-04 17:49:54,970 - synapse.access.http.8008 - 233 - INFO - POST-22152 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: POST /_matrix/client/r0/keys/query
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,005 - synapse.access.http.8008 - 302 - INFO - POST-22152 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 0.030sec/0.003sec (0.007sec, 0.000sec) (0.003sec/0.016sec/2) 971B 200 "POST /_matrix/client/r0/keys/query HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,033 - synapse.access.http.8008 - 302 - INFO - GET-22151 - ::ffff:192.168.10.67 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 0.090sec/0.001sec (0.015sec, 0.003sec) (0.006sec/0.027sec/2) 544B 200 "GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63929_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,067 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22153 - ::ffff:192.168.10.67 - 8008 - Received request: OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63930_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,069 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22153 - ::ffff:192.168.10.67 - 8008 - {None} Processed request: 0.001sec/0.001sec (0.000sec, 0.002sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63930_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,076 - synapse.access.http.8008 - 233 - INFO - GET-22154 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63930_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,190 - synapse.access.http.8008 - 302 - INFO - GET-22154 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 0.112sec/0.002sec (0.029sec, 0.004sec) (0.008sec/0.028sec/3) 454B 200 "GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63930_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,208 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22155 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63931_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,211 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22155 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {None} Processed request: 0.001sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63931_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,225 - synapse.access.http.8008 - 233 - INFO - GET-22156 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63931_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,360 - synapse.access.http.8008 - 302 - INFO - GET-22156 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 0.128sec/0.006sec (0.019sec, 0.000sec) (0.010sec/0.029sec/3) 461B 200 "GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63931_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,372 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22157 - ::ffff:192.168.10.67 - 8008 - Received request: OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63932_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,374 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22157 - ::ffff:192.168.10.67 - 8008 - {None} Processed request: 0.001sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63932_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,380 - synapse.access.http.8008 - 233 - INFO - GET-22158 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63932_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,659 - synapse.access.http.8008 - 302 - INFO - GET-22158 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 0.276sec/0.003sec (0.019sec, 0.005sec) (0.011sec/0.023sec/3) 456B 200 "GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63932_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,677 - synapse.access.http.8008 - 233 - INFO - OPTIONS-22159 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63933_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,679 - synapse.access.http.8008 - 302 - INFO - OPTIONS-22159 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {None} Processed request: 0.001sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63933_2670892_506_88299_213_35_1_18345_1 HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]
Feb 04 18:49:55 server 6f4d10a4094a[784]: 2020-02-04 17:49:55,686 - synapse.access.http.8008 - 233 - INFO - GET-22160 - ::ffff:192.168.10.67 - 8008 - Received request: GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s63933_2670892_506_88299_213_35_1_18345_1
Feb 04 18:49:56 server 6f4d10a4094a[784]: 2020-02-04 17:49:56,322 - synapse.access.http.8008 - 233 - INFO - GET-22161 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: GET /_matrix/federation/v1/version
Feb 04 18:49:56 server 6f4d10a4094a[784]: 2020-02-04 17:49:56,325 - synapse.access.http.8008 - 302 - INFO - GET-22161 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {None} Processed request: 0.002sec/0.001sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 51B 200 "GET /_matrix/federation/v1/version HTTP/1.0" "-" [0 dbevts]
Feb 04 18:49:57 server 6f4d10a4094a[784]: 2020-02-04 17:49:57,168 - synapse.access.http.8008 - 233 - INFO - PUT-22162 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - Received request: PUT /_matrix/federation/v1/send/1580849784
Feb 04 18:49:57 server 6f4d10a4094a[784]: 2020-02-04 17:49:57,194 - synapse.access.http.8008 - 302 - INFO - PUT-22162 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {encom.eu.org} Processed request: 0.024sec/0.001sec (0.005sec, 0.000sec) (0.002sec/0.014sec/3) 12B 200 "PUT /_matrix/federation/v1/send/1580849784 HTTP/1.0" "-" [0 dbevts]
Feb 04 18:49:57 server 6f4d10a4094a[784]: 2020-02-04 17:49:57,329 - synapse.access.http.8008 - 302 - INFO - POST-22148 - ipv6_prefix:a00:aaaa:aaaa:3:1 - 8008 - {@hntourne:matrix.nilux.be} Processed request: 2.923sec/0.002sec (0.281sec, 0.018sec) (0.083sec/0.390sec/35) 67B 500 "POST /_matrix/client/r0/createRoom HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Safari/605.1.15 Epiphany/605.1.15" [0 dbevts]

Convert Username to lowercase

Hello,

we are currently using the matrix-synapse server with riot for our users in the Active Directory. During the past, we where using the mxisd for the authentication but since the development was halted, we started using the internal ldap authentication mechanism of matrix.
Our biggest problem was that when a user typed-in the username using uppercase or a combination of upper and lowercase, the user could not login.
I find out through another issue that you have this wonderful project that does exactly what the mxisd did.

I have set up the ma1sd, configured nginx etc, but we still have problems when the user types in the username in the form of f.i. First_name.Last_name. If the user logs in with first_name.last_name, the login works.

I read about the possibility to rewrite some auth credentials but I cannot make it to work.
Can you please help me out?

LDAP: ERR_04058 Cannot have a null initial, any and final substring

Hello,

I'm using ma1sd for auth/directory/identity. When I want to invite someone on a Matrix room, search works fine but I get many errors in log. I guess error happens when search is an empty string.

Thanks

[XNIO-1 task-8] INFO io.kamax.mxisd.directory.DirectoryManager - Original request URL: http://matrix.example.com/_matrix/client/r0/user_directory/search
[XNIO-1 task-8] INFO io.kamax.mxisd.directory.DirectoryManager - Querying HS at http://synapse:8008/_matrix/client/r0/user_directory/search
[XNIO-1 task-8] INFO io.kamax.mxisd.directory.DirectoryManager - Found 0 match(es) in HS for ''
[XNIO-1 task-8] INFO io.kamax.mxisd.directory.DirectoryManager - Using Directory provider LdapDirectoryProvider
[XNIO-1 task-8] INFO io.kamax.mxisd.backend.ldap.LdapDirectoryProvider - Performing LDAP directory search on display name using ''
[XNIO-1 task-8] WARN org.apache.directory.ldap.client.api.LdapNetworkConnection - org.apache.directory.api.ldap.codec.api.MessageEncoderException: ERR_04058 Cannot have a null initial, any and final substring
org.apache.mina.filter.codec.ProtocolEncoderException: org.apache.directory.api.ldap.codec.api.MessageEncoderException: ERR_04058 Cannot have a null initial, any and final substring
	at org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:360)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:744)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1500(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterWrite(DefaultIoFilterChain.java:1132)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.filterWrite(DefaultIoFilterChain.java:1020)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterWrite(DefaultIoFilterChain.java:744)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterWrite(DefaultIoFilterChain.java:737)
	at org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:570)
	at org.apache.mina.core.session.AbstractIoSession.write(AbstractIoSession.java:515)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.writeRequest(LdapNetworkConnection.java:4277)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.searchAsync(LdapNetworkConnection.java:1840)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.search(LdapNetworkConnection.java:1867)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.search(LdapNetworkConnection.java:1752)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.search(LdapNetworkConnection.java:1763)
	at io.kamax.mxisd.backend.ldap.LdapDirectoryProvider.search(LdapDirectoryProvider.java:71)
	at io.kamax.mxisd.backend.ldap.LdapDirectoryProvider.searchByDisplayName(LdapDirectoryProvider.java:104)
	at io.kamax.mxisd.directory.DirectoryManager.search(DirectoryManager.java:116)
	at io.kamax.mxisd.http.undertow.handler.directory.v1.UserDirectorySearchHandler.handleRequest(UserDirectorySearchHandler.java:47)
	at io.kamax.mxisd.http.undertow.handler.SaneHandler.handleRequest(SaneHandler.java:71)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:376)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.directory.api.ldap.codec.api.MessageEncoderException: ERR_04058 Cannot have a null initial, any and final substring
	at org.apache.directory.api.ldap.codec.api.LdapEncoder.encodeMessage(LdapEncoder.java:214)
	at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolEncoder.encode(LdapProtocolEncoder.java:82)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:329)
	... 23 more
Caused by: org.apache.directory.api.asn1.EncoderException: ERR_04058 Cannot have a null initial, any and final substring
	at org.apache.directory.api.ldap.codec.search.SubstringFilter.encode(SubstringFilter.java:314)
	at org.apache.directory.api.ldap.codec.search.ConnectorFilter.encode(ConnectorFilter.java:143)
	at org.apache.directory.api.ldap.codec.search.OrFilter.encode(OrFilter.java:125)
	at org.apache.directory.api.ldap.codec.decorators.SearchRequestDecorator.encode(SearchRequestDecorator.java:1026)
	at org.apache.directory.api.ldap.codec.api.LdapEncoder.encodeMessage(LdapEncoder.java:182)
	... 25 more
[XNIO-1 task-8] ERROR io.kamax.mxisd.http.undertow.handler.SaneHandler - Transaction #1593774686031 - org.apache.directory.api.ldap.codec.api.MessageEncoderException: ERR_04058 Cannot have a null initial, any and final substring
[XNIO-1 task-8] INFO io.kamax.mxisd.http.undertow.handler.BasicHttpHandler - Request POST http://matrix.example.com/_matrix/client/r0/user_directory/search - Error M_UNKNOWN: An internal server error occurred. Contact your administrator with reference Transaction #1593774686031

Document how to create new identity stores

Hello. Thanks for this identity server! Very helpful.

I would like to create a new identity store to authenticate against cryptocurrency wallets (e.g. Ethereum) using blockchain addresses as 3PIDs. I have reviewed all of the documentation I could find, but unfortunately, it does not seem like any of the existing stores can be used for this purpose as far as I can tell. (e.g. store documentation)

Could you provide documentation how to create and integrate a new identity store like this? Perhaps it would be best to make a general purpose "browser store" to allow for arbitrary in-browser verifications, which in this case could involve demonstrating the ability to sign something with the private key associated with their blockchain address. (Here is a library to do this from metamask, for example: https://github.com/metamask/eth-sig-util)

A side question: I am not a java programmer, so would it be possible to allow for new "plugin" stores to be added that might be written in an arbitrary language and perhaps exposed as another service on another port? Or some other method?

Thanks for any guidance or documentation.

LDAP Login failed

Well, I've installed Matrix, Nginx as a reverse proxy and everything is properly configugred. Also installed ma1sd, load LDAP parameters in ma1sd.yaml (disable in homeserver.yaml) but can't authenticate LDAP against AD.
When I check the syslog I see this

synapse.rest.client.v1.login - 195 - INFO - POST-59 - Got login request with identifier: {'type': 'm.id.user', 'user': 'user'}, medium: None, address: None, user: 'user'
2020-09-08 09:27:32,778 - synapse.handlers.auth - 842 - WARNING - POST-59 - Failed password login for user @user:matrix.domain.com

Followed the guide for the installation (Debian 10) but nothing seems to work
matrix-react-sdk version: 0.10.3
riot-web version: 0.12.3

Can't set new password

I'm pretty sure users were able to change their LDAP password via the Element (at the time Riot) client. This was maybe in April or May when I last set up a Synapse + ma1sd with LDAP store.

Now that option is no longer available. In the Element settings I can see the heading "Account", where the option for setting a new password should be, but after that the heading "Email addresses" immediately follows:
nopassword

Not sure if this is intended but it would be nice for users to be able to change their LDAP password from within Element.

The following is my ma1sd.yaml:

matrix:
  domain: 'XXXX'
  v1: true
  v2: false

key:
  path: '/var/ma1sd/sign.key'

storage:
  backend: postgresql
  provider:
    postgresql:
      database: //XXXX:5432/ma1sd
      username: XXXX
      password: XXXX

ldap:
    attribute:
        uid:
            type: 'uid'
            value: 'uid'
        avatar: 'jpegPhoto'
    connection:
        baseDNs:
        - OU=XXXX
        bindDn: XXXX
        bindPassword: XXXX
        host: XXXX
        port: 389
        tls: false
    enabled: true

M_UNAUTHORIZED

When a user clicks on their User Profile in Riot, Ma1sd says:
It appears the client has requested:
2.244.174.94 - - [13/Jun/2020:14:25:18 +0200] "GET /_matrix/identity/v2/hash_details HTTP/1.1" 401 502 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Riot/1.6.4 Chrome/80.0.3987.134 Electron/8.0.3 Safari/537.36"

ma1sd says:
[XNIO-1 task-6] WARN io.kamax.mxisd.auth.AccountManager - Account not found.
[XNIO-1 task-6] ERROR io.kamax.mxisd.http.undertow.handler.AuthorizationHandler - Account not found from request from: matrix.cambsac.org.uk
[XNIO-1 task-6] INFO io.kamax.mxisd.http.undertow.handler.BasicHttpHandler - Request GET http://matrix.cambsac.org.uk/_matrix/identity/v2/hash_details - Error M_UNAUTHORIZED: Supplied credentials are invalid

But no clue as to why.

Is this an issue?
I note that the user profile does not show an email address.
This user (myself) has always been in Synapse.

Is this related to my attempt to authenticate some users through django?

Thanks,
Jason

Integration with Django

I want to try an integrate ma1sd as a source of IDs from Django.
my auth_user table in django has
username, first_name, last_name, full_name, email in a view auth_fullnames
Database is currently sqlite, though it will soon become postgres.
Would this be correct for sqlite:

sql:
  enabled: true
  type: sqlite
  connection: /usr/lib/database/django
  directory:
    query:
      name:
        type: 'localpart'
        value: 'SELECT username, full_name FROM auth_fullnames WHERE full_name LIKE ?'
      threepid:
        type: 'localpart'
        value: 'SELECT username, full_name FROM auth_fullnames WHERE email LIKE ?'

Also:
1: If and how to use the Identity and Profile stanzas?
2: Possible to synchronise passwords using ma1sd between Django and Synapse?

For 2 I guess we use either the exec or REST backends, but the latter has limited examples.

Deactivating a user: Failed to remove threepid from ID server

Hi, I tried to deactivate a user with

curl -X POST 'https://matrix.xxx.xxx/_matrix/client/r0/admin/deactivate/%40someuser%3Amatrix.xxx.xxx?access_token=SOMETOKEN' --data '{}'

got

{
    "errcode": "M_UNKNOWN",
    "error": "Failed to remove threepid from ID server"
}

Is it me or is it ma1sd?

Email binding is not being stored

Reproduction steps:

  • In Riot/Web, I had already used Settings->Email Addresses to add two email addresses to my Matrix account (this part is handled by Synapse). I know this worked properly because I am able to log into my account by providing my email address instead of my mxid.
  • Under Settings->Discovery->Email addresses, I click "Share" next to an email address. POST _matrix/identity/v2/validate/email/requestToken gets a numeric sid as a response.
  • ma1sd correctly sends an email to the account I wanted to share. Clicking the token contained in the email says "verification successful".
  • Back in Riot/Web, clicking "Complete" has it turn into a "Revoke" button, suggesting that the binding worked. GET _matrix/identity/v2/account responds with my user_id.
  • Trying to do the same thing with a different email address, but without clicking on the token link sent by ma1sd, fails. This means that ma1sd is handling verification correctly.

The problem: Closing & reopening the Settings window still has the "Share" button next to the email address that should have been shared already. POST _matrix/identity/v2/lookup responds with an empty mappings object.

Expected results: The lookup of my just-shared email should succeed.

Other info:

  • The accepted and hashes tables in ma1sd's database are empty, but account is not.

  • My (partially redacted) config:

matrix:
  domain: 'domain'
  v1: true
  v2: true
  ## Remove default matrix-org server
  identity:
    servers:
      myOtherServers: []

server:
  name: 'identity.domain'

key:
  path: '/var/lib/ma1sd/keys'

storage:
  backend: postgresql
  provider:
    postgresql:
      database: '//localhost/ma1sd'
      username: 'user'
      password: 'pass'

synapseSql:
  enabled: true
  type: 'postgresql'
  connection: '//localhost/synapse?user=synapse_user&password=pass'

threepid:
  medium:
    email:
      identity:
        from: "noreply-id@domain"
        name: "Name"
      connectors:
        smtp:
          host: "localhost"
          tls: 0
          port: 25

hashing:
  enabled: true
  pepperLength: 20
  rotationPolicy: per_requests
  hashStorageType: sql
  algorithms:
    - sha256
  delay: 2m
  requests: 10

synapseSql:
  lookup:
    query: 'select user_id as mxid, medium, address from user_threepids'
  legacyRoomNames: false

logging:
  root: debug
  app: debug

Note that using the none algorithm (before or after sha256) doesn't fix the problem, and neither does changing hashStorageType to in_memory.

  • Log contents (partially redacted) for when a binding is attempted:
INFO io.kamax.mxisd.auth.AccountManager - Found account for user: @mxid:domain
INFO io.kamax.mxisd.http.undertow.handler.auth.v2.AccountGetUserInfoHandler - Get User Info.
INFO io.kamax.mxisd.http.undertow.handler.auth.v2.AccountGetUserInfoHandler - Account found: @mxid:domain
INFO io.kamax.mxisd.auth.AccountManager - Found account for user: @mxid:domain
INFO io.kamax.mxisd.session.SessionManager - Server 127.0.0.1 is asking to create session for io.kamax.matrix.ThreePid@deadbeef (Attempt #1) - Next link: null
INFO io.kamax.mxisd.session.SessionManager - No existing session for io.kamax.matrix.ThreePid@deadbeef
INFO io.kamax.mxisd.session.SessionManager - Generated new session 1234567891234 to validate io.kamax.matrix.ThreePid@deadbeef from server 127.0.0.1
INFO io.kamax.mxisd.session.SessionManager - Stored session 1234567891234
INFO io.kamax.mxisd.session.SessionManager - Session 1234567891234 for io.kamax.matrix.ThreePid@deadbeef: sending validation notification
INFO io.kamax.mxisd.threepid.generator.GenericTemplateNotificationGenerator - Generating notification content for 3PID Session validation
INFO io.kamax.mxisd.threepid.connector.email.EmailSmtpConnector - Sending email to email@domain via SMTP using localhost:25
INFO io.kamax.mxisd.threepid.connector.email.EmailSmtpConnector - Connecting to localhost:25
INFO io.kamax.mxisd.threepid.connector.email.EmailSmtpConnector - Not using SMTP authentication
INFO io.kamax.mxisd.threepid.connector.email.EmailSmtpConnector - Email to email@domain was sent
INFO io.kamax.mxisd.http.undertow.handler.identity.share.SessionValidationGetHandler - Handling GET request to validate session
INFO io.kamax.mxisd.session.SessionManager - Validating session 1234567891234
INFO io.kamax.mxisd.session.SessionManager - Session 1234567891234 is from 127.0.0.1
INFO io.kamax.mxisd.session.SessionManager - Session 1234567891234 has been validated
INFO io.kamax.mxisd.http.undertow.handler.identity.share.SessionValidationGetHandler - Session 1234567891234 was validated
INFO io.kamax.mxisd.auth.AccountManager - Found account for user: @mxid:domain
INFO io.kamax.mxisd.http.undertow.handler.auth.v2.AccountGetUserInfoHandler - Get User Info.
INFO io.kamax.mxisd.http.undertow.handler.auth.v2.AccountGetUserInfoHandler - Account found: @mxid:domain
INFO io.kamax.mxisd.auth.AccountManager - Found account for user: @mxid:domain
INFO io.kamax.mxisd.session.SessionManager - Session 1234567891234: Binding of email:email@domain to Matrix ID @gxid:domain is accepted

ma1sd fails if it gets a connection refused for well-known URL

I'm running a self hosted homeserver. My homeserver name is my top level domain (lapiole.org), and I'm using SRV DNS records to publish the name of the homeserver (matrix.lapiole.org). The top level domain does not run a web server (so I cannot publish well-known files, nor should I need as I have SRV)
ma1sd fails in such a setup to register my 3pids (emails fetched from LDAP). From Riot web, I open preferences, and the POST to /_matrix/identity/v2/account/register gets a 500 error.
In ma1sd log I get :

Jun 04 23:45:14 mtx matrix-ma1sd[13347]: java.lang.RuntimeException: Error while trying to lookup well-known for lapiole.org
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.matrix.HomeserverFederationResolver.resolveWellKnown(HomeserverFederationResolver.java:133)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.matrix.HomeserverFederationResolver.resolve(HomeserverFederationResolver.java:196)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.auth.AccountManager.getUserId(AccountManager.java:69)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.auth.AccountManager.register(AccountManager.java:53)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.http.undertow.handler.auth.v2.AccountRegisterHandler.handleRequest(AccountRegisterHandler.java:54)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.http.undertow.handler.SaneHandler.handleRequest(SaneHandler.java:71)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:376)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.lang.Thread.run(Thread.java:748)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: Caused by: org.apache.http.conn.HttpHostConnectException: Connect to lapiole.org:443 [lapiole.org/10.99.3.12, lapiole.org/10.99.3.13] failed: Connection refused (Connection refused)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at io.kamax.mxisd.matrix.HomeserverFederationResolver.resolveWellKnown(HomeserverFederationResolver.java:114)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: ... 10 more
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: Caused by: java.net.ConnectException: Connection refused (Connection refused)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.net.PlainSocketImpl.socketConnect(Native Method)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at java.net.Socket.connect(Socket.java:607)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: ... 20 more
Jun 04 23:45:14 mtx matrix-ma1sd[13347]: [XNIO-1 task-1] INFO io.kamax.mxisd.http.undertow.handler.BasicHttpHandler - Request POST http://matrix.lapiole.org/_matrix/identity/v2/account/register - Error M_UNKNOWN: Error while trying to lookup well-known for lapiole.org

The problem is here : https://github.com/ma1uta/ma1sd/blob/master/src/main/java/io/kamax/mxisd/matrix/HomeserverFederationResolver.java#L133

Shouldn't failing to connect be treated like non 200, or invalid JSON : just log and go on ?

java.lang.NullPointerException at io.kamax.mxisd.hash.HashManager.initStorage()

after setting

hashing:
    enabled: true

ma1sd (latest-dev from "3 days ago", 71efcb85f0c8) crashes.. so I deactivated that again ๐Ÿ˜…

[main] INFO io.kamax.mxisd.config.HashingConfig - --- Hash configuration ---
[main] INFO io.kamax.mxisd.config.HashingConfig -    Pepper length: 20
[main] INFO io.kamax.mxisd.config.HashingConfig -    Rotation policy: null
[main] INFO io.kamax.mxisd.config.HashingConfig -    Hash storage type: null
[main] INFO io.kamax.mxisd.config.HashingConfig -    Algorithms: []
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ChangelogDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ThreePidInviteIO
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: HistoricalThreePidInviteIO
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ThreePidSessionDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: ASTransactionDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: AccountDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: AcceptedDao
[main] INFO io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage - Create the dao: HashDao
[main] INFO io.kamax.mxisd.storage.crypto.FileKeyStore - Key store is already in directory format
[main] INFO com.mchange.v2.c3p0.C3P0Registry - Initializing c3p0-0.9.5.4 [built 23-March-2019 23:00:48 -0700; debug? true; trace: 10]
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig - --- E-mail Generator templates config ---
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig - Invite: Built-in (/threepids/email/invite-template.eml)
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig - Session:
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig -   Validation: classpath:/threepids/email/validate-template.eml
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig -   Unbind:
[main] INFO io.kamax.mxisd.config.threepid.medium.EmailTemplateConfig -     Notification: classpath:/threepids/email/unbind-notification.eml
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - --- E-mail SMTP Connector config ---
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Host: smtp.example.foo
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Port: 587
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - TLS Mode: 1
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Login: smtpuser
[main] INFO io.kamax.mxisd.config.threepid.connector.EmailSmtpConfig - Has password: true
java.lang.NullPointerException
        at io.kamax.mxisd.hash.HashManager.initStorage(HashManager.java:45)
        at io.kamax.mxisd.hash.HashManager.init(HashManager.java:37)
        at io.kamax.mxisd.Mxisd.build(Mxisd.java:127)
        at io.kamax.mxisd.Mxisd.start(Mxisd.java:218)
        at io.kamax.mxisd.HttpMxisd.start(HttpMxisd.java:102)
        at io.kamax.mxisd.MxisdStandaloneExec.main(MxisdStandaloneExec.java:109)
[Thread-1] INFO App - ma1sd stopped
 matrix-ma1sd.service: Main process exited, code=exited, status=1/FAILURE
 matrix-ma1sd.service: Failed with result 'exit-code'.
 matrix-ma1sd.service: Service RestartSec=30s expired, scheduling restart.
 matrix-ma1sd.service: Scheduled restart job, restart counter is at 14.
 Stopped Matrix ma1sd Identity server.

Multiple user invitations with email uses email "to: instead of "bcc"

Inviting people in a room with email ( sending multiple invites with one click ) during email invitation all emails that has been invited to join room show into to: which creates some "problem" because somebody can copy and paste this emails later somewhere else for spam or whatever reason.

Solution: When there is only one email invite use to: when there are more than two then put email recipients into BCC or known as blind carbon copy.

Ma1sd can't started

The standard config contains an error that the parser refers to - the key names do not contain tabs in front of them.
After fixing this error throughout the entire config, a new error appeared that could not be fixed by creating this path:
[main] ERROR App - null [main] ERROR App - Invalid or empty value for configuration item: key.path

3PID bind fails with synapse 1.1.0

I don't know when it last worked, but it worked at some point with that configuration.

Currently I am running
mxisd 1.4.3
synapse 1.1.0
riot-web 1.2.1

When trying to attach an email to a user, the following requests are done:

POST /_matrix/client/r0/account/3pid HTTP/1.0

{"threePidCreds":{"sid":"XXXXXXXX08088","client_secret":"XXXXXXXXXXXXXXXXXXXXXXXXXXBbXBx2","id_server":"xxx"},"bind":true}

  GET /_matrix/identity/api/v1/3pid/getValidated3pid?sid=XXXXXXXX08088&client_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXBbXBx2 HTTP/1.0
  Returns: 200 OK
  Content: {"medium":"email","address":"xxx@xxx","validated_at":1562509218632}

  POST /_matrix/identity/api/v1/3pid/bind HTTP/1.0
  sid=XXXXXXXX08088&client_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXbXBx2&mxid=@xxx:xxx
  Returns 200 OK
  Result: {} (Empty JSON)


Returns: 500
Result: {"errcode":"M_UNKNOWN","error":"Internal server error"}

The GET and POST within the POST come from Synapse.

Relevant logs:

Jul 07 22:12:14 synapse[8337]: synapse.http.client: [POST-3805] Sending request POST https://xxx/_matrix/identity/api/v1/3pid/bind
Jul 07 22:12:14 mxisd[2538]: [XNIO-1 task-5] INFO io.kamax.mxisd.session.SessionManager - Session 15625
29899056: Binding of email:xxx@xxx to Matrix ID @xxx:xxx is accepted
Jul 07 22:12:14 synapse[8337]: synapse.http.client: [POST-3805] Received response to POST https://xxx/_matrix/identity/api/v1/3pid/bind: 200
Jul 07 22:12:14 synapse[8337]: synapse.http.server: [POST-3805] Failed handle request via 'ThreepidRest
Servlet': <XForwardedForRequest at 0x7fae853f5860 method='POST' uri='/_matrix/client/r0/account/3pid' clientproto='HTTP/1.0' site=800
8>
                                                            Traceback (most recent call last):
                                                               File "/nix/store/j5pz84glsnnsx7gks2ny91w4an40i4wg-python3.7-Twisted-18
.9.0/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
                                                                 result = g.send(result)
                                                               File "/nix/store/48dl4n1qh7wbpsi18kq1yjdj9rzh66mq-matrix-synapse-1.1.0
/lib/python3.7/site-packages/synapse/http/client.py", line 348, in post_urlencoded_get_json
                                                                 defer.returnValue(json.loads(body))
                                                               File "/nix/store/j5pz84glsnnsx7gks2ny91w4an40i4wg-python3.7-Twisted-18
.9.0/lib/python3.7/site-packages/twisted/internet/defer.py", line 1362, in returnValue
                                                                 raise _DefGen_Return(val)
                                                             twisted.internet.defer._DefGen_Return: {}

                                                             During handling of the above exception, another exception occurred:

                                                             Traceback (most recent call last):
                                                               File "/nix/store/48dl4n1qh7wbpsi18kq1yjdj9rzh66mq-matrix-synapse-1.1.0
/lib/python3.7/site-packages/synapse/http/server.py", line 76, in wrapped_request_handler
                                                                 await h(self, request)
                                                               File "/nix/store/48dl4n1qh7wbpsi18kq1yjdj9rzh66mq-matrix-synapse-1.1.0
/lib/python3.7/site-packages/synapse/http/server.py", line 301, in _async_render
                                                                 callback_return = await callback_return
                                                               File "/nix/store/j5pz84glsnnsx7gks2ny91w4an40i4wg-python3.7-Twisted-18
.9.0/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
                                                                 result = result.throwExceptionIntoGenerator(g)
                                                               File "/nix/store/j5pz84glsnnsx7gks2ny91w4an40i4wg-python3.7-Twisted-18
.9.0/lib/python3.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
                                                                 return g.throw(self.type, self.value, self.tb)
                                                               File "/nix/store/48dl4n1qh7wbpsi18kq1yjdj9rzh66mq-matrix-synapse-1.1.0
/lib/python3.7/site-packages/synapse/rest/client/v2_alpha/account.py", line 569, in on_POST
                                                                 yield self.identity_handler.bind_threepid(threePidCreds, user_id)
                                                               File "/nix/store/j5pz84glsnnsx7gks2ny91w4an40i4wg-python3.7-Twisted-18
.9.0/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
                                                                 result = g.send(result)
                                                               File "/nix/store/48dl4n1qh7wbpsi18kq1yjdj9rzh66mq-matrix-synapse-1.1.0
/lib/python3.7/site-packages/synapse/handlers/identity.py", line 130, in bind_threepid
                                                                 medium=data["medium"],
                                                             KeyError: 'medium'

Identity server has no terms of service

I just installed ma1sd on my homeserver. When I try adding identity server on Riot web app. A warning appeared "Identity server has no terms of service." It can add after clicking continue. What is the reason for this warning?

mxid generation rules

In some scenarios, one might not want to use any of the attributes in a LDAP dataset 1:1 but might wish for some operation to be applied to generate the mxisd. F.e. taking the email field, cutting off the '@.*' and appending -[department field]-[first 5 digits from phone field] .. would this be sensible to implement? What about collisions?

Fatal error in OpenJDK 11

Using OpenJDK 11, the following JRE error occurs & gets dumped to hs_err_pid*.log soon after starting ma1sd.

  • OS: Debian 10
  • ma1sd version: 2.2.x (from deb package)
  • Java package: openjdk-11-jre-headless:armhf

A similar crash happens with the JDK 8 from adoptopenjdk, so I doubt this is a Java versioning issue. I had to install JDK 8 to satisfy the dependencies of ma1sd's deb package, but later used update-alternatives to set my system's default JVM to the one from OpenJDK.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x911b56ac, pid=28628, tid=28633
#
# JRE version: OpenJDK Runtime Environment (11.0.5+10) (build 11.0.5+10-post-Debian-1deb10u1)
# Java VM: OpenJDK Server VM (11.0.5+10-post-Debian-1deb10u1, mixed mode, g1 gc, linux-)
# Problematic frame:
# C  [sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so+0x96ac]
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   https://bugs.debian.org/openjdk-11
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: /usr/lib/ma1sd/ma1sd.jar -c /etc/ma1sd/ma1sd.yaml

Host: rev 10 (v7l), 4 cores, 1G, Debian GNU/Linux 10 (buster)
Time: Thu Jan  9 21:48:53 2020 EST elapsed time: 9 seconds (0d 0h 0m 9s)

---------------  T H R E A D  ---------------

Current thread (0xb630d800):  JavaThread "main" [_thread_in_native, id=28633, stack(0xb6437000,0xb6487000)]

Stack: [0xb6437000,0xb6487000],  sp=0xb64843c0,  free space=308k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so+0x96ac]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.lang.ClassLoader$NativeLibrary.load0(Ljava/lang/String;Z)Z+0 [email protected]
j  java.lang.ClassLoader$NativeLibrary.load()Z+53 [email protected]
j  java.lang.ClassLoader$NativeLibrary.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)Z+216 [email protected]
j  java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+46 [email protected]
j  java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+48 [email protected]
j  java.lang.Runtime.load0(Ljava/lang/Class;Ljava/lang/String;)V+57 [email protected]
j  java.lang.System.load(Ljava/lang/String;)V+7 [email protected]
j  org.sqlite.SQLiteJDBCLoader.loadNativeLibrary(Ljava/lang/String;Ljava/lang/String;)Z+29
j  org.sqlite.SQLiteJDBCLoader.extractAndLoadLibraryFile(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Z+372
j  org.sqlite.SQLiteJDBCLoader.loadSQLiteNativeLibrary()V+238
j  org.sqlite.SQLiteJDBCLoader.initialize()Z+9
j  org.sqlite.core.NativeDB.load()Z+19
j  org.sqlite.SQLiteConnection.open(Ljava/lang/String;Ljava/lang/String;Ljava/util/Properties;)Lorg/sqlite/core/DB;+387
j  org.sqlite.SQLiteConnection.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/util/Properties;)V+13
j  org.sqlite.jdbc3.JDBC3Connection.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/util/Properties;)V+4
j  org.sqlite.jdbc4.JDBC4Connection.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/util/Properties;)V+4
j  org.sqlite.JDBC.createConnection(Ljava/lang/String;Ljava/util/Properties;)Lorg/sqlite/SQLiteConnection;+24
j  org.sqlite.JDBC.connect(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+2
j  java.sql.DriverManager.getConnection(Ljava/lang/String;Ljava/util/Properties;Ljava/lang/Class;)Ljava/sql/Connection;+130 [email protected]
j  java.sql.DriverManager.getConnection(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+5 [email protected]
j  com.j256.ormlite.jdbc.JdbcConnectionSource.makeConnection(Lcom/j256/ormlite/logger/Logger;)Lcom/j256/ormlite/support/DatabaseConnection;+53
j  com.j256.ormlite.jdbc.JdbcConnectionSource.getReadWriteConnection(Ljava/lang/String;)Lcom/j256/ormlite/support/DatabaseConnection;+79
j  com.j256.ormlite.table.TableUtils.doCreateTable(Lcom/j256/ormlite/support/ConnectionSource;Lcom/j256/ormlite/table/TableInfo;Z)I+52
j  com.j256.ormlite.table.TableUtils.doCreateTable(Lcom/j256/ormlite/dao/Dao;Z)I+21
j  com.j256.ormlite.table.TableUtils.createTableIfNotExists(Lcom/j256/ormlite/support/ConnectionSource;Ljava/lang/Class;)I+8
j  io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.createDaoAndTable(Lcom/j256/ormlite/support/ConnectionSource;Ljava/lang/Class;)Lcom/j256/ormlite/dao/Dao;+22
j  io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.lambda$new$0(Ljava/lang/String;Ljava/lang/String;)V+41
j  io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage$$Lambda$35.run()V+12
j  io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.withCatcher(Lio/kamax/mxisd/storage/ormlite/OrmLiteSqlStorage$Doer;)V+1
j  io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.<init>(Ljava/lang/String;Ljava/lang/String;)V+47
j  io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.<init>(Lio/kamax/mxisd/config/MxisdConfig;)V+21
j  io.kamax.mxisd.Mxisd.build()V+88
j  io.kamax.mxisd.Mxisd.start()V+1
j  io.kamax.mxisd.HttpMxisd.start()V+4
j  io.kamax.mxisd.MxisdStandaloneExec.main([Ljava/lang/String;)V+396
v  ~StubRoutines::call_stub

siginfo: si_signo: 4 (SIGILL), si_code: 1 (ILL_ILLOPC), si_addr: 0x911b56ac

Register to memory mapping:

  r0  = 0x9127d698
0x9127d698: __bss_start__+0x00000000 in /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so at 0x911ac000

  r1  = 0x00000000
0x0 is NULL

  r2  = 0x00000002
0x00000002 is an unknown value

  r3  = 0x9127c000
0x9127c000: <offset 0x000d0000> in /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so at 0x911ac000

  r4  = 0x9127a828
0x9127a828: <offset 0x000ce828> in /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so at 0x911ac000

  r5  = 0x911b5758
0x911b5758: <offset 0x00009758> in /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so at 0x911ac000

  r6  = 0x00000005
0x00000005 is an unknown value

  r7  = 0xbe91bde4
0xbe91bde4 points into unknown readable memory: da be 91 be

  r8  = 0xbe91bdfc
0xbe91bdfc points into unknown readable memory: 16 bf 91 be

  r9  = 0x9127a828
0x9127a828: <offset 0x000ce828> in /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so at 0x911ac000

  r10 = 0x00000000
0x0 is NULL

  fp  = 0xb6fa1970
0xb6fa1970: _rtld_global_ro+0x00000000 in /lib/ld-linux-armhf.so.3 at 0xb6f79000

  r12 = 0x916a9ea8
0x916a9ea8 points into unknown readable memory: 2f 74 6d 70

  sp  = 0xb64843c0
0xb64843c0 is pointing into the stack for thread: 0xb630d800

  lr  = 0xb6f84581
0xb6f84581: <offset 0x0000b581> in /lib/ld-linux-armhf.so.3 at 0xb6f79000

  pc  = 0x911b56ac
0x911b56ac: <offset 0x000096ac> in /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so at 0x911ac000



Registers:
  r0  = 0x9127d698
  r1  = 0x00000000
  r2  = 0x00000002
  r3  = 0x9127c000
  r4  = 0x9127a828
  r5  = 0x911b5758
  r6  = 0x00000005
  r7  = 0xbe91bde4
  r8  = 0xbe91bdfc
  r9  = 0x9127a828
  r10 = 0x00000000
  fp  = 0xb6fa1970
  r12 = 0x916a9ea8
  sp  = 0xb64843c0
  lr  = 0xb6f84581
  pc  = 0x911b56ac
  cpsr = 0x600c0010

Top of Stack: (sp=0xb64843c0)
0xb64843c0:   be91bdfc 00000000 60000000 916ac1b0
0xb64843d0:   00000005 be91bde4 be91bdfc b6f84633
0xb64843e0:   ffffffff 916ac1b0 00000003 b6484410
0xb64843f0:   00000003 00000008 b6484630 b6f86ff9
0xb6484400:   b6485344 00000003 916ac1b0 00000000
0xb6484410:   00000000 00000001 00000001 00000000
0xb6484420:   b6484630 b6484408 00000000 80000001
0xb6484430:   0002f8c0 00000000 b64846b8 80000001 

Instructions: (pc=0x911b56ac)
0x911b55ac:   e1a01005 e5940000 eb020c3b e2504000
0x911b55bc:   1a00000a e59d000c eb01caf1 e3500064
0x911b55cc:   1a000003 e59d000c e1a01004 eb00574c
0x911b55dc:   e1c600f0 e59d000c eb016799 e1a04000
0x911b55ec:   e1a00005 eb0037bd e1a00004 e28dd010
0x911b55fc:   e8bd8070 000b1d14 e59f3014 e59f2014
0x911b560c:   e08f3003 e7932002 e3520000 012fff1e
0x911b561c:   eaffef74 000c69ec 0000013c e59f3048
0x911b562c:   e59f0048 e08f3003 e08f0000 e2833003
0x911b563c:   e59f203c e0603003 e3530006 e08f2002
0x911b564c:   912fff1e e59f302c e24dd008 e7923003
0x911b565c:   e58d3004 e3530000 0a000001 e28dd008
0x911b566c:   e12fff13 e28dd008 e12fff1e 000c8060
0x911b567c:   000c805c 000c69b0 00000138 e59f0050
0x911b568c:   e3a02002 e59f104c e08f0000 e08f1001
0x911b569c:   e59f3044 e0601001 e1a01141 e08f3003
0x911b56ac:   e711f211 e3510000 012fff1e e59f202c
0x911b56bc:   e24dd008 e7933002 e58d3004 e3530000
0x911b56cc:   0a000001 e28dd008 e12fff13 e28dd008
0x911b56dc:   e12fff1e 000c7ffc 000c7ff8 000c6950
0x911b56ec:   00000144 e59f204c e92d4008 e08f2002
0x911b56fc:   e59f3044 e5d22000 e08f3003 e3520000
0x911b570c:   18bd8008 e59f2034 e7933002 e3530000
0x911b571c:   0a000002 e59f3028 e79f0003 ebffeecb
0x911b572c:   ebffffbd e59f301c e3a02001 e08f3003
0x911b573c:   e5c32000 e8bd8008 000c7f98 000c68f4
0x911b574c:   00000134 000c6a1c 000c7f58 e92d4007
0x911b575c:   e59f0038 e59f3038 e08f0000 e5902000
0x911b576c:   e08f3003 e3520000 0a000005 e59f2024
0x911b577c:   e7933002 e58d3004 e3530000 0a000000
0x911b578c:   e12fff33 e28dd00c e49de004 eaffffba
0x911b579c:   000c50c0 000c688c 00000140 e92d41f0 


Stack slot to memory mapping:
stack at sp + 0 slots: 0xbe91bdfc points into unknown readable memory: 16 bf 91 be
stack at sp + 1 slots: 0x0 is NULL
stack at sp + 2 slots: 0x60000000 is an unknown value
stack at sp + 3 slots: 0x916ac1b0 points into unknown readable memory: 00 c0 1a 91
stack at sp + 4 slots: 0x00000005 is an unknown value
stack at sp + 5 slots: 0xbe91bde4 points into unknown readable memory: da be 91 be
stack at sp + 6 slots: 0xbe91bdfc points into unknown readable memory: 16 bf 91 be
stack at sp + 7 slots: 0xb6f84633: <offset 0x0000b633> in /lib/ld-linux-armhf.so.3 at 0xb6f79000


---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x916a9698, length=9, elements={
0xb630d800, 0x92722800, 0x92726000, 0x9272ec00,
0x92730400, 0x92732c00, 0x92791400, 0x92796800,
0x916a8800
}

Java Threads: ( => current thread )
=>0xb630d800 JavaThread "main" [_thread_in_native, id=28633, stack(0xb6437000,0xb6487000)]
  0x92722800 JavaThread "Reference Handler" daemon [_thread_blocked, id=28640, stack(0x92e2e000,0x92e7e000)]
  0x92726000 JavaThread "Finalizer" daemon [_thread_blocked, id=28641, stack(0x926b0000,0x92700000)]
  0x9272ec00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=28642, stack(0x92660000,0x926b0000)]
  0x92730400 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=28643, stack(0x92006000,0x92086000)]
  0x92732c00 JavaThread "Sweeper thread" daemon [_thread_blocked, id=28644, stack(0x92610000,0x92660000)]
  0x92791400 JavaThread "Service Thread" daemon [_thread_blocked, id=28645, stack(0x91fb6000,0x92006000)]
  0x92796800 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=28647, stack(0x91ee4000,0x91f34000)]
  0x916a8800 JavaThread "process reaper" daemon [_thread_blocked, id=28655, stack(0x93103000,0x93123000)]

Other Threads:
  0x9271c400 VMThread "VM Thread" [stack: 0x92834000,0x928b4000] [id=28639]
  0x92792c00 WatcherThread [stack: 0x91f36000,0x91fb6000] [id=28646]
  0xb6322000 GCTaskThread "GC Thread#0" [stack: 0xb4062000,0xb40e2000] [id=28634]
  0x92500c00 GCTaskThread "GC Thread#1" [stack: 0x91e64000,0x91ee4000] [id=28649]
  0xb636e000 ConcurrentGCThread "G1 Main Marker" [stack: 0x933c8000,0x93448000] [id=28635]
  0xb636f400 ConcurrentGCThread "G1 Conc#0" [stack: 0x93346000,0x933c6000] [id=28636]
  0xb63c7800 ConcurrentGCThread "G1 Refine#0" [stack: 0x93180000,0x93200000] [id=28637]
  0x92fbec00 ConcurrentGCThread "G1 Refine#1" [stack: 0x91580000,0x91600000] [id=28653]
  0xb63c8c00 ConcurrentGCThread "G1 Young RemSet Sampling" [stack: 0x92e80000,0x92f00000] [id=28638]

Threads with active compile tasks:

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:
 garbage-first heap   total 32768K, used 9981K [0x94600000, 0xb3e00000)
  region size 1024K, 4 young (4096K), 1 survivors (1024K)
 Metaspace       used 10925K, capacity 11034K, committed 11160K, reserved 11568K
Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TAMS=top-at-mark-start (previous, next)
|   0|0x94600000, 0x94700000, 0x94700000|100%| O|  |TAMS 0x94600000, 0x94600000| Untracked 
|   1|0x94700000, 0x94800000, 0x94800000|100%| O|  |TAMS 0x94700000, 0x94700000| Untracked 
|   2|0x94800000, 0x94808400, 0x94900000|  3%| O|  |TAMS 0x94800000, 0x94800000| Untracked 
|   3|0x94900000, 0x94a00000, 0x94a00000|100%|HS|  |TAMS 0x94900000, 0x94900000| Complete 
|   4|0x94a00000, 0x94b00000, 0x94b00000|100%|HC|  |TAMS 0x94a00000, 0x94a00000| Complete 
|   5|0x94b00000, 0x94c00000, 0x94c00000|100%|HC|  |TAMS 0x94b00000, 0x94b00000| Complete 
|   6|0x94c00000, 0x94d00000, 0x94d00000|100%| O|  |TAMS 0x94c00000, 0x94c00000| Untracked 
|   7|0x94d00000, 0x94e00000, 0x94e00000|100%| O|  |TAMS 0x94d00000, 0x94d00000| Untracked 
|   8|0x94e00000, 0x94e00000, 0x94f00000|  0%| F|  |TAMS 0x94e00000, 0x94e00000| Untracked 
|   9|0x94f00000, 0x94f00000, 0x95000000|  0%| F|  |TAMS 0x94f00000, 0x94f00000| Untracked 
|  10|0x95000000, 0x95000000, 0x95100000|  0%| F|  |TAMS 0x95000000, 0x95000000| Untracked 
|  11|0x95100000, 0x95100000, 0x95200000|  0%| F|  |TAMS 0x95100000, 0x95100000| Untracked 
|  12|0x95200000, 0x95200000, 0x95300000|  0%| F|  |TAMS 0x95200000, 0x95200000| Untracked 
|  13|0x95300000, 0x95300000, 0x95400000|  0%| F|  |TAMS 0x95300000, 0x95300000| Untracked 
|  14|0x95400000, 0x95400000, 0x95500000|  0%| F|  |TAMS 0x95400000, 0x95400000| Untracked 
|  15|0x95500000, 0x95500000, 0x95600000|  0%| F|  |TAMS 0x95500000, 0x95500000| Untracked 
|  16|0x95600000, 0x95600000, 0x95700000|  0%| F|  |TAMS 0x95600000, 0x95600000| Untracked 
|  17|0x95700000, 0x95700000, 0x95800000|  0%| F|  |TAMS 0x95700000, 0x95700000| Untracked 
|  18|0x95800000, 0x95800000, 0x95900000|  0%| F|  |TAMS 0x95800000, 0x95800000| Untracked 
|  19|0x95900000, 0x95900000, 0x95a00000|  0%| F|  |TAMS 0x95900000, 0x95900000| Untracked 
|  20|0x95a00000, 0x95a00000, 0x95b00000|  0%| F|  |TAMS 0x95a00000, 0x95a00000| Untracked 
|  21|0x95b00000, 0x95b00000, 0x95c00000|  0%| F|  |TAMS 0x95b00000, 0x95b00000| Untracked 
|  22|0x95c00000, 0x95c00000, 0x95d00000|  0%| F|  |TAMS 0x95c00000, 0x95c00000| Untracked 
|  23|0x95d00000, 0x95db7230, 0x95e00000| 71%| S|CS|TAMS 0x95d00000, 0x95d00000| Complete 
|  24|0x95e00000, 0x95e00000, 0x95f00000|  0%| F|  |TAMS 0x95e00000, 0x95e00000| Untracked 
|  25|0x95f00000, 0x95f00000, 0x96000000|  0%| F|  |TAMS 0x95f00000, 0x95f00000| Untracked 
|  26|0x96000000, 0x96000000, 0x96100000|  0%| F|  |TAMS 0x96000000, 0x96000000| Untracked 
|  27|0x96100000, 0x96100000, 0x96200000|  0%| F|  |TAMS 0x96100000, 0x96100000| Untracked 
|  28|0x96200000, 0x96200000, 0x96300000|  0%| F|  |TAMS 0x96200000, 0x96200000| Untracked 
|  29|0x96300000, 0x9632df98, 0x96400000| 17%| E|  |TAMS 0x96300000, 0x96300000| Complete 
|  30|0x96400000, 0x96500000, 0x96500000|100%| E|CS|TAMS 0x96400000, 0x96400000| Complete 
|  31|0x96500000, 0x96600000, 0x96600000|100%| E|CS|TAMS 0x96500000, 0x96500000| Complete 

Card table byte_map: [0x94504000,0x94600000] _byte_map_base: 0x94061000

Marking Bits (Prev, Next): (CMBitMap*) 0xb636c6d4, (CMBitMap*) 0xb636c6f0
 Prev Bits: [0x93c28000, 0x94408000)
 Next Bits: [0x93448000, 0x93c28000)

Polling page: 0xb6f9c000

Metaspace:

Usage:
   10.78 MB capacity,    10.67 MB (>99%) used,    85.94 KB ( <1%) free+waste,    23.44 KB ( <1%) overhead. 

Virtual space:
     11.30 MB reserved,      10.90 MB ( 96%) committed 

Chunk freelists:
5.50 KB

MaxMetaspaceSize: unlimited


CodeCache: size=32768Kb used=546Kb max_used=559Kb free=32221Kb
 bounds [0xb4122000, 0xb42a2000, 0xb6122000]
 total_blobs=814 nmethods=262 adapters=117
 compilation: enabled
              stopped_count=0, restarted_count=0
 full_count=0

Compilation events (20 events):
Event: 8.377 Thread 0x92730400  244             java.io.BufferedInputStream::getBufIfOpen (21 bytes)
Event: 8.379 Thread 0x92730400 nmethod 244 0xb41a1b88 code [0xb41a1c80, 0xb41a1cfc]
Event: 8.414 Thread 0x92730400  245             java.util.HashSet::add (20 bytes)
Event: 8.421 Thread 0x92730400 nmethod 245 0xb41a3a08 code [0xb41a3b10, 0xb41a3c48]
Event: 8.444 Thread 0x92730400  246             java.lang.String::indexOf (7 bytes)
Event: 8.457 Thread 0x92730400 nmethod 246 0xb41a3688 code [0xb41a3780, 0xb41a390c]
Event: 8.582 Thread 0x92730400  247             sun.util.locale.LocaleUtils::isUpper (18 bytes)
Event: 8.584 Thread 0x92730400 nmethod 247 0xb41a3488 code [0xb41a3580, 0xb41a35cc]
Event: 8.600 Thread 0x92730400  248             java.util.concurrent.ConcurrentHashMap::get (162 bytes)
Event: 8.618 Thread 0x92730400 nmethod 248 0xb41a4f08 code [0xb41a5020, 0xb41a529c]
Event: 8.620 Thread 0x92730400  249             sun.util.locale.LocaleUtils::toLower (17 bytes)
Event: 8.621 Thread 0x92730400 nmethod 249 0xb41a7988 code [0xb41a7a80, 0xb41a7adc]
Event: 8.799 Thread 0x92730400  259             java.lang.StringCoding::decodeUTF8_0 (897 bytes)
Event: 8.858 Thread 0x92730400 nmethod 259 0xb41a8408 code [0xb41a8540, 0xb41a8af4]
Event: 8.859 Thread 0x92730400  260             java.lang.Character::toLowerCase (9 bytes)
Event: 8.869 Thread 0x92730400 nmethod 260 0xb41a8088 code [0xb41a8190, 0xb41a82bc]
Event: 8.879 Thread 0x92730400  261  s          java.io.BufferedInputStream::read (49 bytes)
Event: 8.893 Thread 0x92730400 nmethod 261 0xb41a7b88 code [0xb41a7c90, 0xb41a7fa8]
Event: 8.897 Thread 0x92730400  262 %           org.sqlite.SQLiteJDBCLoader::contentsEquals @ 37 (78 bytes)
Event: 8.938 Thread 0x92730400 nmethod 262% 0xb41ac688 code [0xb41ac7b0, 0xb41acef4]

GC Heap History (8 events):
Event: 6.121 GC heap before
{Heap before GC invocations=0 (full 0):
 garbage-first heap   total 32768K, used 20480K [0x94600000, 0xb3e00000)
  region size 1024K, 14 young (14336K), 0 survivors (0K)
 Metaspace       used 8153K, capacity 8258K, committed 8344K, reserved 8496K
}
Event: 6.177 GC heap after
{Heap after GC invocations=1 (full 0):
 garbage-first heap   total 32768K, used 6549K [0x94600000, 0xb3e00000)
  region size 1024K, 2 young (2048K), 2 survivors (2048K)
 Metaspace       used 8153K, capacity 8258K, committed 8344K, reserved 8496K
}
Event: 6.574 GC heap before
{Heap before GC invocations=1 (full 0):
 garbage-first heap   total 32768K, used 7573K [0x94600000, 0xb3e00000)
  region size 1024K, 3 young (3072K), 2 survivors (2048K)
 Metaspace       used 8735K, capacity 8837K, committed 8984K, reserved 9520K
}
Event: 6.604 GC heap after
{Heap after GC invocations=2 (full 0):
 garbage-first heap   total 32768K, used 6858K [0x94600000, 0xb3e00000)
  region size 1024K, 1 young (1024K), 1 survivors (1024K)
 Metaspace       used 8735K, capacity 8837K, committed 8984K, reserved 9520K
}
Event: 6.928 GC heap before
{Heap before GC invocations=2 (full 0):
 garbage-first heap   total 32768K, used 9930K [0x94600000, 0xb3e00000)
  region size 1024K, 4 young (4096K), 1 survivors (1024K)
 Metaspace       used 8801K, capacity 8914K, committed 8984K, reserved 9520K
}
Event: 6.937 GC heap after
{Heap after GC invocations=3 (full 0):
 garbage-first heap   total 32768K, used 7308K [0x94600000, 0xb3e00000)
  region size 1024K, 1 young (1024K), 1 survivors (1024K)
 Metaspace       used 8801K, capacity 8914K, committed 8984K, reserved 9520K
}
Event: 8.179 GC heap before
{Heap before GC invocations=3 (full 0):
 garbage-first heap   total 32768K, used 13452K [0x94600000, 0xb3e00000)
  region size 1024K, 8 young (8192K), 1 survivors (1024K)
 Metaspace       used 10075K, capacity 10173K, committed 10264K, reserved 10544K
}
Event: 8.199 GC heap after
{Heap after GC invocations=4 (full 0):
 garbage-first heap   total 32768K, used 7933K [0x94600000, 0xb3e00000)
  region size 1024K, 1 young (1024K), 1 survivors (1024K)
 Metaspace       used 10075K, capacity 10173K, committed 10264K, reserved 10544K
}

Deoptimization events (20 events):
Event: 6.427 Thread 0xb630d800 Uncommon trap: trap_request=0xffffff4d fr.pc=0xb419c720 relative=0x00000f90
Event: 6.427 Thread 0xb630d800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xb419c720 method=java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object; @ 81 c2
Event: 6.427 Thread 0xb630d800 DEOPT PACKING pc=0xb419c720 sp=0xb6485410
Event: 6.427 Thread 0xb630d800 DEOPT UNPACKING pc=0xb4137604 sp=0xb64853f8 mode 2
Event: 7.105 Thread 0xb630d800 Uncommon trap: trap_request=0xffffff4d fr.pc=0xb41873e4 relative=0x000009e4
Event: 7.105 Thread 0xb630d800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xb41873e4 method=java.lang.CharacterData.of(I)Ljava/lang/CharacterData; @ 4 c2
Event: 7.105 Thread 0xb630d800 DEOPT PACKING pc=0xb41873e4 sp=0xb64856e8
Event: 7.105 Thread 0xb630d800 DEOPT UNPACKING pc=0xb4137604 sp=0xb6485664 mode 2
Event: 7.876 Thread 0xb630d800 Uncommon trap: trap_request=0xffffff4d fr.pc=0xb419a128 relative=0x00000188
Event: 7.876 Thread 0xb630d800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xb419a128 method=java.util.HashMap.getNode(ILjava/lang/Object;)Ljava/util/HashMap$Node; @ 62 c2
Event: 7.876 Thread 0xb630d800 DEOPT PACKING pc=0xb419a128 sp=0xb6484cb8
Event: 7.876 Thread 0xb630d800 DEOPT UNPACKING pc=0xb4137604 sp=0xb6484c8c mode 2
Event: 8.801 Thread 0xb630d800 Uncommon trap: trap_request=0xffffff4d fr.pc=0xb418c5a8 relative=0x00000118
Event: 8.801 Thread 0xb630d800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xb418c5a8 method=java.util.regex.Pattern$Branch.match(Ljava/util/regex/Matcher;ILjava/lang/CharSequence;)Z @ 19 c2
Event: 8.801 Thread 0xb630d800 DEOPT PACKING pc=0xb418c5a8 sp=0xb64853a0
Event: 8.801 Thread 0xb630d800 DEOPT UNPACKING pc=0xb4137604 sp=0xb648538c mode 2
Event: 9.032 Thread 0xb630d800 Uncommon trap: trap_request=0xffffff4d fr.pc=0xb41acd00 relative=0x00000550
Event: 9.032 Thread 0xb630d800 Uncommon trap: reason=unstable_if action=reinterpret pc=0xb41acd00 method=org.sqlite.SQLiteJDBCLoader.contentsEquals(Ljava/io/InputStream;Ljava/io/InputStream;)Z @ 39 c2
Event: 9.032 Thread 0xb630d800 DEOPT PACKING pc=0xb41acd00 sp=0xb64854f8
Event: 9.032 Thread 0xb630d800 DEOPT UNPACKING pc=0xb4137604 sp=0xb64854dc mode 2

Classes redefined (0 events):
No events

Internal exceptions (12 events):
Event: 2.701 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x961eabb0}: 'int java.lang.invoke.DirectMethodHandle$Holder.invokeSpecialIFC(java.lang.Object, java.lang.Object, int)'> (0x961eabb0) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+10/src/hotspot/share/int
Event: 4.841 Thread 0xb630d800 Exception <a 'java/lang/NoClassDefFoundError'{0x95a529f0}: org/apache/log4j/Priority> (0x95a529f0) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+10/src/hotspot/share/classfile/systemDictionary.cpp, line 220]
Event: 5.633 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x959c92d8}: 'int java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x959c92d8) thrown at [/build/openjdk-11-LHjan6/openjdk-11-
Event: 5.652 Thread 0xb630d800 Exception <a 'java/lang/IncompatibleClassChangeError'{0x959d2ec0}: Found class java.lang.Object, but interface was expected> (0x959d2ec0) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+10/src/hotspot/share/interpreter/linkResolver.cpp, line 839]
Event: 5.657 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x959d5be8}: 'void java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object)'> (0x959d5be8) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+10/src/hotspot/share/interpr
Event: 5.671 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x959db498}: 'int java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object)'> (0x959db498) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+10/src/hotspot/share/interpre
Event: 6.426 Thread 0xb630d800 Implicit null exception at 0xb4195e38 to 0xb4195ee4
Event: 8.709 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x964aabe8}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStaticInit(java.lang.Object, java.lang.Object, long, java.lang.Object)'> (0x964aabe8) thrown at [/build/openjdk-11-LHjan6/openjdk-11
Event: 8.721 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x964b6ed0}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, long, java.lang.Object)'> (0x964b6ed0) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+10/src/hotspot/sha
Event: 8.742 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x964be098}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x964be098) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0
Event: 8.745 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x964c1d40}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object, int)'> (0x964c1d40) thrown at [/build/openjdk-11-LHj
Event: 8.747 Thread 0xb630d800 Exception <a 'java/lang/NoSuchMethodError'{0x964c53c0}: 'java.lang.Object java.lang.invoke.Invokers$Holder.linkToTargetMethod(java.lang.Object, java.lang.Object, int, java.lang.Object)'> (0x964c53c0) thrown at [/build/openjdk-11-LHjan6/openjdk-11-11.0.5+1

Events (20 events):
Event: 8.787 loading class java/util/concurrent/ThreadPoolExecutor$Worker
Event: 8.788 loading class java/util/concurrent/ThreadPoolExecutor$Worker done
Event: 8.788 loading class java/lang/Thread$State
Event: 8.789 loading class java/lang/Thread$State done
Event: 8.790 Thread 0x916a8800 Thread added: 0x916a8800
Event: 8.791 Protecting memory [0x93103000,0x93106000] with protection modes 0
Event: 8.795 loading class java/lang/ProcessBuilder$NullInputStream
Event: 8.795 loading class java/lang/ProcessBuilder$NullInputStream done
Event: 8.795 loading class java/io/FileOutputStream$1
Event: 8.797 loading class java/io/FileOutputStream$1 done
Event: 8.797 loading class java/lang/ProcessBuilder$NullOutputStream
Event: 8.797 loading class java/lang/ProcessBuilder$NullOutputStream done
Event: 8.802 loading class java/util/Formattable
Event: 8.802 loading class java/util/Formattable done
Event: 8.808 loading class java/util/UUID$Holder
Event: 8.808 loading class java/util/UUID$Holder done
Event: 8.872 loading class java/io/DeleteOnExitHook
Event: 8.872 loading class java/io/DeleteOnExitHook done
Event: 8.873 loading class java/io/DeleteOnExitHook$1
Event: 8.873 loading class java/io/DeleteOnExitHook$1 done


Dynamic libraries:
004e6000-004e7000 r-xp 00000000 b3:01 792397     /usr/lib/jvm/java-11-openjdk-armhf/bin/java
004f6000-004f7000 r--p 00000000 b3:01 792397     /usr/lib/jvm/java-11-openjdk-armhf/bin/java
004f7000-004f8000 rw-p 00001000 b3:01 792397     /usr/lib/jvm/java-11-openjdk-armhf/bin/java
00dac000-00dcd000 rw-p 00000000 00:00 0          [heap]
911ac000-9126a000 r-xp 00000000 b3:01 20269      /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so
9126a000-9127a000 ---p 000be000 b3:01 20269      /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so
9127a000-9127c000 r--p 000be000 b3:01 20269      /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so
9127c000-9127e000 rw-p 000c0000 b3:01 20269      /tmp/sqlite-3.28.0-a5aa2bef-6af5-4c28-919e-684ebb68d93c-libsqlitejdbc.so
9127e000-9131e000 rw-p 00000000 00:00 0 
9131e000-9137e000 ---p 00000000 00:00 0 
9137e000-9147e000 rw-p 00000000 00:00 0 
9147e000-9157e000 rw-p 00000000 00:00 0 
9157e000-9157f000 ---p 00000000 00:00 0 
9157f000-91600000 rw-p 00000000 00:00 0 
91600000-916bb000 rw-p 00000000 00:00 0 
916bb000-91700000 ---p 00000000 00:00 0 
91700000-91800000 rw-p 00000000 00:00 0 
91800000-91900000 rw-p 00000000 00:00 0 
91900000-91a00000 rw-p 00000000 00:00 0 
91a00000-91b00000 rw-p 00000000 00:00 0 
91b00000-91bff000 rw-p 00000000 00:00 0 
91bff000-91c00000 ---p 00000000 00:00 0 
91c00000-91ce3000 rw-p 00000000 00:00 0 
91ce3000-91d00000 ---p 00000000 00:00 0 
91d00000-91e00000 rw-p 00000000 00:00 0 
91e62000-91e63000 ---p 00000000 00:00 0 
91e63000-91ee4000 rw-p 00000000 00:00 0 
91ee4000-91ee7000 ---p 00000000 00:00 0 
91ee7000-91f34000 rw-p 00000000 00:00 0 
91f34000-91f35000 ---p 00000000 00:00 0 
91f35000-91fb6000 rw-p 00000000 00:00 0 
91fb6000-91fb9000 ---p 00000000 00:00 0 
91fb9000-92006000 rw-p 00000000 00:00 0 
92006000-92009000 ---p 00000000 00:00 0 
92009000-92086000 rw-p 00000000 00:00 0 
92086000-92300000 r--p 0006b000 b3:01 5229       /usr/lib/locale/locale-archive
92300000-92500000 r--p 00000000 b3:01 5229       /usr/lib/locale/locale-archive
92500000-92521000 rw-p 00000000 00:00 0 
92521000-92600000 ---p 00000000 00:00 0 
92610000-92613000 ---p 00000000 00:00 0 
92613000-92660000 rw-p 00000000 00:00 0 
92660000-92663000 ---p 00000000 00:00 0 
92663000-926b0000 rw-p 00000000 00:00 0 
926b0000-926b3000 ---p 00000000 00:00 0 
926b3000-92700000 rw-p 00000000 00:00 0 
92700000-92800000 rw-p 00000000 00:00 0 
92832000-92833000 ---p 00000000 00:00 0 
92833000-92cfa000 rw-p 00000000 00:00 0 
92cfa000-92d00000 ---p 00000000 00:00 0 
92d00000-92d21000 rw-p 00000000 00:00 0 
92d21000-92e00000 ---p 00000000 00:00 0 
92e2e000-92e31000 ---p 00000000 00:00 0 
92e31000-92e7e000 rw-p 00000000 00:00 0 
92e7e000-92e7f000 ---p 00000000 00:00 0 
92e7f000-92f00000 rw-p 00000000 00:00 0 
92f00000-92fff000 rw-p 00000000 00:00 0 
92fff000-93000000 ---p 00000000 00:00 0 
93000000-93021000 rw-p 00000000 00:00 0 
93021000-93100000 ---p 00000000 00:00 0 
93103000-93106000 ---p 00000000 00:00 0 
93106000-9317e000 rw-p 00000000 00:00 0 
9317e000-9317f000 ---p 00000000 00:00 0 
9317f000-93200000 rw-p 00000000 00:00 0 
93200000-93221000 rw-p 00000000 00:00 0 
93221000-93300000 ---p 00000000 00:00 0 
93317000-93332000 r-xp 00000000 b3:01 792552     /usr/lib/jvm/java-11-openjdk-armhf/lib/libsunec.so
93332000-93341000 ---p 0001b000 b3:01 792552     /usr/lib/jvm/java-11-openjdk-armhf/lib/libsunec.so
93341000-93343000 r--p 0001a000 b3:01 792552     /usr/lib/jvm/java-11-openjdk-armhf/lib/libsunec.so
93343000-93344000 rw-p 0001c000 b3:01 792552     /usr/lib/jvm/java-11-openjdk-armhf/lib/libsunec.so
93344000-93345000 ---p 00000000 00:00 0 
93345000-933c6000 rw-p 00000000 00:00 0 
933c6000-933c7000 ---p 00000000 00:00 0 
933c7000-934c8000 rw-p 00000000 00:00 0 
934c8000-93c28000 ---p 00000000 00:00 0 
93c28000-93ca8000 rw-p 00000000 00:00 0 
93ca8000-94408000 ---p 00000000 00:00 0 
94408000-94418000 rw-p 00000000 00:00 0 
94418000-94504000 ---p 00000000 00:00 0 
94504000-94514000 rw-p 00000000 00:00 0 
94514000-94600000 ---p 00000000 00:00 0 
94600000-96600000 rw-p 00000000 00:00 0 
96600000-b3e00000 ---p 00000000 00:00 0 
b3e04000-b3e14000 rw-p 00000000 00:00 0 
b3e14000-b3f00000 ---p 00000000 00:00 0 
b3f00000-b3f21000 rw-p 00000000 00:00 0 
b3f21000-b4000000 ---p 00000000 00:00 0 
b4005000-b4014000 r-xp 00000000 b3:01 792546     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnet.so
b4014000-b4023000 ---p 0000f000 b3:01 792546     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnet.so
b4023000-b4024000 r--p 0000e000 b3:01 792546     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnet.so
b4024000-b4025000 rw-p 0000f000 b3:01 792546     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnet.so
b4025000-b402f000 r-xp 00000000 b3:01 792547     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnio.so
b402f000-b403e000 ---p 0000a000 b3:01 792547     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnio.so
b403e000-b403f000 r--p 00009000 b3:01 792547     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnio.so
b403f000-b4040000 rw-p 0000a000 b3:01 792547     /usr/lib/jvm/java-11-openjdk-armhf/lib/libnio.so
b4040000-b4060000 rw-p 00000000 00:00 0 
b4060000-b4061000 ---p 00000000 00:00 0 
b4061000-b40e5000 rw-p 00000000 00:00 0 
b40e5000-b4122000 ---p 00000000 00:00 0 
b4122000-b42a2000 rwxp 00000000 00:00 0 
b42a2000-b6122000 ---p 00000000 00:00 0 
b6122000-b628e000 r--s 00000000 b3:01 792556     /usr/lib/jvm/java-11-openjdk-armhf/lib/modules
b628e000-b6291000 r-xp 00000000 b3:01 792538     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjimage.so
b6291000-b62a0000 ---p 00003000 b3:01 792538     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjimage.so
b62a0000-b62a1000 r--p 00002000 b3:01 792538     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjimage.so
b62a1000-b62a2000 rw-p 00003000 b3:01 792538     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjimage.so
b62a2000-b62a7000 r-xp 00000000 b3:01 792555     /usr/lib/jvm/java-11-openjdk-armhf/lib/libzip.so
b62a7000-b62b6000 ---p 00005000 b3:01 792555     /usr/lib/jvm/java-11-openjdk-armhf/lib/libzip.so
b62b6000-b62b7000 r--p 00004000 b3:01 792555     /usr/lib/jvm/java-11-openjdk-armhf/lib/libzip.so
b62b7000-b62b8000 rw-p 00005000 b3:01 792555     /usr/lib/jvm/java-11-openjdk-armhf/lib/libzip.so
b62b8000-b62bf000 r-xp 00000000 b3:01 3424       /usr/lib/arm-linux-gnueabihf/libnss_files-2.28.so
b62bf000-b62ce000 ---p 00007000 b3:01 3424       /usr/lib/arm-linux-gnueabihf/libnss_files-2.28.so
b62ce000-b62cf000 r--p 00006000 b3:01 3424       /usr/lib/arm-linux-gnueabihf/libnss_files-2.28.so
b62cf000-b62d0000 rw-p 00007000 b3:01 3424       /usr/lib/arm-linux-gnueabihf/libnss_files-2.28.so
b62d0000-b62d6000 rw-p 00000000 00:00 0 
b62d6000-b62ee000 r-xp 00000000 b3:01 792534     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjava.so
b62ee000-b62fe000 ---p 00018000 b3:01 792534     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjava.so
b62fe000-b62ff000 r--p 00018000 b3:01 792534     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjava.so
b62ff000-b6300000 rw-p 00019000 b3:01 792534     /usr/lib/jvm/java-11-openjdk-armhf/lib/libjava.so
b6300000-b63f9000 rw-p 00000000 00:00 0 
b63f9000-b6400000 ---p 00000000 00:00 0 
b6408000-b640f000 r-xp 00000000 b3:01 792554     /usr/lib/jvm/java-11-openjdk-armhf/lib/libverify.so
b640f000-b641f000 ---p 00007000 b3:01 792554     /usr/lib/jvm/java-11-openjdk-armhf/lib/libverify.so
b641f000-b6420000 r--p 00007000 b3:01 792554     /usr/lib/jvm/java-11-openjdk-armhf/lib/libverify.so
b6420000-b6421000 rw-p 00008000 b3:01 792554     /usr/lib/jvm/java-11-openjdk-armhf/lib/libverify.so
b6421000-b6426000 r-xp 00000000 b3:01 3431       /usr/lib/arm-linux-gnueabihf/librt-2.28.so
b6426000-b6435000 ---p 00005000 b3:01 3431       /usr/lib/arm-linux-gnueabihf/librt-2.28.so
b6435000-b6436000 r--p 00004000 b3:01 3431       /usr/lib/arm-linux-gnueabihf/librt-2.28.so
b6436000-b6437000 rw-p 00005000 b3:01 3431       /usr/lib/arm-linux-gnueabihf/librt-2.28.so
b6437000-b643a000 ---p 00000000 00:00 0 
b643a000-b6487000 rw-p 00000000 00:00 0 
b6487000-b649f000 r-xp 00000000 b3:01 3409       /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
b649f000-b64ae000 ---p 00018000 b3:01 3409       /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
b64ae000-b64af000 r--p 00017000 b3:01 3409       /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
b64af000-b64b0000 rw-p 00018000 b3:01 3409       /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
b64b0000-b651a000 r-xp 00000000 b3:01 3419       /usr/lib/arm-linux-gnueabihf/libm-2.28.so
b651a000-b6529000 ---p 0006a000 b3:01 3419       /usr/lib/arm-linux-gnueabihf/libm-2.28.so
b6529000-b652a000 r--p 00069000 b3:01 3419       /usr/lib/arm-linux-gnueabihf/libm-2.28.so
b652a000-b652b000 rw-p 0006a000 b3:01 3419       /usr/lib/arm-linux-gnueabihf/libm-2.28.so
b652b000-b661e000 r-xp 00000000 b3:01 3410       /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.25
b661e000-b662d000 ---p 000f3000 b3:01 3410       /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.25
b662d000-b6632000 r--p 000f2000 b3:01 3410       /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.25
b6632000-b6634000 rw-p 000f7000 b3:01 3410       /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.25
b6634000-b6636000 rw-p 00000000 00:00 0 
b6636000-b6d66000 r-xp 00000000 b3:01 792563     /usr/lib/jvm/java-11-openjdk-armhf/lib/server/libjvm.so
b6d66000-b6d75000 ---p 00730000 b3:01 792563     /usr/lib/jvm/java-11-openjdk-armhf/lib/server/libjvm.so
b6d75000-b6daf000 r--p 0072f000 b3:01 792563     /usr/lib/jvm/java-11-openjdk-armhf/lib/server/libjvm.so
b6daf000-b6dcf000 rw-p 00769000 b3:01 792563     /usr/lib/jvm/java-11-openjdk-armhf/lib/server/libjvm.so
b6dcf000-b6df9000 rw-p 00000000 00:00 0 
b6df9000-b6edd000 r-xp 00000000 b3:01 3416       /usr/lib/arm-linux-gnueabihf/libc-2.28.so
b6edd000-b6eed000 ---p 000e4000 b3:01 3416       /usr/lib/arm-linux-gnueabihf/libc-2.28.so
b6eed000-b6eef000 r--p 000e4000 b3:01 3416       /usr/lib/arm-linux-gnueabihf/libc-2.28.so
b6eef000-b6ef0000 rw-p 000e6000 b3:01 3416       /usr/lib/arm-linux-gnueabihf/libc-2.28.so
b6ef0000-b6ef3000 rw-p 00000000 00:00 0 
b6ef3000-b6ef5000 r-xp 00000000 b3:01 3418       /usr/lib/arm-linux-gnueabihf/libdl-2.28.so
b6ef5000-b6f04000 ---p 00002000 b3:01 3418       /usr/lib/arm-linux-gnueabihf/libdl-2.28.so
b6f04000-b6f05000 r--p 00001000 b3:01 3418       /usr/lib/arm-linux-gnueabihf/libdl-2.28.so
b6f05000-b6f06000 rw-p 00002000 b3:01 3418       /usr/lib/arm-linux-gnueabihf/libdl-2.28.so
b6f06000-b6f10000 r-xp 00000000 b3:01 792518     /usr/lib/jvm/java-11-openjdk-armhf/lib/jli/libjli.so
b6f10000-b6f1f000 ---p 0000a000 b3:01 792518     /usr/lib/jvm/java-11-openjdk-armhf/lib/jli/libjli.so
b6f1f000-b6f20000 r--p 00009000 b3:01 792518     /usr/lib/jvm/java-11-openjdk-armhf/lib/jli/libjli.so
b6f20000-b6f21000 rw-p 0000a000 b3:01 792518     /usr/lib/jvm/java-11-openjdk-armhf/lib/jli/libjli.so
b6f21000-b6f33000 r-xp 00000000 b3:01 3429       /usr/lib/arm-linux-gnueabihf/libpthread-2.28.so
b6f33000-b6f42000 ---p 00012000 b3:01 3429       /usr/lib/arm-linux-gnueabihf/libpthread-2.28.so
b6f42000-b6f43000 r--p 00011000 b3:01 3429       /usr/lib/arm-linux-gnueabihf/libpthread-2.28.so
b6f43000-b6f44000 rw-p 00012000 b3:01 3429       /usr/lib/arm-linux-gnueabihf/libpthread-2.28.so
b6f44000-b6f46000 rw-p 00000000 00:00 0 
b6f46000-b6f59000 r-xp 00000000 b3:01 3525       /usr/lib/arm-linux-gnueabihf/libz.so.1.2.11
b6f59000-b6f68000 ---p 00013000 b3:01 3525       /usr/lib/arm-linux-gnueabihf/libz.so.1.2.11
b6f68000-b6f69000 r--p 00012000 b3:01 3525       /usr/lib/arm-linux-gnueabihf/libz.so.1.2.11
b6f69000-b6f6a000 rw-p 00013000 b3:01 3525       /usr/lib/arm-linux-gnueabihf/libz.so.1.2.11
b6f70000-b6f79000 rw-p 00000000 00:00 0 
b6f79000-b6f91000 r-xp 00000000 b3:01 3412       /usr/lib/arm-linux-gnueabihf/ld-2.28.so
b6f92000-b6f94000 r--s 00007000 b3:01 664978     /usr/share/java/java-atk-wrapper.jar
b6f94000-b6f9c000 rw-s 00000000 b3:01 410479     /tmp/hsperfdata_ma1sd/28628
b6f9c000-b6f9d000 r--p 00000000 00:00 0 
b6f9d000-b6fa1000 rw-p 00000000 00:00 0 
b6fa1000-b6fa2000 r--p 00018000 b3:01 3412       /usr/lib/arm-linux-gnueabihf/ld-2.28.so
b6fa2000-b6fa3000 rw-p 00019000 b3:01 3412       /usr/lib/arm-linux-gnueabihf/ld-2.28.so
be8fb000-be91c000 rw-p 00000000 00:00 0          [stack]
bed8f000-bed90000 r-xp 00000000 00:00 0          [sigpage]
ffff0000-ffff1000 r-xp 00000000 00:00 0          [vectors]


VM Arguments:
java_command: /usr/lib/ma1sd/ma1sd.jar -c /etc/ma1sd/ma1sd.yaml
java_class_path (initial): /usr/lib/ma1sd/ma1sd.jar
Launcher Type: SUN_STANDARD

[Global flags]
     uint ConcGCThreads                            = 1                                         {product} {ergonomic}
     uint G1ConcRefinementThreads                  = 4                                         {product} {ergonomic}
   size_t G1HeapRegionSize                         = 1048576                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 33554432                                  {product} {ergonomic}
   size_t MarkStackSize                            = 32768                                     {product} {ergonomic}
   size_t MaxHeapSize                              = 528482304                                 {product} {ergonomic}
   size_t MaxNewSize                               = 316669952                                 {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 1048576                                   {product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 0                                      {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 0                                      {pd product} {ergonomic}
     bool UseAOT                                   = false                                     {product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

Logging:
Log output configuration:
 #0: stdout all=warning uptime,level,tags
 #1: stderr all=off uptime,level,tags

Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/sh

Signal Handlers:
SIGSEGV: [libjvm.so+0x6966b5], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0x6966b5], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x6966b5], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x555281], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x555281], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x6966b5], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0x55519d], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x55583d], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0x55583d], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0x55583d], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x55583d], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
uname:Linux 4.9.150-imx6-sr #1 SMP Sun Sep 8 01:52:51 UTC 2019 armv7l
libc:glibc 2.28 NPTL 2.28 
rlimit: STACK 8192k, CORE 0k, NPROC 13714, NOFILE 524288, AS infinity, DATA infinity, FSIZE infinity
load average:1.70 0.60 0.32

/proc/meminfo:
MemTotal:        2063228 kB
MemFree:           66652 kB
MemAvailable:    1244896 kB
Buffers:           85276 kB
Cached:          1190160 kB
SwapCached:            0 kB
Active:           742244 kB
Inactive:         971452 kB
Active(anon):     464228 kB
Inactive(anon):    46240 kB
Active(file):     278016 kB
Inactive(file):   925212 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:       1343488 kB
HighFree:          11168 kB
LowTotal:         719740 kB
LowFree:           55484 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              1328 kB
Writeback:             0 kB
AnonPages:        438296 kB
Mapped:           158468 kB
Shmem:             72208 kB
Slab:              89592 kB
SReclaimable:      48424 kB
SUnreclaim:        41168 kB
KernelStack:        2648 kB
PageTables:         7976 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1031612 kB
Committed_AS:    1680088 kB
VmallocTotal:     245760 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:         294912 kB
CmaFree:             472 kB


/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
27428


/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
65530


/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
32768



container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-3
cpu_memory_nodes: 0
active_processor_count: 4
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -1
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 66686976
memory_max_usage_in_bytes: 66686976

Steal ticks since vm start: 0
Steal ticks percentage since vm start:  0.000

CPU:total 4 (initial active 4) (ARMv7), vfp, vfp3-32, simd

/proc/cpuinfo:
processor	: 0
model name	: ARMv7 Processor rev 10 (v7l)
BogoMIPS	: 7.54
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc09
CPU revision	: 10

processor	: 1
model name	: ARMv7 Processor rev 10 (v7l)
BogoMIPS	: 7.54
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc09
CPU revision	: 10

processor	: 2
model name	: ARMv7 Processor rev 10 (v7l)
BogoMIPS	: 7.54
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc09
CPU revision	: 10

processor	: 3
model name	: ARMv7 Processor rev 10 (v7l)
BogoMIPS	: 7.54
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc09
CPU revision	: 10

Hardware	: Freescale i.MX6 Quad/DualLite (Device Tree)
Revision	: 63015
Serial		: 0000000000000000

Memory: 4k page, physical 2063228k(66652k free), swap 0k(0k free)

vm_info: OpenJDK Server VM (11.0.5+10-post-Debian-1deb10u1) for linux- JRE (11.0.5+10-post-Debian-1deb10u1), built on Oct 18 2019 18:28:52 by "buildd" with gcc 8.3.0

END.

Invitation - LDAP

Hi
I'm testing matrix-synapse with ma1sd and active directory integration. I would like to use the "Invites-Only" feature to give a group of users permission to invite external contacts so they can register on our matrix server.
How do I need to setup ma1sd to achieve this? How can I set the roles? Is possible to connect AD groups with roles?
Thanks for your help
dennis

ma1sd allows sql connections to time out

ma1sd-76f7445dc-4crnc ma1sd [XNIO-1 task-12] ERROR io.kamax.mxisd.http.undertow.handler.SaneHandler - Unknown error when handling http://matrix-identity.g4v.dev/_matrix/identity/v2/account/register
ma1sd-76f7445dc-4crnc ma1sd java.lang.RuntimeException: java.sql.SQLException: Connection has already been closed
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.withCatcher(OrmLiteSqlStorage.java:176)
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.insertToken(OrmLiteSqlStorage.java:310)
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.auth.AccountManager.register(AccountManager.java:60)
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.http.undertow.handler.auth.v2.AccountRegisterHandler.handleRequest(AccountRegisterHandler.java:54)
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.http.undertow.handler.SaneHandler.handleRequest(SaneHandler.java:71)
ma1sd-76f7445dc-4crnc ma1sd     at io.undertow.server.Connectors.executeRootHandler(Connectors.java:376)
ma1sd-76f7445dc-4crnc ma1sd     at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
ma1sd-76f7445dc-4crnc ma1sd     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
ma1sd-76f7445dc-4crnc ma1sd     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
ma1sd-76f7445dc-4crnc ma1sd     at java.lang.Thread.run(Thread.java:748)
ma1sd-76f7445dc-4crnc ma1sd Caused by: java.sql.SQLException: Connection has already been closed
ma1sd-76f7445dc-4crnc ma1sd     at com.j256.ormlite.jdbc.JdbcConnectionSource.getReadWriteConnection(JdbcConnectionSource.java:182)
ma1sd-76f7445dc-4crnc ma1sd     at com.j256.ormlite.dao.BaseDaoImpl.create(BaseDaoImpl.java:327)
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.lambda$insertToken$11(OrmLiteSqlStorage.java:311)
ma1sd-76f7445dc-4crnc ma1sd     at io.kamax.mxisd.storage.ormlite.OrmLiteSqlStorage.withCatcher(OrmLiteSqlStorage.java:174)
ma1sd-76f7445dc-4crnc ma1sd     ... 9 more

Migrating from mxisd to ma1sd

Not really an issue, but to avoid downtime I'm just checking if the process is seamless.

Now it's had a while to stabilise I want to remove mxisd and migrate to ma1sd.

Is there a document that describes the process, indicating any changes required?
Are there any features removed/changed that may trip me up?
Any other advice?

I'm running latest Debian package for matrix-synapse 1.8.0.
mxisd is from 1.4.3-2 it's been running fine for ages.

Feature: Add possibility to search user_directory of another homeserver

I have a synapse homserver my.homeserver with ma1sd that uses our own small LDAP server as a backend. At our university we have another federated homeserver (other.homeserver) without ma1sd. I would like to include the user_directory of other.homeserver in the search results. It could be done using the EXEC backend, as follows.

Configuration snippet /etc/ma1sd/ma1sd.yaml

exec:
  directory:
    enabled: true
    search:
      byName:
        command: '/opt/ma1sd-exec/other.homserver/user-directory.py'
        args: ['{query}']
        output:
          type: 'json'

EXEC script /opt/ma1sd-exec/other.homserver/user-directory.py:

#!/usr/bin/env python3

import argparse
import requests
import json

from secret import access_token

parser = argparse.ArgumentParser()
parser.add_argument("term", help="search term")
args = parser.parse_args()

data = {'search_term': args.term}
json_data = json.dumps(data)

response = requests.post(
    'https://other.homserver/_matrix/client/r0/user_directory/search',
    headers={'Authorization': ' '.join(['Bearer', access_token])},
    data=json_data)

json_response = response.json()
json_filtered = {'limited': json_response['limited'], 'results': []}

for user in json_response['results']:
    if user['user_id'].endswith(':other.homserver'):
        json_filtered['results'].append(user)

print(json.dumps(json_filtered, indent=4, sort_keys=True))

The script queries https://other.homserver/_matrix/client/r0/user_directory/search and filters the results:

$ /opt/ma1sd-exec/other.homserver/user-directory.py user
{
    "limited": false,
    "results": [
        {
            "avatar_url": null,
            "display_name": "User",
            "user_id": "@user:other.homserver"
        }
    ]
}

Since the backend expects the localpart of the user_id and always appends the my.homeserver domain, ma1sd returns the following mxid in the search results:

@@user:other.homserver:my.homeserver

It would be great to have a configuration option to allow raw mxid output from exec scripts.

I currently made the following patch to 2.4.0, which works for our use case:

diff --git a/src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java b/src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java
index 5bf4d2e..9b75533 100644
--- a/src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java
+++ b/src/main/java/io/kamax/mxisd/backend/exec/ExecDirectoryStore.java
@@ -20,6 +20,8 @@

 package io.kamax.mxisd.backend.exec;

+import java.util.regex.Matcher;
+
 import io.kamax.matrix.MatrixID;
 import io.kamax.matrix.json.GsonUtil;
 import io.kamax.mxisd.config.ExecConfig;
@@ -65,7 +67,12 @@ public class ExecDirectoryStore extends ExecStore implements DirectoryProvider {

             UserDirectorySearchResult response = GsonUtil.get().fromJson(output, UserDirectorySearchResult.class);
             for (UserDirectorySearchResult.Result result : response.getResults()) {
-                result.setUserId(MatrixID.asAcceptable(result.getUserId(), mxCfg.getDomain()).getId());
+                Matcher m = MatrixID.LAX_PATTERN.matcher(result.getUserId());
+                if (m.matches()) {
+                    result.setUserId(MatrixID.asAcceptable(result.getUserId()).getId());
+                } else {
+                    result.setUserId(MatrixID.asAcceptable(result.getUserId(), mxCfg.getDomain()).getId());
+                }
             }
             return response;
         });

Log in with ma1sd in other apps?

Hello,

I would like to know if it will be possible to use ma1sd to log to other apps than matrix?
I have in mind something like OAuth2. Not sure if this a realistic request, if there is any existing implementation or pointers about how to implement this?

Many thanks for the work!

Disabling homeserver user directory should also disable the synapse sql directory provider

Goal: We want to disable user directory search while using sql synapse the identity provider

Expectation: Setting the homserver to excluded (as per the docs) should prevent users from the homserver from being returned during user directory searches.

Actual Behavior: ma1sd uses the Synapse sql directory provider to perform searches, regardless of what the directory homserver exclusion setting is.

You can see this on these lines here: https://github.com/ma1uta/ma1sd/blob/master/src/main/java/io/kamax/mxisd/backend/sql/synapse/SynapseSqlDirectoryProvider.java#L37-L43

IMHO the SynapseSqlDirectoryProvider should observe the homeserver exclusion setting, or an additional exclusion setting should be added.

Response with 401 M_UNAUTHORIZED

Hello @ma1uta ! First thank you for implementing v2 API and supporting this project.

I have an improvement suggestion : I think ma1sd should response with 401 M_UNAUTHORIZED when the access_token is missing or invalid, as specified by the spec. It would allow clients to trying login in again (i.e. account/register) on such an error.

Today it's not a priority because matrix-js-sdk doesn't seem to handle 401, so if my is_access_token expires, it seems that riot doesn't try to register again. Still if they implement such behavior in the future they would need to have a 401 response. Right now ma1sd is answering with 500 and the following exception:
io.kamax.mxisd.exception.InvalidCredentialsException: Supplied credentials are invalid

What do you think @ma1uta ?

Email capitalization issues

Not performing sanitization (decapitalization) on email addresses consistently when storing/retrieving email invitations leads to very bad UX in certain cases.

Consider the following scenario in a email invite-only configuration:

  1. An invitation to [email protected] is causing ma1sd to send an invitation mail to the user.
  2. The user tries to register with the email address [email protected]
  3. The user gets the notification: "Your email address cannot be used for registration"

This can be even extended by the following:

  1. The inviting user assumes the error has something to do with the initial invite sent to an address with a captial letter in it and re-invites [email protected]
  2. The user tries to register again with [email protected] and it doesn't work either.

I suppose consistently performing decapitalization

  1. before storing it as a pending invite
  2. when comparing a registration attempt to the value stored in the database

would solve this issue.

Email address does not show

Hello!

Thank you very much for this plugin. It seems to be exactly what I need, though I have barely tested it so far.

Today I installed ma1sd using the matrix-docker-ansible-deploy. I configured it to connect to to our LDAP Server and now one can login to Riot using LDAP user profiles. That is cool already.

I was not able though to have Riot show the email address of a user in the general settings. The configuration I used should be correct (as far as I've taken from the docs); even the default config of the ansible script should normally work itself (but it doesn't).
This is what I have in the configuration file:

matrix_synapse_registrations_require_3pid: ["email"]
matrix_ma1sd_matrixorg_forwarding_enabled: false

matrix_ma1sd_threepid_medium_email_identity_from: "matrix@{{ matrix_domain }}"
matrix_ma1sd_threepid_medium_email_connectors_smtp_host: "mail.mymailserver.de"
matrix_ma1sd_threepid_medium_email_connectors_smtp_port: 587
matrix_ma1sd_threepid_medium_email_connectors_smtp_tls: 1
matrix_ma1sd_threepid_medium_email_connectors_smtp_login: "[email protected]"
matrix_ma1sd_threepid_medium_email_connectors_smtp_password: "mysecretpassword"

matrix_ma1sd_verbose_logging: true

matrix_synapse_ext_password_provider_rest_auth_enabled: true
matrix_synapse_ext_password_provider_rest_auth_endpoint: "http://matrix-ma1sd:8090"

matrix_ma1sd_configuration_extension_yaml: |
ldap:
enabled: true
connection:
host: 'ucs.myldapserver.com'
tls: true
port: 636
baseDNs:
- 'cn=users,dc=myhomedomain,dc=local'
bindDn: 'uid=ldapuser,cn=users,dc=myhomedomain,dc=local'
bindPassword: 'myverysecretpassword'
attribute:
name: 'displayName'
uid:
type: 'uid'
value: 'uid'
threepid:
email:
- 'email'
- 'mailPrimaryAddress'

(indentation is correct, though not correctly shown here)

The search for email addresses in the 'start chat' window is working though. It's only just strange that this search always creates log entries of the type

Mai 12 22:28:57 matrix matrix-ma1sd[26511]: [XNIO-1 task-2] INFO io.kamax.mxisd.directory.DirectoryManager - Threepid: found 1 match(es) for 'test@'
Mai 12 22:28:57 matrix matrix-ma1sd[26511]: [XNIO-1 task-2] INFO io.kamax.mxisd.directory.DirectoryManager - Threepid: found 0 match(es) for 'test@'

So immediately following each line with one or more matches there is a line with no matches for the same search term.

What am I missing? Thanks for any hint you may have!
Best, Tom

Please mandate hahsing.enabled = true when matrix.v2 = true

This relates to element-hq/element-web#12235

As as you type something that looks like an email address ([email protected]) with a domain.tld part, a match is found and shown, but after a few milliseconds it shows: Something went wrong!. The dev-tools console shows: Error: "Unsupported identity server: bad response". The ma1sd log and the tcpdump both look good.
I noticed this on ma1sd 2.2.0 and 2.3.0 with synapse 1.6.1 and 1.9.1. The problem occurs when using Riot web/desktop on Linux and Windows

The error described above occurs when the v2 API is enabled but hashing left disabled (default):

matrix:
  v2: true

hashing:
  enabled: false

When the browser requests /_matrix/identity/v2/hash_details it gets the following response:

{
  "algorithms": []
}

Riot then throws Unsupported identity server: bad response

I am not sure if the response should at least look look like this:

{
  "lookup_pepper": "x",
  "algorithms": []
}

According to MSC2134, a valid pepper value of at least length 1 is required:

The name lookup_pepper was chosen in order to account for pepper values being returned for other endpoints in the future. The contents of lookup_pepper MUST match the regular expression [a-zA-Z0-9]+, whether hashing is being performed or not. When no hashing is occuring, a valid pepper value of at least length 1 is still required.

When hashing is enabled, the error does not occur:

hashing:
  enabled: true
  pepperLength: 20
  rotationPolicy: per_requests
  hashStorageType: sql
  algorithms:
    - sha256
  delay: 2m
  requests: 10

If enabling hashing is a requirement for enabling v2 API, it should be documented.

Multi-Tenant LDAP

Hello! I was testing Matrix-Synapse with mxisd but I stopped tests because my requirements are tied tied multiple LDAP Servers (Different Active Directory Domains with trust relationships)
Is ma1sd capable to authenticate across different ldap servers?
Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.