github / github-ldap

LDAP client for humans. Part of GitHub Enterprise.

Github::Ldap

GitHub-Ldap is a wrapper on top of Net::LDAP to make it human friendly.

Installation

Add this line to your application's Gemfile:

gem 'github-ldap'

And then execute:

$ bundle

Or install it yourself as:

$ gem install github-ldap

Usage

Initialization

GitHub-Ldap let you use an external ldap server to authenticate your users with.

There are a few configuration options required to use this adapter:

• host: is the host address where the ldap server lives.
• port: is the port where the ldap server lives.
• hosts: (optional) an enumerable of pairs of hosts and corresponding ports with which to attempt opening connections (default [[host, port]]). Overrides host and port if set.
• encryption: is the encryption protocol, disabled by default. The valid options are ssl and tls.
• uid: is the field name in the ldap server used to authenticate your users, in ActiveDirectory this is sAMAccountName.

Using administrator credentials is optional but recommended. You can pass those credentials with these two options:

• admin_user: is the the ldap administrator user dn.
• admin_password: is the password for the administrator user.

Initialize a new adapter using those required options:

  ldap = GitHub::Ldap.new options

See GitHub::Ldap#initialize for additional options.

Querying

Searches are performed against an individual domain base, so the first step is to get a new GitHub::Ldap::Domain object for the connection:

  ldap = GitHub::Ldap.new options
  domain = ldap.domain("dc=github,dc=com")

When we have the domain, we can check if a user can log in with a given password:

  domain.valid_login? 'calavera', 'secret'

Or whether a user is member of the given groups:

  entry = ldap.domain('uid=calavera,dc=github,dc=com').bind
  domain.is_member? entry, %w(Enterprise)

Virtual Attributes

Some LDAP servers have support for virtual attributes, or overlays. These allow to perform queries more efficiently on the server.

To enable virtual attributes you can set the option virtual_attributes initializing the ldap connection. We use our default set of virtual names if this option is just set to true.

  ldap = GitHub::Ldap.new {virtual_attributes: true}

You can also override our defaults by providing your server mappings into a Hash. The only mapping supported for now is to check virtual membership of individuals in groups.

  ldap = GitHub::Ldap.new {virtual_attributes: {virtual_membership: 'memberOf'}}

Testing support

GitHub-Ldap uses ladle for testing. Ladle is not required by default, so you'll need to add it to your gemfile separatedly and require it.

Once you have it installed you can start the testing ldap server in the setup phase for your tests:

require 'github/ldap/server'

def setup
  GitHub::Ldap.start_server
end

def teardown
  GitHub::Ldap.stop_server
end

GitHub-Ldap includes a set of configured users for testing, but you can provide your own users into a ldif file:

def setup
  GitHub::Ldap.start_server \
    user_fixtures: ldif_path
end

If you provide your own user fixtures, you'll probably need to change the default user domain, the administrator name and her password:

def setup
  GitHub::Ldap.start_server \
    user_fixtures:  ldif_path,
    user_domain:    'dc=evilcorp,dc=com'
    admin_user:     'uid=eviladmin,dc=evilcorp,dc=com',
    admin_password: 'correct horse battery staple'
end

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Releasing

This section is for gem maintainers to cut a new version of the gem. See jch/release-scripts for original source of release scripts.

• Create a new branch from master named release-x.y.z, where x.y.z is the version to be released
• Update github-ldap.gemspec to x.y.z following semver
• Run script/changelog and paste the draft into CHANGELOG.md. Edit as needed
• Create pull request to solict feedback
• After merging the pull request, on the master branch, run script/release

github-ldap's Issues

Add support for AD recursive membership

Active directory has a super handy operator (oid:1.2.840.113556.1.4.1941) which automatically unrolls and flattens nested nested group membership efficiently server side. This will significantly improve performance for our most used ldap server.

`rake` and `rake test` fails

Running rake or rake test fails with the following (with and without bundle exec):

➜  github-ldap git:(feature/multi_domain) rake
/Users/davesims/github/github-ldap/test/test_helper.rb:20:in `<top (required)>': undefined method `after_run' for MiniTest:Module (NoMethodError)
        from /Users/davesims/github/github-ldap/test/domain_test.rb:1:in `require_relative'
        from /Users/davesims/github/github-ldap/test/domain_test.rb:1:in `<top (required)>'
        from /Users/davesims/github/github/vendor/ruby/11323111113067d8fc579071c01d83e9f161adc4/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /Users/davesims/github/github/vendor/ruby/11323111113067d8fc579071c01d83e9f161adc4/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb:10:in `block (2 levels) in <main>'
        from /Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb:9:in `each'
        from /Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb:9:in `block in <main>'
        from /Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb:4:in `select'
        from /Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb:4:in `<main>'
rake aborted!
Command failed with status (1): [ruby -I"lib:test" -I"/Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib" "/Users/davesims/github/github/vendor/gems/2.1.7/bundler/../ruby/2.1.0/gems/rake-10.5.0/lib/rake/rake_test_loader.rb" "test/**/*_test.rb" ]

Tasks: TOP => default => test
(See full trace by running task with --trace)

Workaround is to run test files directly through ruby & bundler

➜  github-ldap git:(feature/multi_domain) bundle exec ruby -Ilib:test test/**/*_test.rb
Run options: --seed 6062

# Running:

....................................................

Finished in 3.112345s, 16.7077 runs/s, 23.4550 assertions/s.

52 runs, 73 assertions, 0 failures, 0 errors, 0 skips

Add support for handling Active Directory member attribute with Range suffix

Depending on a MaxValRange setting on an Active Directory server, the server can return the member;Range=[low]-[high] (e.g. member;Range=0-1499) attribute including certain number of users instead of the member attribute including all users when a number of users in a group exceeds the MaxValRange value.

Since the filters in the lib/github/ldap/filter.rb don't care the member attribute with the Range suffix, no users can be found.

The filter should be able to handle the member attribute with the Range suffix and also it needs to support pagination if the Range suffix exists.

Net::LDAP.new deprecation warning

When encryption parameter is passed as String to Net::LDAP.new

ldap_options = {
  ...
  encryption: 'ssl',
}
ldap = GitHub::Ldap.new ldap_options

it produces the following deprecation warning
Deprecation warning: please give :encryption option as a Hash to Net::LDAP.new

At present, the encryption value must be a Hash with parameters, which consists of two keys:
method: - :simple_tls or :start_tls
tls_options: - Hash of options for that method

TCP sockets left open leading to file descriptor exhaustion

Hey!

We are evaluating github-ldap as a replacement for the LDAP team syncing, since we will move to SAML auth. We are hosting our own Github Enterprise instance internally.

We've been running this for a couple of weeks and this week we found out that the app was throwing errors (excerpt shown below).
The issue clearly states that we've reached the host's ulimit, and thus we cannot establish new connections. After listing all open TCP connections the result was that a lot of connections were left open on the client side (on CLOSE_WAIT state):

Jan 14 08:00:00 teamsync gunicorn3[31171]: github3.exceptions.ConnectionError: <class 'requests.exceptions.ConnectionError'>: A connection-level exception occurred: HTTPSConnectionPool(host='github.skroutz.gr', port=443): Max retries exceeded with url: /api/v3/app/installations?per_page=100 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f664b09b978>: Failed to establish a new connection: [Errno 24] Too many open files'))

$ netstat -ta
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp6      25      0 github-ldap.instance:40248 github-enterprise.instance:https CLOSE_WAIT 
tcp6      25      0 github-ldap.instance:51326 github-enterprise.instance:https CLOSE_WAIT 
tcp6      25      0 github-ldap.instance:58482 github-enterprise.instance:https CLOSE_WAIT 
tcp6      25      0 github-ldap.instance:41444 github-enterprise.instance:https CLOSE_WAIT 
tcp6      25      0 github-ldap.instance:45118 github-enterprise.instance:https CLOSE_WAIT 
tcp6      25      0 github-ldap.instance:34604 github-enterprise.instance:https CLOSE_WAIT 
[...]

We've dug into this and here's what we've found up to this point.

Everytime sync_all_teams runs, two new GitHubEnterprise objects that initialize a GithubSession object are created. GithubSession extends the requests.Session that holds a connection pool across multiple requests (also keeping some state such as cookies etc)

Adding attributes to _app_ctx_stack does not seem to survive across
requests, therefore on every single run, new GithubEnterprise objects are created.
Also, some reference to _app_ctx_stack.top.githubapp_installation is held somewhere, hence
these GitHubEnterprise objects are never garbage collected, alongside
their connection pools that reside in the underlying GithubSession object.

This leads to leaking requests.Session objects effectively. Eventually,
after several runs of sync_all_teams we end up reaching the host's
ulimit, and thus, no new connections can be opened after this point.

We've worked around this issue by popping the _app_ctx_stack.top from the
stack. This lets the GitHubEnterprise objects get garbage collected and
so close the open connections on our side (hanging on CLOSE_WAIT
state).

diff --git a/app.py b/app.py
index ac33caf..d435172 100644
--- a/app.py
+++ b/app.py
@@ -6,7 +6,7 @@ import logging
 
 from apscheduler.schedulers.background import BackgroundScheduler
 from apscheduler.triggers.cron import CronTrigger
-from flask import Flask
+from flask import Flask, _app_ctx_stack
 
 from githubapp import GitHubApp, LDAPClient, CRON_INTERVAL, TEST_MODE
 
@@ -258,6 +258,7 @@ def sync_all_teams():
                     slug=team.slug,
                     config_dir=config_dir
                 )
+    _app_ctx_stack.pop()
 
 
 if __name__ == '__main__':

If I understand correctly, _app_ctx_stack shouldn't be used that way, but rather use it to share state between functions for a single request and not to hold some global state. Some other in-memory store could be used for this I suppose.

I am not sure what the correct fix would be, but the more frequent the syncing is (CRON_INTERVAL), the sooner people will run into this.

Replace ApacheDS/Ladle test server with OpenLDAP in CI

Found some prior art getting OpenLDAP running for TravisCI here: https://github.com/cloudfoundry/uaa/blob/master/.travis.yml#L33

I don't know if we can replace ApacheDS/Ladle completely (local tests, for instance), but it would be great to run all tests against OpenLDAP regularly.

Connection required on initialization

When instantiating a GitHub::Ldap object, a connection is established and query performed to determine the server's capabilities which are used to configure the appropriate membership search strategies.

I'd like to avoid having to connect and query on object instantiation and only perform the query when it's absolutely needed.

Migrate off of Travis CI (travis-ci.org shutting down)

See https://blog.travis-ci.com/2021-05-07-orgshutdown for details on the shutdown of travis-ci.org.

Given builds here will inevitably stop running, we need to migrate builds off. Our best option is to migrate to GitHub Actions.

Breaking change: `validate_encryption` replaced with `tls_options`

#102 made a breaking change to the initializer. Noting this here to add a follow up PR to make it backwards compatible before the next release.

/cc @dbussink @jonabc

Github robots giving false positives

I am an active github user working with two organisations. I push code almost on daily basis and participate in issues discussion for various libraries. Recently github robots blocked my profile from public when I reported an issue for EduSec repo. I reported this problem at support and profile was restored in less than half an hour. I was actively working from my office and this blockage did quite panic for me. I think the NOT SO HARMLESS robots are programmed with weird too-strict rules. It would be great if you guys can revise the rules, to avoid false positives.

And sorry if I am reporting this issue to wrong repo, I just could not find an appropriate location to raise my concerns. I just want to send you my voice, so that this issue does not happen again.