GithubHelp home page GithubHelp logo

youtube / doorman Goto Github PK

View Code? Open in Web Editor NEW
1.9K 1.9K 118.0 10.48 MB

Doorman: Global Distributed Client Side Rate Limiting.

License: Apache License 2.0

Go 67.59% Protocol Buffer 4.25% Makefile 0.06% Python 27.83% Shell 0.27%

doorman's People

Contributors

johnsonc avatar josvisser avatar rushanny avatar ryszard avatar xn0px90 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

doorman's Issues

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

  • doc/loadtest/docker/client/client
  • doc/loadtest/docker/target/target

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.


Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Ruby client

  • Ruby client talking to the Doorman service.
  • QPS based rate limiter.
  • Installable as Gem.

Note: this may depend on the state of gRPC for Ruby.

prometheus config error related to role

level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): role missing (one of: container, pod, service, endpoint, node, apiserver)" source="main.go:149"

Client-initiated capacity update

Feature request.

For some use cases it would be great to have an option for a client-initiated capacity update.

E.g. despite of being granted a lease, the target service returns some "rate exceeded" error. A Doorman client knows that but there is no way [I found] to notify the Doorman server of it, and to shrink the quotas for other clients using the same target service.

Error while building loadtest docker server

How to resolve this ?

docker build -t glocalregistry/doorman/doorman-server:v0.1.7 docker/server/
Sending build context to Docker daemon 3.584 kB
Step 1 : FROM golang:1.5
1.5: Pulling from library/golang
357ea8c3d80b: Pull complete
52befadefd24: Pull complete
3c0732d5313c: Pull complete
fee55c622298: Pull complete
70ff2aeff174: Pull complete
01195e06f03d: Pull complete
2f0f050412f9: Pull complete
Digest: sha256:3be07b667a868a246b9cee4ddc5ecce2ad1e211958bd6043a25fc1d19d55e6ba
Status: Downloaded newer image for golang:1.5
---> 99668503de15
Step 2 : ADD config.yml .
---> cc142bba6fac
Removing intermediate container 5d804b03735d
Step 3 : RUN go get github.com/youtube/doorman/go/cmd/doorman
---> Running in 2afcdbff0a9d

github.com/youtube/doorman/proto/doorman

src/github.com/youtube/doorman/proto/doorman/doorman.pb.go:879: cannot use _Capacity_Discovery_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
src/github.com/youtube/doorman/proto/doorman/doorman.pb.go:883: cannot use _Capacity_GetCapacity_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
src/github.com/youtube/doorman/proto/doorman/doorman.pb.go:887: cannot use _Capacity_GetServerCapacity_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value
src/github.com/youtube/doorman/proto/doorman/doorman.pb.go:891: cannot use _Capacity_ReleaseCapacity_Handler (type func(interface {}, context.Context, func(interface {}) error) (interface {}, error)) as type grpc.methodHandler in field value

Python client

A Python client for Doorman:

  • A client talking to the Doorman service using gRPC.
  • A QPS based rate limiter.
  • Installable with Pip.

Your super loud ads compared to the video damage my ears

Can I sue you? I will. You caused me hearing loss and Tinnitus.
It has been a known problem for years. Have you, at this point, lowered your hiring standards so much that every single one of your engineers is too retarded to fix that issue or do you simply not care, because you are an illegal monopoly?

I WILL FIX THIS PERSONALLY IF YOU LET ME!

It is not that hard, is it?

If your engineers are too stupid to adjust the sound volume to match the video, AT LEAST make the INCREDIBLY LOUD EAR DAMAGING ADS "slowly" fade in, so I get at least 1-2 seconds to prevent them from destroying my ears.

Great, again, YouTube physically damaged me.

SINCE YOU ARE TOO STUPID TO DO SO, JUST LET ME FIX IT!

PLEASE!

WHAT DO YOU WANT FROM ME??

IF I MAKE A LIBRARY THAT LETS YOU ADJUST THE SOUNDS OF TWO VIDEOS TO EACH OTHER; WILL YOU PLEASE FUCKING USE IT?

I SWEAR TO YOU I AM GONNA DO IT!

DO YOU WANT ME TO CODE A LIBRARY FOR YOU THAT ALLOWS YOU TO FADE IN THE SOUND OF A VIDEO OVER THE COURSE OF 1-2 SECONDS?

PLEASE! I WILL DO IT! JUST ASK FOR IT!

Jeez,
fucking Hell

[leader election] master should't give up it's leadership easily

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
feature request

What happened:
There is leader election in doorman server, which is achieved by etcd. Set a key with delay ttl and continually refresh it every 1/3 delay interval.

when leader is down, this etcd key will expire. And then new leader is elected.

see the source code:

	go func() {
		for {
			log.V(2).Infof("trying to become master at %v", e.lock)
			if _, err := kapi.Set(ctx, e.lock, id, &client.SetOptions{
				TTL:       e.delay,
				PrevExist: client.PrevNoExist,
			}); err != nil {
				log.V(2).Infof("failed becoming the master, retrying in %v: %v", e.delay, err)
				time.Sleep(e.delay)
				continue
			}
			e.isMaster <- true
			log.V(2).Info("Became master at %v as %v.", e.lock, id)
			for {
				time.Sleep(e.delay / 3)
				log.V(2).Infof("Renewing mastership lease at %v as %v", e.lock, id)
				_, err := kapi.Set(ctx, e.lock, id, &client.SetOptions{
					TTL:       e.delay,
					PrevExist: client.PrevExist,
					PrevValue: id,
				})

				if err != nil {
					log.V(2).Info("lost mastership")
					e.isMaster <- false
					break
				}
			}
		}
	}()

when master fail to renew lease because some temp reasons, for example network jitter, it just loses leadership easily. But actually, if the master try again, it will renew lease successfully.

This problem will resulting in unnecessary learning mode and it takes time to converge.

What you expected to happen or what your proposal is:

I think we shold add retry mechanism when renew lease. If it fails twice or other retry-counts, then lose its leadership.

doorman client can't connect new master server when old master is down forever

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
feature request

What happened:
Assume only have root level node, there are three doorman servers, one is master and other two are slave.

In doorman client, I configure the doorman master server addr. As the time going on, the master server is down, so new master is elected. However, doorman client don't know anything about it, doorman client just retry connect the configured addr continually and get failure response continually. So client rate limiting not work normally.

What you expected to happen or what your proposal is:
I think we should configure all addrs,including master addr and slave addr, in doorman client. So when master server is down forever, doorman client can retry connect with new addr.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.