astrotomic / opendor.me Goto Github PK
View Code? Open in Web Editor NEWHome Page: https://opendor.me
License: Other
Home Page: https://opendor.me
License: Other
Add a second masked DB dump that can be used for local development and can also be shared to contributors.
https://beyondco.de/docs/laravel-masked-db-dump/installation
https://github.com/Astrotomic/opendor.me/discussions/new?category=ideas
@Gummibeer: Until further requests I will reject this one because of conflicts with recruiter tools, privacy and the primary target of opendor.me isn't a recruiting platform in it's original sense.
Some pages have many contributions. I think it would be nice to see the repos with the most stars.
Using GraphQL we are able to retrieve all important information in one request.
This would allow us to create a profile preview for new users.
Using a 7 day cache this also can't be misused to hit rate limits.
There's no need for a package like that anymore/yet. Abilities will be checked, if needed, by policies.
As we create users before they register we need another column to indicate when they've registered.
That column should be anonymized when needed and filled during first sign-in. In best case the core registered event is also fired.
You are running PHP CS Fixer v2, which is not maintained anymore. Please update to v3.
If you need help while solving warnings, ask at https://gitter.im/PHP-CS-Fixer, we will help you!
Could be fun for users that have an account on opendor.me to star/upvote another profile.
And with that you can hight the top upvoted profiles in the homepage on the given week.
If Stars it's to similar to GitHub just name it to "Knock" β it would be fun saying:
Hey! Look how many knocks I have on opendor.me.
Description
Unable to log in on Mac Safari and iPad Safari.
To Reproduce
On Mac and iPad Safari when I navigate to the opendor.me URL and click βsign-inβ, the page appears to start to redirect to GitHub but does not complete and instead the OpenDor home page just refreshes. If I try to sign in on other pages e.g. Sponsors the same happens and the page just goes back to the home page.
On some elements throughout the app I use the truncate
class - but this doesn't work if the truncated element is a direct child of a flexbox one.
This isn't a heavy bug but a bit annoying to see overflows.
@aidan-casey has already proposed two general solutions - so in case anyone fixes this, please author Aidan as well.
You can use https://github-author.astrotomic.info to find the required git author line by username.
https://play.tailwindcss.com/7kUHkqwVLS
https://play.tailwindcss.com/q2o1zT466U
Screenshots
If applicable, add screenshots to help explain your problem.
Describe
I get a 404 when I try to use Sign-in using OAuth on Github in a local development environment.
cause
Is the environment variable setting in the .env.example file described in the Github Setup section of the README wrong?
Environment variable settings in the README(Screenshots)
Environment variable settings in the .env.example file(Screenshots)
As mentioned above, isn't the content to be written in the README to write the environment variable settings for .env.example?
As any user I want to see a maximum of 6 repositories per vendor on the profile page - additional will be collapsed.
Only 6 repositories should be shown, all exceeding will be wrapped in a collapsible which has a toggle button/link below the last row of repositories.
Right now the user-autocomplete on the homepage hero section is limited to 3 results as the required overflow-hidden
on some wrapping elements cuts of the absolute positioned suggestions as well and/or make the container scroll.
The wanted result is that they are positioned and z-indexed "outside" the hero section and are just an overlay that can show 3, 5 or even 10 results without any other element scrolling and also not getting cut by any container.
Here's a screenshot of the current behavior - you can see that the results are cut-off and there's a scrollbar on the right.
The GitHub API defines a Link
header containing pagination relevant links. Using that one would be more precise than the current count($response)
solution.
This should be done in the GitHub base job.
From time to time requests fail with a 502 Bad Gateway
as even the GitHub API has downtimes. Similar to #140 we should catch this exception and do a proper retry of the job. But in that case we can use the third argument of the retry()
method in the Client builder as we want to just retry the exact same request if we get a 502
. laravel/framework#38531
The when
Closure should use a full closure - likely static
(https://twitter.com/devgummibeer/status/1425452816043286536) - to allow more complex checks in the future and add more status codes to retry without having a too large diff. In best case it should also use the Closure option for the delay to increase the delay with every try. Starting with 500ms and ending with 30s or something over 3-5 retries.
This logic should be added in the Http::github()
macro to have the same logic for all GitHub API requests.
After the GitHub access token is removed the user profile should be fully anonymized.
There should also be a nova action to trigger this manually as an administrator.
I'm a public member of @035media org which has some public repos, but they are not showing up on my profile. Anything I'm doing wrong, or some bug?
When I view my contributions page or profile page, I can't see all repositories I've contributed to.
Example:
View a repo I've committed to: https://github.com/ploi-deploy/ploi-translations/graphs/contributors
View my profile on opendor.me: https://opendor.me/@adevade
It's not included on my profile.
If I'm not mistaken, I expect it to show all repos I've contributed to? Both small and big contributions.
Or does it only show repos from users that are also connected to opendor.me?
If that's the case, maybe add some information about it in the FAQ?
relates to #82
Instead of doing $this->paginated(...)
in the job it would be cool to do Http::github()->get()->paginated(fn() => ...)
.
The paginated
macro should accept one closure argument that runs for each response and gets the response instance as it's own argument.
Http::github()
->get("/orgs/{$this->organization->name}/repos")
->paginated(function(\Illuminate\Http\Client\Response $response): void {
$response->collect()->map(...);
});
opendor.me/app/Jobs/SyncUserContributions.php
Lines 25 to 47 in b258e25
GitHub has an option for organizations to enforce SAML SSO for all API requests made to their organization by members.
https://docs.github.com/en/rest/overview/other-authentication-methods#authenticating-for-saml-sso
In case this happens the response is a 403 Forbidden
with the following headers and body:
x-github-sso: required; url=https://github.com/orgs/stjude/sso?authorization_request=ABCXYZ
{
"message":"Resource protected by organization SAML enforcement. You must grant your OAuth token access to this organization.",
"documentation_url":"https://docs.github.com/articles/authenticating-to-a-github-organization-with-saml-single-sign-on/"
}
This has a massive impact on the queue right now they are catched by the rate-limit retry logic and do 12 requests over the duration of 12 hours to the API without any benefit/reason.
So far I know there's no attribute provided by API that indicates if an organization enforces SAML SSO. So the only way is to try'n'catch.
All these requests can pass and get a response if they are done with a non-member access-token. This works for us as we only request public data that wouldn't need an access-token at all. This problem is already reported to GitHub Support and is on their backlog/roadmap without any ETA.
As it prevents us right now from syncing two organizations:
We have to solve it with a "temporary" fix. The idea is to detect this response the same as the rate-limit one and instead of just retrying it 1h later we would have to switch to a non-member access-token. So this will have to be solved in a Guzzle middleware.
I would like to maintain the sponsors as models using YAML files. Similar to the FAQ model.
Right now there's no public API for opendor.me data - but there could/should be one.
A GraphQL API would be pretty cool and powerful as the user would be able to retrieve whatever they need without having us to adjust the API.
It should be possible to authenticate as an user or organization - so we would need a new table api_keys
that has a morph relationship to user or organization model. The API should be read-only for the moment and be limited to the basic and fully public information. It should also only return the data for the authenticated user for the moment but use policies and to limit access so that we could extend it easily.
https://github.com/rebing/graphql-laravel
https://twitter.com/enunomaduro/status/1429210254827565065
https://github.com/miniaturebase/pest-plugin-graphql
A repository can be disabled by GitHub - this results in a 403
API response. Right now this isn't explicitly handled and results in "infinite" retries and after all failing the job.
https://api.github.com/repos/humaneguy/go-waller-web-version/contributors
Request URL: https://api.github.com/repos/humaneguy/go-waller-web-version/contributors
Request Method: GET
Status Code: 403
{
"message": "Repository access blocked",
"block": {
"reason": "tos",
"created_at": "2021-03-25T16:05:28Z",
"html_url": "https://github.com/tos"
}
}
The InvalidStateException
shouldn't be thrown but can happen - so it should be handled properly instead of showing a 500 page.
Right now I'm using simple SSH deployment - because I have multiple hosts and want to maintain the deployment script next to the code I want to use https://deployer.org instead.
If you don't know deployer yet - @lorisleiva has a great course about deployer with Laravel https://lorisleiva.com/deploy-your-laravel-app-from-scratch
The current deployment script is:
$FORGE_PHP artisan backup:run
rm -f bootstrap/cache/config.php
$FORGE_PHP artisan optimize:clear
git pull origin main
$FORGE_COMPOSER update --no-interaction --prefer-dist --optimize-autoloader --no-dev
( flock -w 10 9 || exit 1
echo 'Restarting FPM...'; sudo -S service $FORGE_PHP_FPM reload ) 9>/tmp/fpmlock
yarn install
yarn production
$FORGE_PHP artisan blade-fontawesome:sync-pro
$FORGE_PHP artisan icons:cache
$FORGE_PHP artisan nova:publish
$FORGE_PHP artisan horizon:publish
$FORGE_PHP artisan config:cache
$FORGE_PHP artisan event:cache
$FORGE_PHP artisan route:cache
$FORGE_PHP artisan view:cache
$FORGE_PHP artisan migrate --force
$FORGE_PHP artisan permission:cache-reset
$FORGE_PHP artisan permission:ensure
$FORGE_PHP artisan schedule-monitor:sync
The current homepage performance isn't the best. Even with caching it takes some time.
This should be improved.
First idea is to remove some global scopes to increase query performance.
Laravel Debugbar should help here to measure things.
Add Opengraph tags including large social image for better shareability.
On my profile currently the "vendors" (orgs and my personal account) appear to be rendered in alphabetical order. I would like to be able to set an order for them to promote orgs that I more actively work in at the top.
As unregistered profiles aren't public anymore the counts on homepage should be adjusted to the theoretically visible values.
The Microsoft profile page is too slow. Even the spatie one was slow already but acceptable. But Microsoft takes multiple seconds to load.
We have to find a solution to keep the page loading speed consistent independent of repository count.
There are two heavy things:
contributions
(user) or repositories
(organization) are slow.The query isn't that bad yet - so I would focus on the blade part. There should be some kind of pagination. The current approach on the user profile seems good - show 6 repos and have a "show more/all" button. There should be an info like "showing 12 of 258 repositories".
But the current user profile still renders all entries Serverside but hides them.
This has to be changed to a client rendered version and API.
Another idea would be to offload this to Algolia. Show 6 by default and have a search bar - with autosuggestion (?).
The members of an organization face the same problem - microsoft has ~4500 public members ...
I think that we could remove that list and instead link to a prefilled advanced user search. We could do the same for contributors.
Right now the GitHub sponsors are matched to a possible local model one by one. This results in n
queries to the database even if the sponsors are only in two tables.
opendor.me/app/View/Components/Web/GithubSponsors.php
Lines 55 to 67 in 84816db
This could be improved to do batch selects - provided that the PHP collection matching is faster than n
queries. Right now this isn't really slow - also because it's cached for some time, but it could lead to problems in the future. Every single query takes ~2 - 5ms
- these will get slower by the increasing count of records in the tables and the total time by an increasing count of sponsors.
View the following link to download the report.
RunnerID:717232359
Right now we are running in some rate-limit problems as the current sync process goes the following way:
You see that with every stage we will need more requests but the limit is defined by the number of users who are registered.
This also comes with some limitations and drawbacks:
All this is based on the limitations of the GitHub REST API. By switching to the "new" GitHub GraphQL API we can retrieve all contributions a user has made in a given timeframe. Like most GraphQL endpoints this is limited to 100 repos per call - so walking back in time in ~3-6 month steps should be fine. (I'm pretty sure that no one contributes to more than 100 different repos within 3 months. π€
So even if we go back to the user registration (2013 in my case) this would result in 4 requests per year (2021 - 2013) * 4 = 32 requests
- so 32 requests to retrieve all repositories I have contributed to. Right now we do ~200000 requests per day to sync ~500 users. So we go down to ~10% of the required requests - which will speed up the process, consume fewer requests of available rate-limits, find all repos and only sync the registered users instead of thousands of users not on opendor.me
The basic GraphQL query I have so far is the following. It's not finished yet - but includes most of the things we need. That's a PoC query that I built to check if I can get all info I need to sync a repo and contribution.
query(
$from: DateTime,
$to: DateTime
) {
viewer {
login
contributionsCollection(from: $from, to: $to) {
hasActivityInThePast
commitContributionsByRepository(maxRepositories: 100) {
repository {
nameWithOwner
isArchived
isDisabled
isEmpty
isFork
isLocked
isPrivate
owner {
__typename
login
}
licenseInfo {
name
spdxId
pseudoLicense
}
primaryLanguage {
name
color
}
languages(first: 100) {
nodes {
name
color
}
}
}
}
}
}
}
The whole code to play around with is the following:
use Carbon\CarbonImmutable;
$github = Http::baseUrl('https://api.github.com')->withToken(GH_PAT);
$repos = collect();
$end = CarbonImmutable::now();
do {
$response = $github
->post('/graphql', [
'query' => trim($query),
'variables' => [
"from"=> $end->subMonths(6)->startOfDay()->toIso8601ZuluString(),
"to"=> $end->endOfDay()->toIso8601ZuluString(),
],
])
->json();
$response = data_get($response, 'data.viewer.contributionsCollection');
$end = $end->subMonths(3);
$repos = $repos->merge(
data_get($response, 'commitContributionsByRepository.*.repository')
);
} while($response['hasActivityInThePast']);
$repos = $repos->unique('nameWithOwner')->keyBy('nameWithOwner')->sortBy('nameWithOwner');
This will be a major refactoring of current code as a lot of code won't be needed anymore, some new will be added and the overall process how to retrieve data will change.
We should add a "sponsor" button on all profiles having sponsorships (via GitHub) enabled.
To do so we will have to sync if they have sponsoring enabled in the \App\Jobs\UpdateUserDetails
job which will also have to switch to GraphQL API as the sponsor information isn't available elsewhere.
This should also result in a PR to https://github.com/Astrotomic/laravel-github-sponsors with a method like hasSponsorsListing()
or hasSponsoringEnabled()
.
https://docs.github.com/en/graphql/reference/objects
With the new attribute on user and organization we can show a conditional button on all profiles linking to the GitHub sponsor page.
The LoadRepositoryContributors
queue job should only do the API calls to retrieve the contributors. It should dispatch one job to the same batch per contributor.
This will result in an insane increment of processed jobs but keep the single job timeouts super low - so we don't have to keep an eye on one job and have it blocking for 6h or more.
One thing to check: can batch jobs be dispatched to a different queue? If so we should check to add a new github-contributor
(TBD) queue so that these jobs aren't blocking for all the other github jobs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.