Repository used to main group ACLs used by Kubeflow developers
License: Apache License 2.0
Kubeflow the cloud-native platform for machine learning operations - pipelines, training and deployment.
Please refer to the official docs at kubeflow.org.
The Kubeflow community is organized into working groups (WGs) with associated repositories, that focus on specific pieces of the ML platform.
Please refer to the Community page.
/kind process
Follow up on #344
We've seen a couple of times that Github sync is broken when a member is added to a team but not the Kubeflow org.
For example:
$ kubectl logs github-sync-1600450800-gz94v -n github-admin
{"component":"peribolos","file":"prow/flagutil/github.go:78","func":"k8s.io/test-infra/prow/flagutil.(*GitHubOptions).Validate","level":"warning","msg":"It doesn't look like you are using ghproxy to cache API calls to GitHub! This has become a required component of Prow and other components will soon be allowed to add features that may rapidly consume API ratelimit without caching. Starting May 1, 2020 use Prow components without ghproxy at your own risk! https://github.com/kubernetes/test-infra/tree/master/ghproxy#ghproxy","time":"2020-09-18T17:40:21Z"}
{"client":"github","component":"peribolos","file":"prow/github/client.go:563","func":"k8s.io/test-infra/prow/github.(*client).log","level":"info","msg":"Throttle(300, 100)","time":"2020-09-18T17:40:21Z"}
{"component":"peribolos","file":"prow/cmd/peribolos/main.go:818","func":"main.configureOrg","level":"info","msg":"Skipping org metadata configuration","time":"2020-09-18T17:40:21Z"}
{"client":"github","component":"peribolos","file":"prow/github/client.go:563","func":"k8s.io/test-infra/prow/github.(*client).log","level":"info","msg":"ListOrgInvitations(kubeflow)","time":"2020-09-18T17:40:21Z"}
{"client":"github","component":"peribolos","file":"prow/github/client.go:563","func":"k8s.io/test-infra/prow/github.(*client).log","level":"info","msg":"User()","time":"2020-09-18T17:40:21Z"}
{"client":"github","component":"peribolos","file":"prow/github/client.go:563","func":"k8s.io/test-infra/prow/github.(*client).log","level":"info","msg":"ListOrgMembers(kubeflow, admin)","time":"2020-09-18T17:40:21Z"}
{"client":"github","component":"peribolos","file":"prow/github/client.go:563","func":"k8s.io/test-infra/prow/github.(*client).log","level":"info","msg":"ListOrgMembers(kubeflow, member)","time":"2020-09-18T17:40:21Z"}
{"component":"peribolos","file":"prow/cmd/peribolos/main.go:201","func":"main.main","level":"fatal","msg":"Configuration failed: failed to configure kubeflow members: all team members/maintainers must also be org members: karkumar","time":"2020-09-18T17:40:24Z"}
Such errors should be caught earlier if we had some presubmit test to check on this.
I submitted and merged a PR that adds "james-jwu" to org.yaml. I expect that I am able to access https://github.com/orgs/kubeflow/teams/pipelines/members, but cannot. It looks like I am not added to kubflow/members, and I should receive an invite to join but did not.
We should move the GitHub sync job into a GKE cluster in kubeflow.org so it can potentially be administered/monitored by non-googlers.
The blocker to doing that is doing a risk assement on what the blast radius could be?
In particular, can periobolos be used to remove/change the admins for the GitHub org? Can we enforce this by using a properly scoped GitHub token?
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
** Please list 2 existing members who are sponsoring your membership:**
** Please test your PR **
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
** Please list 2 existing members who are sponsoring your membership:**
** Please test your PR **
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
Could anyone here grant me write access to the following two repos?
Currently I am listed as maintainer both of these two repos under team "xgboost-operator-team" and "common-team" as specified in org.yaml. However, I don't have write access to these two repos yet. I need these access to:
/cc @jlewi
Since we have some discussions on #348 , starting with manual work may be easy at the beginning.
What make us worry about is there might be too much toil on Webhook Management work.
This issue is created to automate the process, now we have options:
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
kubeflow/kfp-tekton#1359
kubeflow/pipelines#10269
** Please list 2 existing members who are sponsoring your membership:**
@Tomcli
@yhwang
** Please test your PR **
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
** Please list 2 existing members who are sponsoring your membership:**
** Please test your PR **
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
mxnet-operator is missing from the list of repos managed by project-maintainers team.
/cc @abhi-g @jlewi @richardsliu
Could you add it since you are the maintainers of project-maintainers team?
/kind area/process
Replace all @kubeflow.org groups that should contain only Googlers with @google.com addresses
I believe that file was there less to offer explicit permins, and more to allow community members to note their involvement in the project. That being said, it's never been widely used and should probably be deprecated. @yanniszark or @Bobgy would either of you want to make a proposal to that effect?
Originally posted by @theadactyl in #401 (comment)
The member_organiziations.yaml hasn't been updated for a while, I can see a lot of stale information there. Shall we keep it updated or stop using it altogether? I don't see people using it anyway, but I could be wrong.
What?
Here is the PR: #356
cc: @PatrickXYS @jlewi @Bobgy
Right now we run our GitHub sync's as a cron job. It currently runs every 15 minutes and takes about 15 minutes to run.
github-sync-1604346000 0/1 124m 124m
github-sync-1604350200 1/1 14m 57m
github-sync-1604350800 1/1 14m 43m
github-sync-1604352000 1/1 14m 28m
github-sync-1604352600 0/1 14m 14m
If this continues we will get concurrent runs and I wonder if this could potentially cause problems.
It might be better to use the pattern we do for groups-sync. Where we run a reconcile loop
https://github.com/kubeflow/internal-acls/tree/master/google_groups
and use git-sync to synchronize the configs to a side car.
We'd like to add additional team members from GOJEK to the feast-team.members.txt file
We'll be creating a PR to add an internal Google group to that list so that we can manage the members on our side, or is the preferred approach to specify the member e-mails themselves in the list?
Given some GitHub team we established previously is not in use or deprecated,
the list of data points I can collect:
We should clean up those deprecated github team to formalize permission granting, given now we have WG, which should be folks who have write access to the repo, only a few other folks should have write access to the repo, e.g, release-team, google-admins, etc.
Recent change #576 was not synced, error log from the sync job:
Configuration failed: failed to configure kubeflow members: 1 errors: [status code 422 not one of [200], body: {"message":"The request could not be processed.","documentation_url":"https://docs.github.com/rest/reference/orgs#set-organization-membership-for-a-user"}]
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
kubeflow/kfp-tekton#977
kubeflow/kfp-tekton#1378
Please list 2 existing members who are sponsoring your membership:
@Tomcli
@yhwang
Please test your PR
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
We should automate syncing of a subset of groups.
The subset of groups should be restricted to non critical groups (e.g. not google-kubeflow-admins).
We should create a suitable bot account and make this a manager of the groups that we want to administer.
We should then setup a restricted cluster that has the appropriate credentials. Ideally we can setup a GCP service account.
We should use a restricted cluster and follow the process for setting up peribolos
kubeflow/community#55
Given the merged PR #353,
third-party-bots:
description: Team for third party bots
members:
- aws-kf-ci-bot
privacy: closed
repos:
katib: write
pytorch-operator: write
aws-kf-ci-bot account should have write access to katib and pytorch-operator repos.
Obviously, the third-party-bots team don't have access to both repos:
https://github.com/orgs/kubeflow/teams/third-party-bots/repositories
How can we fix the issue? I think it's blocking issue for Kubeflow Shared Test-infra POC, because bot account can't report Github Status.
/kind bug
Latest run of the cron job failed with
{"component":"peribolos","file":"prow/cmd/peribolos/main.go:201","func":"main.main","level":"fatal","msg":"Configuration failed: failed to configure kubeflow members: 1 errors: [status code 404 not one of [200], body: {\"message\":\"Not Found\",\"documentation_url\":\"https://developer.github.com/v3/orgs/members/#add-or-update-organization-membership\"}]","time":"2019-09-13T15:20:30Z"}
/kind area/process
Cannot get the pod logs by ci-viewer team.
[root@jinchi1 ~]# kubectl logs kfserving-controller-manager-0 -n kfserving-system
Error from server (Forbidden): pods "kfserving-controller-manager-0" is forbidden: User "[email protected]" cannot get resource "pods/log" in API group "" in the namespace "kfserving-system": Required "container.pods.getLogs" permission.
[root@jinchi1 ~]#
I also tried following here, still cannot work.
[root@jinchi1 ~]# gcloud --project=kubeflow-ci logging read --freshness=24h --order asc "resource.type=\"k8s_pod\" resource.labels.cluster_name=zing-presubmit-e2e-454-fd82d6a-8615-0ff2 resource.labels.pod_name=kfserving-controller-manager-0 "
ERROR: (gcloud.logging.read) PERMISSION_DENIED: Caller does not have required permission to use project project:kubeflow-ci. Grant the caller the Owner or Editor role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=project:kubeflow-ci and then retry (propagation of new permission may take a few minutes).
- '@type': type.googleapis.com/google.rpc.Help
links:
- description: Google developers console
url: https://console.developers.google.com
- '@type': type.googleapis.com/google.rpc.Help
links:
- description: Google developer console IAM admin
url: https://console.developers.google.com/iam-admin/iam/project?project=project:kubeflow-ci
If you want to invoke the command from a project different from the target resource project, use `--billing-project` or `billing/quota_project` property.
[root@jinchi1 ~]#
After this PR merged,
we should have wg-manifests and wg-deployment Github Team established,
given this query, https://github.com/orgs/kubeflow/teams?query=wg-
It's obviously sync script does not work, I'm guessing the issue happened before, happened again.
First section is aws-kf-ci-bot account is used to delegate AWS prow to run tests against PR.
So there are two options to grant permissions to aws-kf-ci-bot for some repos.
aws-ci-bots:
description: Team for AWS bots
members:
- aws-kf-ci-bot
privacy: closed
repos:
caffe2-operator: write
katib: write
mpi-operator: write
mxnet-operator: write
pytorch-operator: write
tf-operator: write
xgboost-operator: write
Another section is about managing webhook in repo-level.
I was thinking about using prow/hmac tool to manage webhook, and found
To run this tool, you'll need:
1. A github account that has admin permission to the orgs/repos.
...
Ref: https://github.com/kubernetes/test-infra/tree/1dd7c15ee5952ec573744ae4075956049aa71f3a/prow/cmd/hmac
Given it requires admin access for aws-kf-ci-bot to kubeflow repos, I don't think prow/hmac is able to solve the problem for now.
So I think we can manually add webhook into some repos.
The test will validate org.yaml file. We have seen at least a couple of times recently when people added new team members but forgot to add them to the org.
/kind bug
Here's the bug.
Traceback (most recent call last):
File "/code/validate_config.py", line 51, in <module>
fire.Fire(CheckConfig)
File "/usr/local/lib/python2.7/dist-packages/fire/core.py", line 127, in Fire
component_trace = _Fire(component, args, context, name)
File "/usr/local/lib/python2.7/dist-packages/fire/core.py", line 366, in _Fire
component, remaining_args)
File "/usr/local/lib/python2.7/dist-packages/fire/core.py", line 542, in _CallCallable
result = fn(*varargs, **kwargs)
File "/code/validate_config.py", line 18, in check_config
org = yaml.load(hf)
File "/usr/local/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
return loader.get_single_data()
File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 37, in get_single_data
node = self.get_single_node()
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/usr/local/lib/python2.7/dist-packages/yaml/composer.py", line 127, in compose_mapping_node
while not self.check_event(MappingEndEvent):
File "/usr/local/lib/python2.7/dist-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/usr/local/lib/python2.7/dist-packages/yaml/parser.py", line 439, in parse_block_mapping_key
"expected <block end>, but found %r" % token.id, token.start_mark)
yaml.parser.ParserError: while parsing a block mapping
in "/src/kubeflow/internal-acls/github-orgs/kubeflow/org.yaml", line 519, column 11
expected <block end>, but found '<block mapping start>'
in "/src/kubeflow/internal-acls/github-orgs/kubeflow/org.yaml", line 639, column 12
We have a pretty limited test: https://github.com/kubeflow/internal-acls/blob/master/github-orgs/test_org_yaml.py
To validate our config on presubmit. We should do more validation.
bazel build //prow/cmd/peribolos
ERROR: /home/jlewi/.cache/bazel/_bazel_jlewi/79cd09d50d0da543e57e27f66de55660/external/io_bazel_rules_appengine/appengine/py_appengine.bzl:145:48: Traceback (most recent call last):
File "/home/jlewi/.cache/bazel/_bazel_jlewi/79cd09d50d0da543e57e27f66de55660/external/io_bazel_rules_appengine/appengine/py_appengine.bzl", line 139
rule(_py_appengine_binary_base_impl, at..."]))}, <2 more arguments>)
File "/home/jlewi/.cache/bazel/_bazel_jlewi/79cd09d50d0da543e57e27f66de55660/external/io_bazel_rules_appengine/appengine/py_appengine.bzl", line 145, in rule
attr.label_list(allow_files = FileType([".yaml"]...)
File "/home/jlewi/.cache/bazel/_bazel_jlewi/79cd09d50d0da543e57e27f66de55660/external/io_bazel_rules_appengine/appengine/py_appengine.bzl", line 145, in attr.label_list
FileType([".yaml"])
FileType function is not available. You may use a list of strings instead. You can temporarily reenable the function by passing the flag --incompatible_disallow_filetype=false
ERROR: error loading package '': Extension file 'appengine/py_appengine.bzl' has errors
ERROR: error loading package '': Extension file 'appengine/py_appengine.bzl' has errors
INFO: Elapsed time: 0.539s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
This issue was automatically created by Allstar.
Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code
Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.
Remediation Steps
To remediate, remove the generated executable artifacts from the repository.
Artifacts Found
Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.
Allstar has been installed on all Google managed GitHub orgs. Policies are gradually being rolled out and enforced by the GOSST and OSPO teams. Learn more at http://go/allstar
This issue will auto resolve when the policy is in compliance.
Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.
We cannot remove @jlewi as admin, because this github sync is using his token: #384 (comment)
I'll update it to mine.
/kind process
For policy reasons, I don't think we can have non-Googlers approving requests to change the GitHub policy because granting a non-googler permission to approve PRs would be overly permissive; i.e. its probably permissive for non googlers to approve
joining the github org or a possible a github team; but changing github admins should not be allowed.
If we become bottlenecked on Googlers approving routine requests then we should automate the approval of routine requests.
Building a GitHub app that can verify only safe changes are included and adding an LGTM should be relatively straightforward.
/cc @animeshsingh
Take a look at Kubernetes'
https://github.com/kubernetes/org/blob/master/.github/ISSUE_TEMPLATE/membership.md
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
kubeflow/kfp-tekton#1259
kubeflow/pipelines#10640
kubeflow/pipelines#10568
** Please list 2 existing members who are sponsoring your membership:**
@rimolive && @HumairAK
** Please test your PR **
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
I was added as admin, but I cannot set up my GAM CLI properly.
I was following GAM install guide in https://github.com/jay0lee/GAM and I got
GAM is now installed. Are you ready to set up a Google API project for GAM? (yes or no) yes
Go to the following link in your browser:
https://gam-shortn.appspot.com/9zgvtf
Enter verification code:
When clicking the url, it tells me I don't have permission to create a project.
Hey @jbottum @andreyvelich @terrytangyuan @johnugeorge @james-jwu I was trying to tag the KSC in a new issue and noticed that the handle defined here #650 does not exist in GH. Not sure if that file also syncs GH groups or not.
/kind area/process
Please read the guide lines for joining the Kubeflow GitHub org before opening an issue
Please provide links to PRs or other contributions (2-3):
** Please list 2 existing members who are sponsoring your membership:**
@connor-mccarthy
@andreyvelich
** Please test your PR **
Run
cd github_orgs
pytest test_org_yaml.py
Include the output in the PR
===================================================================================== test session starts =====================================================================================
platform darwin -- Python 3.11.4, pytest-7.4.3, pluggy-1.0.0
rootdir: /Users/inn487/code/kubeflow/internal-acls/github-orgs
plugins: anyio-4.0.0, cov-4.1.0
collected 1 item
test_org_yaml.py . [100%]
====================================================================================== 1 passed in 0.11s ======================================================================================
Additional Instructions
After your PR is merged please wait at least 1 hour for changes to propogate.
If after an hour you haven't recieved an invite to join the GitHub org please open an issue.
You can contact build copy in #buildcop in kubeflow.slack.com
Right now the GitHub sync works using personal access tokens. Using a GitHub app would be better.
kubernetes/test-infra#10423 is tracking the upstream peribolos changes.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.